Repository: minio/minio (language: go)

Issue:
Performance issue when serving files stored on HDD
Hello,

I come here after hitting the issue described at https://github.com/minio/minio/issues/13907. Please note that I solved the PUT issue (that arise only for non-multipart uploads) by setting _MINIO_CACHE_COMMIT="writeback"_ to my configuration. Since then I didn't got any issue with this configuration. Meanwhile I want to report my case as maybe it will be useful to others and also because I didn't found any alternative to using the disk cache, making this feature necessary for me.


My Minio cluster is build with 4 identical servers (deployed on a bare-metal cloud):
- 6 cores, 12 threads CPU
- 128GB RAM
- 1Gb public network (used to serve files to clients, Minio being behind Nginx here)
- 2Gb private network (used for Minio communication)
- 6 6TB HDD used as JBOD for Minio (XFS)
- 2 512GB SSD used as RAID1 for OS (Debian 11, Linux 5.10)
- Minio RELEASE.2022-01-04T07-41-07Z

My Workload being:
- Approximately 40TB of files stored on Minio using _EC:3_ (Used the Minio erasure calculator to get one server failure tolerance)
- Small files workload, the vast majority being between 1MB and 10MB (audio files)
- Workload strongly biased toward a very small subset of the files, around 2GB at a time
- Rare, random like access to the other files not in the 2GB of very popular files
- Very low write activity


When I activate Minio caching (cache on the SSD mirror), after between one and two minutes, I get a substantial improvement to my TTFB, the cluster coming from unusable to lightning fast. Please note that I calculate the TTFB as from Minio to Nginx and not from Nginx to the client:
- 50 percentile TTFB: from 3  sec to 0.004 sec
- 75 percentile TTFB: from 10 sec to 0.02 sec
- 95 percentile TTFB: from 30 sec to 0.056 sec
- 99 percentile TTFB: from 50 sec to 0.08 sec

![image](https://user-images.githubusercontent.com/59954771/165051437-26e3ad7a-d63c-4504-b2fe-04674dd45657.png)

Once these performance numbers are reached few minutes after cluster start-up it stay steady state and I never saw any flare in TTFB nor request time.
Minio disk caching is a very stable, performant and efficient solution :smiley: 

As I have 128GB of RAM per server, a mostly read based workload, and only a few GB of files hit by most requests, I first thought I wouldn't need disk caching. Even if Minio doesn't have any internal in RAM object data caching (or I didn't found it), the Linux VFS cache should do his job and just keep the objects that are accessed in RAM, ensuring strong performance. Meanwhile this doesn't happen.

After investigating I discovered by using the trace function that Minio open the file using O_DIRECT by default (as Minio is mainly intended to be used with NVMe drives). After reading this https://github.com/minio/minio/commit/70e1cbda21b2e41069f0167e64bb1f7c55799cbd, I tried to start my cluster with _MINIO_API_DISABLE_ODIRECT="on"_, but it seem to not have any effect, when using the trace function the files where still opened with O_DIRECT and my cluster performance rapidly deteriorated without the disk cache. Maybe I didn't use this option right, but the documentation about it is scarce.

Also, as my cluster is accessed behind Nginx by the clients, I tried to setup Nginx [request caching](https://docs.nginx.com/nginx/admin-guide/content-cache/content-caching/). At first I thought I has found a viable replacement for Minio disk caching. Sadly I didn't ever reach the performance number of Minio. And worst, without explication, rather than performance staying steady state the cluster would sometime see the TTFB and request time flare up for a few minutes or more.

After some investigation, I thought it was because I didn't configured Nginx to perform range requests caching as Minio is doing it. Therefore I looked at the ["slice" module](https://www.nginx.com/resources/wiki/modules/slice/) to try to get an equivalent configuration. I didn't saw a difference once activated. I tried to let the cluster run so, but after few hours the performance numbers suddenly flared to an intolerable level. I has to put back Minio disk cache on even if this is a deprecated feature.


While I get the logic behind "SSDs can provide IOPS but with a concurrent workload that distributed setup supports you are basically choking single SSD v/s a large pool of HDD that will be any day more performant.", I don't see my cluster as under-provisioned. To be fair I even see it as over-provisioned for the target workload. I have 128GB of RAM and only around 2GB of files make most of the accesses. I know that these files aren't overwritten, therefore the Linux VFS cache cannot be invalidated. Also, the LRU replacement algorithm should fit my workload pretty well.

As you don't plan to support the use of disk caching for erasure coded clusters, please can you document how to make Minio work with VFS based caching as this seem the most straightforward replacement of this feature for use cases like mine ?

Thanks,
Best Regards.


Discussion / hints:
All performance issues and architecture review require subscription @mathieu-jamendo 

Also you shouldn't be using caching with distributed setups, caching is designed to be used is edge nodes that serve assets from cloud using gateway. 

You are loosing IOPs by using caching on disk when your HDDs have more IOPs than your local cache SSD. 

AFAICS you simply are building a production cluster in SILO with no MinIO architectural level involvement. We can't imagine your hardware and workload pattern. You want more help come to https://min.io/pricing 
Hi again @harshavardhana,

While reading [xl-storage.go](https://github.com/minio/minio/blob/3a6458066331c5c5ed70e41c920fb0046e2e227e/cmd/xl-storage.go) it seem that:

Minio call _OpenFileDirectIO_ at line 1659:
`file, err := OpenFileDirectIO(filePath, readMode, 0o666)`

Then, check if O_DIRECT is disabled later:
```	alignment := offset%xioutil.DirectioAlignSize == 0
	if !alignment || globalAPIConfig.isDisableODirect() {
		if err = disk.DisableDirectIO(file); err != nil {
			file.Close()
			return nil, err
		}
	}
```

Therefore when using _MINIO_API_DISABLE_ODIRECT="on"_ it would generate a tracing event that look like O_DIRECT is still active, which can be confusing for users.
Hello,

Posting for those who might find themselves in a similar issue.

It seem that Minio is very optimized for running on SDD, therefore when it receive many requests at once it will try to execute them all concurrently. Thus the pressure will fall on the Linux block scheduler which will try to find the best way to run that many requests on HDD.

But if you use Nginx in front of Minio, you can limit the number of concurrent requests sent to Minio using the _upstream_ module. This in addition to Nginx caching and disabling O_DIRECT solved my performance issues for my cache friendly workload on my HDD based Minio cluster.

Fix patch:
diff --git a/go.mod b/go.mod
index fb33192615761..5700bfcc8aab4 100644
--- a/go.mod
+++ b/go.mod
@@ -50,7 +50,7 @@ require (
 	github.com/minio/kes v0.19.2
 	github.com/minio/madmin-go v1.3.12
 	github.com/minio/minio-go/v7 v7.0.24
-	github.com/minio/pkg v1.1.22
+	github.com/minio/pkg v1.1.23
 	github.com/minio/selfupdate v0.4.0
 	github.com/minio/sha256-simd v1.0.0
 	github.com/minio/simdjson-go v0.4.2
diff --git a/go.sum b/go.sum
index 34bb49919af4a..79cace6689816 100644
--- a/go.sum
+++ b/go.sum
@@ -625,8 +625,8 @@ github.com/minio/minio-go/v7 v7.0.23/go.mod h1:ei5JjmxwHaMrgsMrn4U/+Nmg+d8MKS1U2
 github.com/minio/minio-go/v7 v7.0.24 h1:HPlHiET6L5gIgrHRaw1xFo1OaN4bEP/082asWh3WJtI=
 github.com/minio/minio-go/v7 v7.0.24/go.mod h1:x81+AX5gHSfCSqw7jxRKHvxUXMlE5uKX0Vb75Xk5yYg=
 github.com/minio/pkg v1.1.20/go.mod h1:Xo7LQshlxGa9shKwJ7NzQbgW4s8T/Wc1cOStR/eUiMY=
-github.com/minio/pkg v1.1.22 h1:Fm3oPu9LJag0FhD5BjFxj0Ut+M8S8IkiPHaq5OAoZaM=
-github.com/minio/pkg v1.1.22/go.mod h1:z9PfmEI804KFkF6eY4LoGe8IDVvTCsYGVuaf58Dr0WI=
+github.com/minio/pkg v1.1.23 h1:CJSoPslQCWZW3z3T79+pv9dVBDCQEK3ipiwXcoAtzY0=
+github.com/minio/pkg v1.1.23/go.mod h1:z9PfmEI804KFkF6eY4LoGe8IDVvTCsYGVuaf58Dr0WI=
 github.com/minio/selfupdate v0.4.0 h1:A7t07pN4Ch1tBTIRStW0KhUVyykz+2muCqFsITQeEW8=
 github.com/minio/selfupdate v0.4.0/go.mod h1:mcDkzMgq8PRcpCRJo/NlPY7U45O5dfYl2Y0Rg7IustY=
 github.com/minio/sha256-simd v0.1.1/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM=
diff --git a/internal/config/identity/openid/jwt.go b/internal/config/identity/openid/jwt.go
index 62dd8a2ed2d2a..6448f7e4d18d9 100644
--- a/internal/config/identity/openid/jwt.go
+++ b/internal/config/identity/openid/jwt.go
@@ -26,7 +26,6 @@ import (
 	"errors"
 	"fmt"
 	"io"
-	"net"
 	"net/http"
 	"strconv"
 	"strings"
@@ -775,12 +774,11 @@ func LookupConfig(kvsMap map[string]config.KVS, transport http.RoundTripper, clo
 		}
 		seenClientIDs.Add(p.ClientID)
 
-		var configURLDomain string
 		p.URL, err = xnet.ParseHTTPURL(configURL)
 		if err != nil {
 			return c, err
 		}
-		configURLDomain, _, _ = net.SplitHostPort(p.URL.Host)
+		configURLDomain := p.URL.Hostname()
 		p.DiscoveryDoc, err = parseDiscoveryDoc(p.URL, transport, closeRespFn)
 		if err != nil {
 			return c, err
@@ -809,7 +807,17 @@ func LookupConfig(kvsMap map[string]config.KVS, transport http.RoundTripper, clo
 			// `iampolicy.PolicyName` as the claim name explicitly and sets
 			// a role policy, this check is thwarted, but we will be using
 			// the role policy anyway.
-			return c, config.Errorf("Role Policy (=`%s`) and Claim Name (=`%s`) cannot both be set.", p.RolePolicy, p.ClaimName)
+			return c, config.Errorf("Role Policy (=`%s`) and Claim Name (=`%s`) cannot both be set", p.RolePolicy, p.ClaimName)
+		}
+
+		jwksURL := p.DiscoveryDoc.JwksURI
+		if jwksURL == "" {
+			return c, config.Errorf("no JWKS URI found in your provider's discovery doc (config_url=%s)", configURL)
+		}
+
+		p.JWKS.URL, err = xnet.ParseHTTPURL(jwksURL)
+		if err != nil {
+			return c, err
 		}
 
 		if p.RolePolicy != "" {
@@ -821,9 +829,9 @@ func LookupConfig(kvsMap map[string]config.KVS, transport http.RoundTripper, clo
 			domain := configURLDomain
 			if domain == "" {
 				// Attempt to parse the JWKs URI.
-				domain, _, _ = net.SplitHostPort(p.JWKS.URL.Host)
+				domain = p.JWKS.URL.Hostname()
 				if domain == "" {
-					return c, config.Errorf("unable to generate a domain from the OpenID config.")
+					return c, config.Errorf("unable to generate a domain from the OpenID config")
 				}
 			}
 			if p.ClientID == "" {
@@ -850,16 +858,6 @@ func LookupConfig(kvsMap map[string]config.KVS, transport http.RoundTripper, clo
 			return c, config.Errorf("A role policy or claim name must be specified")
 		}
 
-		jwksURL := p.DiscoveryDoc.JwksURI
-		if jwksURL == "" {
-			return c, config.Errorf("no JWKS URI found in your provider's discovery doc (config_url=%s)", configURL)
-		}
-
-		p.JWKS.URL, err = xnet.ParseHTTPURL(jwksURL)
-		if err != nil {
-			return c, err
-		}
-
 		if err = p.initializeProvider(getCfgVal, c.transport); err != nil {
 			return c, err
 		}


Test patch: