Repository: minio/minio (language: go)

Issue:
S3 Select causing high memory consumption on >3GiB file
When running a select statement on a gzip json dataset (~3 GiB), the memory usage spikes significantly more than I would expect.

![image](https://github.com/user-attachments/assets/d9795408-b646-41aa-ad1b-419415e9c332)

## Expected Behavior

I would expect the memory consumption of an S3 Select operation to be relative to the limit on the sql command, and that stream data processing is used, so the entire object is not loaded  into memory.

## Current Behavior
I'm using the mc client to run an S3 Select statement on my ~3GiB gzip json file (11.5 GiB uncompressed). The file is a json array (i.e. not NDJSON), so I query it with the following command:
```
mc sql myminio/mybucket/dataset.json --query "SELECT * FROM S3Object[*]._1[*] s LIMIT 10" --json-input type=DOCUMENT --compression GZIP
```

The command hangs for a while, before I can see the minio pod is OOMKilled - This is running on a 32GiB box, where all other workloads have been removed, so it is free to use all the resources.

I tried this on a 64GiB box, which worked, but the memory stayed high after the command returned a response, for longer than I'd expect (as seen in the diagram of memory usage).

## Possible Solution
¯\_(ツ)_/¯

## Steps to Reproduce (for bugs)
1. Setup Minio on a box with 32GiB memory.
2. Create a 3GiB Gzip JSON file, storing a json array of records: i.e. `[{ "a": 1, ... }, ...]`, and upload to bucket
3. Run a select command:
```
mc sql myminio/mybucket/dataset.json --query "SELECT * FROM S3Object[*]._1[*] s LIMIT 10" --json-input type=DOCUMENT --compression GZIP
```

## Context
I want to allow users to query parts of the dataset in a language they are familiar with (SQL).

## Regression
Not sure, but I have seen this issue before which I thought could be related: https://github.com/minio/minio/issues/17235

## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used (`minio --version`):
minio version DEVELOPMENT.2024-09-09T16-59-28Z (commit-id=0b7aa6af879e088030d63e91a29dabf22fdd3a18)
Runtime: go1.22.7 linux/amd64
License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
Copyright: 2015-2024 MinIO, Inc.

* Server setup and configuration: Deployed to an AKS cluster using the bitnami minio helm chart version 14.7.8 with default config, but using nodeSelector and tolerations to force it onto a node with no workloads running.

* Operating System and version (`uname -a`): Node image version: AKSUbuntu-2204gen2containerd-202409.04.0



Discussion / hints:
An array is effectively one object. There is no streaming possible on this.

Use NDJSON with separate entries. If any change comes from this, it will be limiting single object sizes. So this will not end up working either way.
FWIW AWS limits to 1MB objects: https://docs.aws.amazon.com/AmazonS3/latest/userguide/selecting-content-from-objects.html

Fix patch:
diff --git a/internal/s3select/json/reader.go b/internal/s3select/json/reader.go
index 4285c23fa5211..52eda1404866f 100644
--- a/internal/s3select/json/reader.go
+++ b/internal/s3select/json/reader.go
@@ -26,6 +26,10 @@ import (
 	"github.com/bcicen/jstream"
 )
 
+// Limit single document size to 10MiB, 10x the AWS limit:
+// https://docs.aws.amazon.com/AmazonS3/latest/userguide/selecting-content-from-objects.html
+const maxDocumentSize = 10 << 20
+
 // Reader - JSON record reader for S3Select.
 type Reader struct {
 	args       *ReaderArgs
@@ -80,7 +84,7 @@ func (r *Reader) Close() error {
 // NewReader - creates new JSON reader using readCloser.
 func NewReader(readCloser io.ReadCloser, args *ReaderArgs) *Reader {
 	readCloser = &syncReadCloser{rc: readCloser}
-	d := jstream.NewDecoder(readCloser, 0).ObjectAsKVS()
+	d := jstream.NewDecoder(io.LimitReader(readCloser, maxDocumentSize), 0).ObjectAsKVS()
 	return &Reader{
 		args:       args,
 		decoder:    d,
diff --git a/internal/s3select/select.go b/internal/s3select/select.go
index 6026c9b3c847c..74b8f43ed472d 100644
--- a/internal/s3select/select.go
+++ b/internal/s3select/select.go
@@ -442,6 +442,7 @@ func (s3Select *S3Select) Open(rsc io.ReadSeekCloser) error {
 				s3Select.recordReader = json.NewPReader(s3Select.progressReader, &s3Select.Input.JSONArgs)
 			}
 		} else {
+			// Document mode.
 			s3Select.recordReader = json.NewReader(s3Select.progressReader, &s3Select.Input.JSONArgs)
 		}
 


Test patch: