Repository: minio/minio (language: go)

Issue:
Truncated Response without Continuation Token
**Context**
We have a problem regarding the S3 ListObjectV2 request.

**Current Behavior**
Sometimes we have an error in our application indicating that the response from MinIO server is truncated but there is no continuation token.

![image](https://user-images.githubusercontent.com/78885532/150153185-6c4116d9-315e-4986-a322-c46af499c399.png)

We checked the HTTP packets and in fact it seems that MinIO returns a truncated response without the continuous tokens and key count set to zero.
Check the picture below: In the left side it's the wrong response.

![image](https://user-images.githubusercontent.com/78885532/150154171-18e08274-658a-4d5f-9b6a-2dadd7937c06.png)

In this picture we can see that it doesn't always happen. Some requests return fine (right side) and few milisesond after it returns bad responses for the same S3 prefix.
When we restart the service, it works fine for a few hours and then we observe these responses again.

**Your Environment**
 Minio RELEASE.2022-01-08T03-11-54Z on Centos 7 (Kernel 3.10.0-1160.45.1.el7.x86_64) and installed in standalone mode.






Discussion / hints:
@mesobreira What is your server setup? Does "installed in standalone mode" mean single drive setup? Could you post your command line?
Hello Klauspost, 

Yes, it's a single server with a single drive. What command line are you talking about?
> Yes, it's a single server with a single drive. What command line are you talking about?

how are you running MinIO? 
 Minio is installed in a single VM  (in VMware). 
It's not installed in distributed mode.
The Operating System is CentOS Linux release 7.9.2009 (Core)
The Minio is installed as a Linux service running under a user named minio. We are not using container image.
The command line used to start Minio is:  "/usr/local/bin/minio server --address :9000 --console-address :9199 /data"  
There is only 1 mountpoint : /dev/mapper/minio_vg-data on /data type xfs (rw,relatime,attr2,inode64,noquota)
Currently, the used space is 1.4TB.

Regarding the OS configuration, we set the following kernel settings:
kernel.pid_max=4194303
fs.file-max=4194303
vm.swappiness = 1
vm.vfs_cache_pressure = 10
vm.min_free_kbytes=1000000
net.core.rmem_max=268435456
net.core.wmem_max=268435456
net.core.rmem_default=67108864
net.core.wmem_default=67108864
net.core.netdev_budget=1200
net.core.optmem_max=134217728
net.core.somaxconn=65535
net.core.netdev_max_backlog=250000
net.ipv4.tcp_rmem=67108864 134217728 268435456
net.ipv4.tcp_wmem=67108864 134217728 268435456
net.ipv4.tcp_low_latency=1
net.ipv4.tcp_adv_win_scale=1
net.ipv4.tcp_max_syn_backlog=30000
net.ipv4.tcp_max_tw_buckets=2000000
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_fin_timeout=5
net.ipv4.udp_rmem_min=8192
net.ipv4.udp_wmem_min=8192
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.all.accept_redirects=0
net.ipv4.conf.all.accept_source_route=0
net.ipv4.tcp_mtu_probing=1

Do you need any additional information ?


can you provide `mc admin trace -v` output for this specific error that is seen by the client? 
It seems likely to be because of [this check](https://github.com/minio/minio/blob/f6190d6751590c3b2b2fc4d4f6c2e880a253d7a2/cmd/object-api-common.go#L400).

The `for i = 0; i < maxKeys; i++ {` loop may not add objects, but exits regardless. 

In this case it only adds a prefix, but no objects before exiting.

The concurrency makes the fix non-trivial, though.
> It seems likely to be because of [this check](https://github.com/minio/minio/blob/f6190d6751590c3b2b2fc4d4f6c2e880a253d7a2/cmd/object-api-common.go#L400).
> 
> The `for i = 0; i < maxKeys; i++ {` loop may not add objects, but exits regardless.
> 
> In this case it only adds a prefix, but no objects before exiting.
> 
> The concurrency makes the fix non-trivial, though.

Yes that is correct, it sends an empty response. In such a scenario perhaps Truncated is perhaps incorrectly set.
@harshavardhana I have up trying to fill the keys, which became very messy. I'm trying out:

```
	if !eof {
		result.IsTruncated = true
		if len(objInfos) > 0 {
			result.NextMarker = objInfos[len(objInfos)-1].Name
		} else if len(result.Prefixes) > 0 {
			result.NextMarker = result.Prefixes[len(result.Prefixes)-1]
		}
	}
```

Fix patch:
diff --git a/cmd/object-api-common.go b/cmd/object-api-common.go
index edb1ce542f0b1..69b12991bfee1 100644
--- a/cmd/object-api-common.go
+++ b/cmd/object-api-common.go
@@ -164,6 +164,8 @@ func listObjectsNonSlash(ctx context.Context, bucket, prefix, marker, delimiter
 		result.IsTruncated = true
 		if len(objInfos) > 0 {
 			result.NextMarker = objInfos[len(objInfos)-1].Name
+		} else if len(result.Prefixes) > 0 {
+			result.NextMarker = result.Prefixes[len(result.Prefixes)-1]
 		}
 	}
 
@@ -399,6 +401,8 @@ func listObjects(ctx context.Context, obj ObjectLayer, bucket, prefix, marker, d
 		result.IsTruncated = true
 		if len(objInfos) > 0 {
 			result.NextMarker = objInfos[len(objInfos)-1].Name
+		} else if len(result.Prefixes) > 0 {
+			result.NextMarker = result.Prefixes[len(result.Prefixes)-1]
 		}
 	}
 


Test patch: