Repository: minio/minio (language: go)

Issue:
listing fails to list object with sufficient copies under optimal quorum

<!--- Provide a general summary of the issue in the Title above -->

## Expected Behavior
On a 16 disk setup if we have 14 good copies and optimal quorum is set 
we should be able to list such inconsistent objects correctly.

<!--- If you're describing a bug, tell us what should happen -->
<!--- If you're suggesting a change/improvement, tell us how it should work -->

## Current Behavior
Listing is sporadic and inconsistent i.e lists the versions and does not list them sporadically.
<!--- If describing a bug, tell us what happens instead of the expected behavior -->
<!--- If suggesting a change/improvement, explain the difference from current behavior -->

## Possible Solution
Looks like the bug is in mergeXLV2Versions that returns inconsistent results.
<!--- Not obligatory, but suggest a fix/reason for the bug, -->
<!--- or ideas how to implement the addition or change -->

## Steps to Reproduce (for bugs)

Run Latest MinIO and create a single version first
```
~ minio server /tmp/xl{1...16}
~ mc mb myminio/testbucket/ -l (to enable versioning)
~ mc cp /etc/hosts myminio/testbucket/
```

Change the permissions for two disks.
```
~ sudo chown -R root:root /tmp/xl1
~ sudo chown -R root:root /tmp/xl11
~ mc rm myminio/testbucket/hosts (creates a delete marker)
Creating delete marker `myminio/testbucket/hosts` (versionId=afc0611a-63e0-403b-8078-bf21b9111a2b).
```

Change the permissions back for the disks. Now restart the server and set
 `MINIO_API_LIST_QUORUM=optimal` 

Then subsequently `hosts` file gets listed and not listed sporadically. 
```
~ mc ls -r myminio/testbucket/

~ mc ls myminio/testbucket/ -r
[2022-03-26 22:35:22 PDT]   303B STANDARD hosts
```

To avoid this problem one must always set `strict` quorum.

PS: This happens with ListObjectVersions as well
```
~ mc ls myminio/testbucket/ -r --versions
[2022-03-26 22:40:36 PDT]     0B STANDARD afc0611a-63e0-403b-8078-bf21b9111a2b v2 DEL hosts
[2022-03-26 22:35:22 PDT]   303B STANDARD 12468eb5-e602-4ab5-81de-7aa9a5220a4e v1 PUT hosts
```

```
~ mc ls myminio/testbucket/ -r --versions
[2022-03-26 22:35:22 PDT]   303B STANDARD 12468eb5-e602-4ab5-81de-7aa9a5220a4e v1 PUT hosts
```

## Context
Unexpected listing behavior 

<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->

## Regression
Not sure.
<!-- Is this issue a regression? (Yes / No) -->
<!-- If Yes, optionally please include minio version or commit id or PR# that caused this regression, if you have these details. -->

## Your Environment
Latest master, versioning etc.

[inspect.26645afb.zip](https://github.com/minio/minio/files/8357041/inspect.26645afb.zip)



Discussion / hints:
Reprodued it with `MINIO_API_LIST_QUORUM=optimal`
```
ashish@ashish:~/code/go/src/github/minio/mc$ ./mc ls myminio/testbucket/hosts -r 
ashish@ashish:~/code/go/src/github/minio/mc$ ./mc ls myminio/testbucket/hosts -r 
[2022-03-27 11:57:58 IST]   423B STANDARD hosts
ashish@ashish:~/code/go/src/github/minio/mc$ ./mc ls myminio/testbucket/hosts -r --versions
[2022-03-27 11:57:58 IST]   423B STANDARD b81f56ce-10ec-48c9-8378-3dde85993885 v1 PUT hosts
ashish@ashish:~/code/go/src/github/minio/mc$ ./mc ls myminio/testbucket/hosts -r --versions
[2022-03-27 11:59:37 IST]     0B STANDARD dbd6abac-fc40-4930-ac6e-48d16b133118 v2 DEL hosts
[2022-03-27 11:57:58 IST]   423B STANDARD b81f56ce-10ec-48c9-8378-3dde85993885 v1 PUT hosts
```

The listing were consistent when `MINIO_API_LIST_QUORUM=strict`
```
ashish@ashish:~/code/go/src/github/minio/mc$ ./mc ls myminio/testbucket/hosts -r 
ashish@ashish:~/code/go/src/github/minio/mc$ ./mc ls myminio/testbucket/hosts -r 
ashish@ashish:~/code/go/src/github/minio/mc$ ./mc ls myminio/testbucket/hosts -r --versions
[2022-03-27 11:59:37 IST]     0B STANDARD dbd6abac-fc40-4930-ac6e-48d16b133118 v2 DEL hosts
[2022-03-27 11:57:58 IST]   423B STANDARD b81f56ce-10ec-48c9-8378-3dde85993885 v1 PUT hosts
ashish@ashish:~/code/go/src/github/minio/mc$ ./mc ls myminio/testbucket/hosts -r --versions
[2022-03-27 11:59:37 IST]     0B STANDARD dbd6abac-fc40-4930-ac6e-48d16b133118 v2 DEL hosts
[2022-03-27 11:57:58 IST]   423B STANDARD b81f56ce-10ec-48c9-8378-3dde85993885 v1 PUT hosts
ashish@ashish:~/code/go/src/github/minio/mc$ ./mc ls myminio/testbucket/hosts -r --versions
[2022-03-27 11:59:37 IST]     0B STANDARD dbd6abac-fc40-4930-ac6e-48d16b133118 v2 DEL hosts
[2022-03-27 11:57:58 IST]   423B STANDARD b81f56ce-10ec-48c9-8378-3dde85993885 v1 PUT hosts

```

@harshavardhana This is by design. You cannot do consistent listing, unless you ask all disks - that is a matter of math. 

Using anything other than strict trades a level of consistency on non-synchronized objects.

In #12728 you switched from excluding disks that were healing to using any random disks. This makes the issue much worse, so I propose you revert that.

Asking less than all disks requires that objects are returned from `n-1` disks, so if both your broken disks are included it will not be listed.  You can tweak it to:

```
listingQuorum := o.AskDisks - 1
if o.AskDisks > 3 {
	listingQuorum := o.AskDisks/2
}
```

... but it will just make more non-quorum objects be returned.
> @harshavardhana This is by design. You cannot do consistent listing, unless you ask all disks - that is a matter of math. 
> 
> Using anything other than strict trades a level of consistency on non-synchronized objects.
> 
> In #12728 you switched from excluding disks that were healing to using any random disks. This makes the issue much worse, so I propose you revert that.
> 
> Asking less than all disks requires that objects are returned from `n-1` disks, so if both your broken disks are included it will not be listed.  You can tweak it to:
> 
> ```
> listingQuorum := o.AskDisks - 1
> if o.AskDisks > 3 {
> 	listingQuorum := o.AskDisks/2
> }
> ```
> 
> ... but it will just make more non-quorum objects be returned.

But the problem is not that we have super inconsistent objects - the issue is that we have good copies that satisfy what we want here and that is not getting listed with expected quorum.

i.e 14 good copies but we randomly return the one with 2

How is that logically correct?  @klauspost  - for me this looks like an implementation issue in mergeXLV2 versions

If you see the reproducer you will understand. That's why I assigned it to you for feedback.

Lack of quorum we all understand that it won't work but it is incorrect here to return the objects that have only 2 satisfactory entries.
> > @harshavardhana This is by design. You cannot do consistent listing, unless you ask all disks - that is a matter of math.
> > Using anything other than strict trades a level of consistency on non-synchronized objects.
> > In #12728 you switched from excluding disks that were healing to using any random disks. This makes the issue much worse, so I propose you revert that.
> > Asking less than all disks requires that objects are returned from `n-1` disks, so if both your broken disks are included it will not be listed.  You can tweak it to:
> > ```
> > listingQuorum := o.AskDisks - 1
> > if o.AskDisks > 3 {
> > 	listingQuorum := o.AskDisks/2
> > }
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > ... but it will just make more non-quorum objects be returned.
> 
> But the problem is not that we have super inconsistent objects - the issue is that we have good copies that satisfy what we want here and that is not getting listed with expected quorum.
> 
> i.e 14 good copies but we randomly return the one with 2
> 
> How is that logically correct? @klauspost - for me this looks like an implementation issue in mergeXLV2 versions
> 
> If you see the reproducer you will understand. That's why I assigned it to you for feedback.
> 
> Lack of quorum we all understand that it won't work but it is incorrect here to return the objects that have only 2 satisfactory entries.

The main issue here is that we have to choose maximum occurrences of any object instead of returning least occurrences regardless of the chosen quorum requirement.

This is the fundamental issue in the mergeXLV2 implementation 
@harshavardhana Your inspect files mismatch the case you describe. You describe that you are missing delete markers on 2 disks, but the inspect you post shows the original object missing on 2 disks, but the delete marker on all.

If your delete marker is missing on 2 disks and you are "only" asking 3, it will lead to inconsistent listings. Here is a test I set up to verify that the correct output is being generated:

```Go
func Test_mergeXLV2Versions2(t *testing.T) {
	vDelMarker := xlMetaV2ShallowVersion{header: xlMetaV2VersionHeader{
		VersionID: [16]byte{2},
		ModTime:   1500,
		Signature: [4]byte{5, 6, 7, 8},
		Type:      DeleteType,
		Flags:     0,
	}}
	vDelMarker.meta, _ = base64.StdEncoding.DecodeString("gqRUeXBlAqZEZWxPYmqDoklExBCvwGEaY+BAO4B4vyG5ERorpU1UaW1l0xbgJlsWE9IHp01ldGFTeXOA")

	vObj := xlMetaV2ShallowVersion{header: xlMetaV2VersionHeader{
		VersionID: [16]byte{1},
		ModTime:   1000,
		Signature: [4]byte{1, 2, 3, 4},
		Type:      ObjectType,
		Flags:     xlFlagUsesDataDir | xlFlagInlineData,
	}}
	vObj.meta, _ = base64.StdEncoding.DecodeString("gqRUeXBlAaVWMk9iat4AEaJJRMQQEkaOteYCSrWB3nqppSIKTqRERGlyxBAO8fXSJ5RI+YEtsp8KneVVpkVjQWxnbwGjRWNNDKNFY04Ep0VjQlNpemXSABAAAKdFY0luZGV4BaZFY0Rpc3TcABAFBgcICQoLDA0ODxABAgMEqENTdW1BbGdvAahQYXJ0TnVtc5EBqVBhcnRFVGFnc8CpUGFydFNpemVzkdEBL6pQYXJ0QVNpemVzkdEBL6RTaXpl0QEvpU1UaW1l0xbgJhIa6ABvp01ldGFTeXOBvHgtbWluaW8taW50ZXJuYWwtaW5saW5lLWRhdGHEBHRydWWnTWV0YVVzcoKsY29udGVudC10eXBluGFwcGxpY2F0aW9uL29jdGV0LXN0cmVhbaRldGFn2SBlYTIxMDE2MmVlYjRhZGMzMWZmOTg0Y2I3NDRkNmFmNg==")

	testCases := []struct {
		name        string
		input       [][]xlMetaV2ShallowVersion
		quorum      int
		reqVersions int
		want        []xlMetaV2ShallowVersion
	}{
		{
			name: "obj-on-one",
			input: [][]xlMetaV2ShallowVersion{
				0: {vDelMarker, vObj}, // disk 0
				1: {vDelMarker},       // disk 1
				2: {vDelMarker},       // disk 2
			},
			quorum:      2,
			reqVersions: 0,
			want:        []xlMetaV2ShallowVersion{vDelMarker},
		},
		{
			name: "obj-on-two",
			input: [][]xlMetaV2ShallowVersion{
				0: {vDelMarker, vObj}, // disk 0
				1: {vDelMarker, vObj}, // disk 1
				2: {vDelMarker},       // disk 2
			},
			quorum:      2,
			reqVersions: 0,
			want:        []xlMetaV2ShallowVersion{vDelMarker, vObj},
		},
		{
			name: "obj-on-all",
			input: [][]xlMetaV2ShallowVersion{
				0: {vDelMarker, vObj}, // disk 0
				1: {vDelMarker, vObj}, // disk 1
				2: {vDelMarker, vObj}, // disk 2
			},
			quorum:      2,
			reqVersions: 0,
			want:        []xlMetaV2ShallowVersion{vDelMarker, vObj},
		},
		{
			name: "del-on-one",
			input: [][]xlMetaV2ShallowVersion{
				0: {vDelMarker, vObj}, // disk 0
				1: {vObj},             // disk 1
				2: {vObj},             // disk 2
			},
			quorum:      2,
			reqVersions: 0,
			want:        []xlMetaV2ShallowVersion{vObj},
		},
		{
			name: "del-on-two",
			input: [][]xlMetaV2ShallowVersion{
				0: {vDelMarker, vObj}, // disk 0
				1: {vDelMarker, vObj}, // disk 1
				2: {vObj},             // disk 2
			},
			quorum:      2,
			reqVersions: 0,
			want:        []xlMetaV2ShallowVersion{vDelMarker, vObj},
		},
	}
	for _, test := range testCases {
		t.Run(test.name, func(t *testing.T) {
			// Run multiple times, shuffling the input order.
			for i := int64(0); i < 50; i++ {
				t.Run(fmt.Sprint(i), func(t *testing.T) {
					rng := rand.New(rand.NewSource(i))
					rng.Shuffle(len(test.input), func(i, j int) {
						test.input[i], test.input[j] = test.input[j], test.input[i]
					})
					got := mergeXLV2Versions(test.quorum, true, 0, test.input...)
					if !reflect.DeepEqual(test.want, got) {
						t.Errorf("want %v != got %v", test.want, got)
					}
				})
			}
		})
	}
}
```

This test should be easy enough to modify to show what the fundamental issue is.
[inspect.1e6eda48.zip](https://github.com/minio/minio/files/8392002/inspect.1e6eda48.zip)

Attaching the correct inspect file
Here we go when the input list has 8 copies sent however with only 2 inconsistencies the 2 inconsistent entry is choosen instead of '6' 

```
+func Test_mergeXLV2Versions2(t *testing.T) {
+       vDelMarker := xlMetaV2ShallowVersion{header: xlMetaV2VersionHeader{
+               VersionID: [16]byte{2},
+               ModTime:   1500,
+               Signature: [4]byte{5, 6, 7, 8},
+               Type:      DeleteType,
+               Flags:     0,
+       }}
+       vDelMarker.meta, _ = base64.StdEncoding.DecodeString("gqRUeXBlAqZEZWxPYmqDoklExBCvwGEaY+BAO4B4vyG5ERorpU1UaW1l0xbgJlsWE9IHp01ldGFTeXOA")
+
+       vObj := xlMetaV2ShallowVersion{header: xlMetaV2VersionHeader{
+               VersionID: [16]byte{1},
+               ModTime:   1000,
+               Signature: [4]byte{1, 2, 3, 4},
+               Type:      ObjectType,
+               Flags:     xlFlagUsesDataDir | xlFlagInlineData,
+       }}
+       vObj.meta, _ = base64.StdEncoding.DecodeString("gqRUeXBlAaVWMk9iat4AEaJJRMQQEkaOteYCSrWB3nqppSIKTqRERGlyxBAO8fXSJ5RI+YEtsp8KneVVpkVjQWxnbwGjRWNNDKNFY04Ep0VjQlNpemXSABAAAKdFY0luZGV4BaZFY0Rpc3TcABAFBgcICQoLDA0ODxABAgMEqENTdW1BbGdvAahQYXJ0TnVtc5EBqVBhcnRFVGFnc8CpUGFydFNpemVzkdEBL6pQYXJ0QVNpemVzkdEBL6RTaXpl0QEvpU1UaW1l0xbgJhIa6ABvp01ldGFTeXOBvHgtbWluaW8taW50ZXJuYWwtaW5saW5lLWRhdGHEBHRydWWnTWV0YVVzcoKsY29udGVudC10eXBluGFwcGxpY2F0aW9uL29jdGV0LXN0cmVhbaRldGFn2SBlYTIxMDE2MmVlYjRhZGMzMWZmOTg0Y2I3NDRkNmFmNg==")
+
+       testCases := []struct {
+               name        string
+               input       [][]xlMetaV2ShallowVersion
+               quorum      int
+               reqVersions int
+               want        []xlMetaV2ShallowVersion
+       }{
+               {
+                       name: "del-on-two-16stripe",
+                       input: [][]xlMetaV2ShallowVersion{
+                               0:  {vObj},             // disk 0
+                               1:  {vDelMarker, vObj}, // disk 1
+                               2:  {vDelMarker, vObj}, // disk 2
+                               3:  {vDelMarker, vObj}, // disk 3
+                               4:  {vDelMarker, vObj}, // disk 4
+                               5:  {vDelMarker, vObj}, // disk 5
+                               6:  {vDelMarker, vObj}, // disk 6
+                               7:  {vDelMarker, vObj}, // disk 7
+                               8:  {vDelMarker, vObj}, // disk 8
+                               9:  {vDelMarker, vObj}, // disk 9
+                               10: {vObj},             // disk 10
+                               11: {vDelMarker, vObj}, // disk 11
+                               12: {vDelMarker, vObj}, // disk 12
+                               13: {vDelMarker, vObj}, // disk 13
+                               14: {vDelMarker, vObj}, // disk 14
+                               15: {vDelMarker, vObj}, // disk 15
+                       },
+                       quorum:      7,
+                       reqVersions: 0,
+                       want:        []xlMetaV2ShallowVersion{vDelMarker, vObj},
+               },
+       }
+       for _, test := range testCases {
+               t.Run(test.name, func(t *testing.T) {
+                       // Run multiple times, shuffling the input order.
+                       for i := int64(0); i < 50; i++ {
+                               t.Run(fmt.Sprint(i), func(t *testing.T) {
+                                       rng := rand.New(rand.NewSource(i))
+                                       rng.Shuffle(len(test.input), func(i, j int) {
+                                               test.input[i], test.input[j] = test.input[j], test.input[i]
+                                       })
+                                       got := mergeXLV2Versions(test.quorum, true, 0, test.input[:test.quorum+1]...)
+                                       if !reflect.DeepEqual(test.want, got) {
+                                               t.Errorf("want %v != got %v", test.want, got)
+                                       }
+                               })
+                       }
+               })
+       }
+}
```

I am not saying optimal should work like 'strict' - what I am saying is the the mergeXLV2Versions should return '6' consistent copies instead of '2' consistent ones - when input has 8 entries.

```
--- FAIL: Test_mergeXLV2Versions2 (0.02s)
    --- FAIL: Test_mergeXLV2Versions2/del-on-two-16stripe (0.02s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/0 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/1 (0.00s)
        --- FAIL: Test_mergeXLV2Versions2/del-on-two-16stripe/2 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/3 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/4 (0.00s)
        --- FAIL: Test_mergeXLV2Versions2/del-on-two-16stripe/5 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/6 (0.00s)
        --- FAIL: Test_mergeXLV2Versions2/del-on-two-16stripe/7 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/8 (0.00s)
        --- FAIL: Test_mergeXLV2Versions2/del-on-two-16stripe/9 (0.00s)
        --- FAIL: Test_mergeXLV2Versions2/del-on-two-16stripe/10 (0.00s)
        --- FAIL: Test_mergeXLV2Versions2/del-on-two-16stripe/11 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/12 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/13 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/14 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/15 (0.00s)
        --- FAIL: Test_mergeXLV2Versions2/del-on-two-16stripe/16 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/17 (0.00s)
        --- FAIL: Test_mergeXLV2Versions2/del-on-two-16stripe/18 (0.00s)
        --- FAIL: Test_mergeXLV2Versions2/del-on-two-16stripe/19 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/20 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/21 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/22 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/23 (0.00s)
        --- FAIL: Test_mergeXLV2Versions2/del-on-two-16stripe/24 (0.01s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/25 (0.00s)
        --- FAIL: Test_mergeXLV2Versions2/del-on-two-16stripe/26 (0.01s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/27 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/28 (0.00s)
        --- FAIL: Test_mergeXLV2Versions2/del-on-two-16stripe/29 (0.00s)
        --- FAIL: Test_mergeXLV2Versions2/del-on-two-16stripe/30 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/31 (0.00s)
        --- FAIL: Test_mergeXLV2Versions2/del-on-two-16stripe/32 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/33 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/34 (0.00s)
        --- FAIL: Test_mergeXLV2Versions2/del-on-two-16stripe/35 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/36 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/37 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/38 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/39 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/40 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/41 (0.00s)
        --- FAIL: Test_mergeXLV2Versions2/del-on-two-16stripe/42 (0.00s)
        --- FAIL: Test_mergeXLV2Versions2/del-on-two-16stripe/43 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/44 (0.00s)
        --- FAIL: Test_mergeXLV2Versions2/del-on-two-16stripe/45 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/46 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/47 (0.00s)
        --- PASS: Test_mergeXLV2Versions2/del-on-two-16stripe/48 (0.00s)
        --- FAIL: Test_mergeXLV2Versions2/del-on-two-16stripe/49 (0.00s)
FAIL
exit status 1
FAIL	github.com/minio/minio/cmd	0.057s
```

Passing and failing randomly where as this situation should always pass, we do not meet quorum yes but we shouldn't still be returning an incorrect result because 6 good copies is higher in terms of consistencies instead of '2'
@harshavardhana It is not a "majority wins" algorithm, it will try to achieve the requested quorum for each version.

If you want 6 versions to reach quorum, you should send 6, not 7.
> @harshavardhana It is not a "majority wins" algorithm, it will try to achieve the requested quorum for each version.
> 
> If you want 6 versions to reach quorum, you should send 6, not 7.

@klauspost but however it should never choose 2 - it should have chosen 6 and failed with quorum as expected. Why would it send 2 as good copies ? 

The logic is we want 7 copies - the maximally occurring ones are 6 not 2, even when they do not satisfy quorum.

The idea is if the ask is x, we should return maximally occurring value that is close to the 'x' -  it would be a logical error to return minimally occuring.
Don't you see a problem here? 
Side affect of these values end up getting stored in selected.cached.versions - causing confusion at caller layer.
You are asking for a quorum of `7` and giving input:
```
                               0:  {vObj},             // disk 0
                               1:  {vDelMarker, vObj}, // disk 1
                               2:  {vDelMarker, vObj}, // disk 2
                               3:  {vDelMarker, vObj}, // disk 3
                               4:  {vDelMarker, vObj}, // disk 4
                               5:  {vDelMarker, vObj}, // disk 5
                               6:  {vDelMarker, vObj}, // disk 6
                               7:  {vObj}, // disk 7
```

`vObj` has quorum, but `vDelMarker` does not. Therefore we return `vObj`. The logic is the same for all version types.

We can tweak the quorum, but unless you ask all disks it will never be perfect. `vDelMarker` could exist on all of the remaining disks or none. The `n-1` is meant for 2 or 3 disks, but for more `n/2` is probably more appropriate.
@klauspost I had assumed that maximally occurring is what we should care about because that is what make sense to match with provided quorum value.

If we always make sure that we return maximally occurring from this function then we will mostly cover 90% of the cases.

Currently because of the way logic is handled to look for `consistent` boolean by matching top head signature it ends up returning just v2Obj as latest. 
@harshavardhana Using quorum `(n+1)/2`  would look for majority. 

n  | quorum
--|----------
3 | 2
4 | 2
5 | 3
6 | 3
7 | 4
8 | 4
9 | 5
10 | 5
... | ...

I will send a PR with that change, and also only ask non-healing disks.

Fix patch:
diff --git a/cmd/metacache-set.go b/cmd/metacache-set.go
index 258e75975abca..c427bbb787877 100644
--- a/cmd/metacache-set.go
+++ b/cmd/metacache-set.go
@@ -549,7 +549,7 @@ func getListQuorum(quorum string, driveCount int) int {
 	case "reduced":
 		return 2
 	case "strict":
-		return -1
+		return driveCount
 	}
 	// Defaults to (driveCount+1)/2 drives per set, defaults to "optimal" value
 	if driveCount > 0 {
@@ -563,16 +563,19 @@ func (er *erasureObjects) listPath(ctx context.Context, o listPathOptions, resul
 	defer close(results)
 	o.debugf(color.Green("listPath:")+" with options: %#v", o)
 
+	// get non-healing disks for listing
+	disks, _ := er.getOnlineDisksWithHealing()
 	askDisks := getListQuorum(o.AskDisks, er.setDriveCount)
-	listingQuorum := askDisks - 1
-	disks := er.getDisks()
 	var fallbackDisks []StorageAPI
 
 	// Special case: ask all disks if the drive count is 4
-	if askDisks <= 0 || er.setDriveCount == 4 {
-		askDisks = len(disks)                // with 'strict' quorum list on all drives.
-		listingQuorum = (len(disks) + 1) / 2 // keep this such that we can list all objects with different quorum ratio.
+	if er.setDriveCount == 4 || askDisks > len(disks) {
+		askDisks = len(disks) // use all available drives
 	}
+
+	// However many we ask, versions must exist on ~50%
+	listingQuorum := (askDisks + 1) / 2
+
 	if askDisks > 0 && len(disks) > askDisks {
 		rand.Shuffle(len(disks), func(i, j int) {
 			disks[i], disks[j] = disks[j], disks[i]


Test patch:
diff --git a/cmd/xl-storage-format-v2_test.go b/cmd/xl-storage-format-v2_test.go
index 5516dbe0f5417..fb399f33ef893 100644
--- a/cmd/xl-storage-format-v2_test.go
+++ b/cmd/xl-storage-format-v2_test.go
@@ -21,10 +21,13 @@ import (
 	"bufio"
 	"bytes"
 	"compress/gzip"
+	"encoding/base64"
 	"encoding/json"
 	"fmt"
 	"io"
+	"math/rand"
 	"os"
+	"reflect"
 	"sort"
 	"testing"
 	"time"
@@ -825,3 +828,128 @@ func Test_mergeXLV2Versions(t *testing.T) {
 		})
 	}
 }
+
+func Test_mergeXLV2Versions2(t *testing.T) {
+	vDelMarker := xlMetaV2ShallowVersion{header: xlMetaV2VersionHeader{
+		VersionID: [16]byte{2},
+		ModTime:   1500,
+		Signature: [4]byte{5, 6, 7, 8},
+		Type:      DeleteType,
+		Flags:     0,
+	}}
+	vDelMarker.meta, _ = base64.StdEncoding.DecodeString("gqRUeXBlAqZEZWxPYmqDoklExBCvwGEaY+BAO4B4vyG5ERorpU1UaW1l0xbgJlsWE9IHp01ldGFTeXOA")
+
+	vObj := xlMetaV2ShallowVersion{header: xlMetaV2VersionHeader{
+		VersionID: [16]byte{1},
+		ModTime:   1000,
+		Signature: [4]byte{1, 2, 3, 4},
+		Type:      ObjectType,
+		Flags:     xlFlagUsesDataDir | xlFlagInlineData,
+	}}
+	vObj.meta, _ = base64.StdEncoding.DecodeString("gqRUeXBlAaVWMk9iat4AEaJJRMQQEkaOteYCSrWB3nqppSIKTqRERGlyxBAO8fXSJ5RI+YEtsp8KneVVpkVjQWxnbwGjRWNNDKNFY04Ep0VjQlNpemXSABAAAKdFY0luZGV4BaZFY0Rpc3TcABAFBgcICQoLDA0ODxABAgMEqENTdW1BbGdvAahQYXJ0TnVtc5EBqVBhcnRFVGFnc8CpUGFydFNpemVzkdEBL6pQYXJ0QVNpemVzkdEBL6RTaXpl0QEvpU1UaW1l0xbgJhIa6ABvp01ldGFTeXOBvHgtbWluaW8taW50ZXJuYWwtaW5saW5lLWRhdGHEBHRydWWnTWV0YVVzcoKsY29udGVudC10eXBluGFwcGxpY2F0aW9uL29jdGV0LXN0cmVhbaRldGFn2SBlYTIxMDE2MmVlYjRhZGMzMWZmOTg0Y2I3NDRkNmFmNg==")
+
+	testCases := []struct {
+		name        string
+		input       [][]xlMetaV2ShallowVersion
+		quorum      int
+		reqVersions int
+		want        []xlMetaV2ShallowVersion
+	}{
+		{
+			name: "obj-on-one",
+			input: [][]xlMetaV2ShallowVersion{
+				0: {vDelMarker, vObj}, // disk 0
+				1: {vDelMarker},       // disk 1
+				2: {vDelMarker},       // disk 2
+			},
+			quorum:      2,
+			reqVersions: 0,
+			want:        []xlMetaV2ShallowVersion{vDelMarker},
+		},
+		{
+			name: "obj-on-two",
+			input: [][]xlMetaV2ShallowVersion{
+				0: {vDelMarker, vObj}, // disk 0
+				1: {vDelMarker, vObj}, // disk 1
+				2: {vDelMarker},       // disk 2
+			},
+			quorum:      2,
+			reqVersions: 0,
+			want:        []xlMetaV2ShallowVersion{vDelMarker, vObj},
+		},
+		{
+			name: "obj-on-all",
+			input: [][]xlMetaV2ShallowVersion{
+				0: {vDelMarker, vObj}, // disk 0
+				1: {vDelMarker, vObj}, // disk 1
+				2: {vDelMarker, vObj}, // disk 2
+			},
+			quorum:      2,
+			reqVersions: 0,
+			want:        []xlMetaV2ShallowVersion{vDelMarker, vObj},
+		},
+		{
+			name: "del-on-one",
+			input: [][]xlMetaV2ShallowVersion{
+				0: {vDelMarker, vObj}, // disk 0
+				1: {vObj},             // disk 1
+				2: {vObj},             // disk 2
+			},
+			quorum:      2,
+			reqVersions: 0,
+			want:        []xlMetaV2ShallowVersion{vObj},
+		},
+		{
+			name: "del-on-two",
+			input: [][]xlMetaV2ShallowVersion{
+				0: {vDelMarker, vObj}, // disk 0
+				1: {vDelMarker, vObj}, // disk 1
+				2: {vObj},             // disk 2
+			},
+			quorum:      2,
+			reqVersions: 0,
+			want:        []xlMetaV2ShallowVersion{vDelMarker, vObj},
+		},
+		{
+			name: "del-on-two-16stripe",
+			input: [][]xlMetaV2ShallowVersion{
+				0:  {vObj},             // disk 0
+				1:  {vDelMarker, vObj}, // disk 1
+				2:  {vDelMarker, vObj}, // disk 2
+				3:  {vDelMarker, vObj}, // disk 3
+				4:  {vDelMarker, vObj}, // disk 4
+				5:  {vDelMarker, vObj}, // disk 5
+				6:  {vDelMarker, vObj}, // disk 6
+				7:  {vDelMarker, vObj}, // disk 7
+				8:  {vDelMarker, vObj}, // disk 8
+				9:  {vDelMarker, vObj}, // disk 9
+				10: {vObj},             // disk 10
+				11: {vDelMarker, vObj}, // disk 11
+				12: {vDelMarker, vObj}, // disk 12
+				13: {vDelMarker, vObj}, // disk 13
+				14: {vDelMarker, vObj}, // disk 14
+				15: {vDelMarker, vObj}, // disk 15
+			},
+			quorum:      7,
+			reqVersions: 0,
+			want:        []xlMetaV2ShallowVersion{vDelMarker, vObj},
+		},
+	}
+	for _, test := range testCases {
+		t.Run(test.name, func(t *testing.T) {
+			// Run multiple times, shuffling the input order.
+			for i := int64(0); i < 50; i++ {
+				t.Run(fmt.Sprint(i), func(t *testing.T) {
+					rng := rand.New(rand.NewSource(i))
+					rng.Shuffle(len(test.input), func(i, j int) {
+						test.input[i], test.input[j] = test.input[j], test.input[i]
+					})
+					got := mergeXLV2Versions(test.quorum, true, 0, test.input...)
+					if !reflect.DeepEqual(test.want, got) {
+						t.Errorf("want %v != got %v", test.want, got)
+					}
+				})
+			}
+		})
+	}
+}