Repository: minio/minio (language: go)

Issue:
CreateBucket during server init causes bucket deletion (causing huge data losses) due to race condition at server initialization
I was able to finally **reproduce** the problem I've been having for the past weeks which caused the **deletion of multiple buckets on every drive and replica**.
During the server initialization it is possible to create buckets even if MinIO hasn't yet figured out which buckets already exist.

I've got a replicated minio with 4 nodes and this is how I reproduced the issue:

**Step 1**
Shutdown the MinIO instance(s).

**Step 2**
Run the following script, which sends bucket creation HTTP requests to all the MinIO replicas:

```
#!/bin/bash

while true 
do
    for i in $(seq 1 4)
    do
        curl -k "http://minio$i:9001/api/v1/buckets" \
          -H "authority: minio$i:9001" \
          -H 'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="98", "Google Chrome";v="98"' \
          -H 'sec-ch-ua-mobile: ?0' \
          -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36' \
          -H 'sec-ch-ua-platform: "Linux"' \
          -H 'content-type: application/json' \
          -H 'accept: */*' \
          -H "origin: http://minio$i:9001" \
          -H 'sec-fetch-site: same-origin' \
          -H 'sec-fetch-mode: cors' \
          -H 'sec-fetch-dest: empty' \
          -H "referer: http://minio$i:9001/add-bucket" \
          -H 'accept-language: en-US,en;q=0.9' \
          -H 'cookie: minio_cookie=xxxxxxxxxxxxxxxx; token=xxxxxxxxxxxxxxxxxxx' \
          --data-raw '{"name":"bucketnamehere","versioning":false,"locking":false}' \
          --compressed &

        sleep 0.01
        echo $i
    done
done
```

**Step 3**
Start the instances again (while keeping the script running in background)

---

After all of these steps I was able to **recreate** a bucket that existed already, and as a consequence, few minutes later that bucket was empty as it was been recreated even if it shouldn't have been, both on the filesystem and on MinIO's UI.

Of course my program does not run that script, but it does something similar by creating buckets at server's startup.

Also this problem shouldn't be dependent on the number of available nodes and/or drives as it looks like a problem directly correlated with the ObjectAPI object: the function initServer is called before the buckets metadata are fully loaded, and the HTTP API is already available to be used during this "unready" period.

I've reproduced it on two versions (output of "minio --version"): the latest one and one dated 27 December 2021.



Discussion / hints:
Thanks for the reproducer @dottgonzo will check and get back. 
> I've reproduced it on two versions (output of "minio --version"): the latest one and one dated 27 December 2021.

Have you reproduced this with latest release? @dottgonzo 
I think this issue is fixed with 

```
commit b7c5e45fff935994791f89237a4891185c55b91f (tag: RELEASE.2022-01-08T03-11-54Z)
Author: Harshavardhana <harsha@minio.io>
Date:   Fri Jan 7 19:11:54 2022 -0800

    heal: isObjectDangling should return false when it cannot decide (#14053)
    
    In a multi-pool setup when disks are coming up, or in a single pool
    setup let's say with 100's of erasure sets with a slow network.
    
    It's possible when healing is attempted on `.minio.sys/config`
    folder, it can lead to healing unexpectedly deleting some policy
    files as dangling due to a mistake in understanding when `isObjectDangling`
    is considered to be 'true'.
    
    This issue happened in commit 30135eed86f470407123775f3c11ecd3e991337b
    when we assumed the validMeta with empty ErasureInfo is considered
    to be fully dangling. This implementation issue gets exposed when
    the server is starting up.
    
    This is most easily seen with multiple-pool setups because of the
    disconnected fashion pools that come up. The decision to purge the
    object as dangling is taken incorrectly prior to the correct state
    being achieved on each pool, when the corresponding drive let's say
    returns 'errDiskNotFound', a 'delete' is triggered. At this point,
    the 'drive' comes online because this is part of the startup sequence
    as drives can come online lazily.
    
    This kind of situation exists because we allow (totalDisks/2) number
    of drives to be online when the server is being restarted.
    
    Implementation made an incorrect assumption here leading to policies
    getting deleted.
    
    Added tests to capture the implementation requirements.
```


Fix patch:
diff --git a/cmd/erasure-server-pool.go b/cmd/erasure-server-pool.go
index c128aa55665d3..97b5516a868da 100644
--- a/cmd/erasure-server-pool.go
+++ b/cmd/erasure-server-pool.go
@@ -671,7 +671,9 @@ func (z *erasureServerPools) MakeBucketWithLocation(ctx context.Context, bucket
 		if err != nil {
 			if _, ok := err.(BucketExists); !ok {
 				// Delete created buckets, ignoring errors.
-				z.DeleteBucket(context.Background(), bucket, DeleteBucketOptions{Force: true, NoRecreate: true})
+				z.DeleteBucket(context.Background(), bucket, DeleteBucketOptions{
+					Force: false, NoRecreate: true,
+				})
 			}
 			return err
 		}


Test patch: