Repository: minio/minio (language: go)

Issue:
[arm32] Broken Minio instance after "admin bucket remote add" commande
## Broken Minio instance after "admin bucket remote add" commande

After sendind a "admin bucket remote add" command which fail, the service remain broken even after restarting Minio server.

## Expected Behavior

Try to setup a bucket replication with the blog : [https://blog.min.io/active-active-replication/](https://blog.min.io/active-active-replication/) with an Rasberry Pi 4.

I ran in a reliability issue. The minio service is broken. Data can not be acces any more.

If this issue is a 32bits host problem. The REST call "admin bucket remote add" should not accept on 32bits server.

## Current Behavior

Just try the post [https://blog.min.io/active-active-replication/](https://blog.min.io/active-active-replication/) to understand "Bucket Replication".

## Steps to Reproduce (for bugs)

    $ mc admin config get store/ site
    site name= region=lab
    $ mc mb store/mirror
    $ mc admin user add store/ repl yyyyyy
    $ mc admin bucket remote add store/mirror https://repl:yyyyyy@xxxxxx/mirror --service replication --region lab
    mc: <ERROR> Unable to configure remote target. Remote service connection error - please check remote service credentials and target bucket (Remote service endpoint or target bucket not available: mirror 
	500 Internal Server Error).

Here is the server log, even after restarting:

      minio[2746]: panic: "HEAD /mirror/": unaligned 64-bit atomic operation
      minio[2746]: goroutine 881568 [running]:
      minio[2746]: runtime/debug.Stack()
      minio[2746]:         runtime/debug/stack.go:24 +0x78
      minio[2746]: github.com/minio/minio/cmd.setCriticalErrorHandler.func1.1(0x13a2d900, {0x4ab1cec, 0x7914320})
      minio[2746]:         github.com/minio/minio/cmd/generic-handlers.go:469 +0x50
      minio[2746]: panic({0x22eb6f0, 0x4a6f750})
      minio[2746]:         runtime/panic.go:1038 +0x23c
      minio[2746]: runtime/internal/atomic.panicUnaligned()
      minio[2746]:         runtime/internal/atomic/unaligned.go:8 +0x24
      minio[2746]: runtime/internal/atomic.Xadd64(0x7914024, 0x1)
      minio[2746]:         runtime/internal/atomic/atomic_arm.s:256 +0x14
      minio[2746]: github.com/minio/minio/cmd.(*HTTPStats).incS3RequestsIncoming(...)
      minio[2746]:         github.com/minio/minio/cmd/http-stats.go:159
      minio[2746]: github.com/minio/minio/cmd.maxClients.func1({0x4aab088, 0x7979c80}, 0x13a2da00)
      minio[2746]:         github.com/minio/minio/cmd/handler-api.go:247 +0x58
      minio[2746]: net/http.HandlerFunc.ServeHTTP(...)
      minio[2746]:         net/http/server.go:2047
      minio[2746]: github.com/minio/minio/cmd.collectAPIStats.func1({0x4aab088, 0x7979bc0}, 0x13a2da00)
      minio[2746]:         github.com/minio/minio/cmd/handler-utils.go:387 +0xd0
      minio[2746]: net/http.HandlerFunc.ServeHTTP(0x8108cb0, {0x4aab088, 0x7979bc0}, 0x13a2da00)
      minio[2746]:         net/http/server.go:2047 +0x34
      minio[2746]: github.com/minio/minio/cmd.setBucketForwardingHandler.func1({0x4aab088, 0x7979bc0}, 0x13a2da00)
      minio[2746]:         github.com/minio/minio/cmd/generic-handlers.go:377 +0x214
      minio[2746]: net/http.HandlerFunc.ServeHTTP(0x8df2b70, {0x4aab088, 0x7979bc0}, 0x13a2da00)
      minio[2746]:         net/http/server.go:2047 +0x34
      minio[2746]: github.com/minio/minio/cmd.addCustomHeaders.func1({0x4aab050, 0x8df2c00}, 0x13a2da00)
      minio[2746]:         github.com/minio/minio/cmd/generic-handlers.go:452 +0x420
      minio[2746]: net/http.HandlerFunc.ServeHTTP(0x8df2b80, {0x4aab050, 0x8df2c00}, 0x13a2da00)
      minio[2746]:         net/http/server.go:2047 +0x34
      minio[2746]: github.com/minio/minio/cmd.setRequestValidityHandler.func1({0x4aab050, 0x8df2c00}, 0x13a2da00)
      minio[2746]:         github.com/minio/minio/cmd/generic-handlers.go:365 +0x8b0
      minio[2746]: net/http.HandlerFunc.ServeHTTP(0x8df2b90, {0x4aab050, 0x8df2c00}, 0x13a2da00)
      minio[2746]:         net/http/server.go:2047 +0x34
      minio[2746]: github.com/minio/minio/cmd.setHTTPStatsHandler.func1({0x4ab1cec, 0x7914320}, 0x13a2da00)
      minio[2746]:         github.com/minio/minio/cmd/generic-handlers.go:279 +0x110
      minio[2746]: net/http.HandlerFunc.ServeHTTP(0x8df2ba0, {0x4ab1cec, 0x7914320}, 0x13a2da00)
      minio[2746]:         net/http/server.go:2047 +0x34
      minio[2746]: github.com/minio/minio/cmd.setRequestLimitHandler.func1({0x4ab1cec, 0x7914320}, 0x13a2da00)
      minio[2746]:         github.com/minio/minio/cmd/generic-handlers.go:111 +0x2e8
      minio[2746]: net/http.HandlerFunc.ServeHTTP(0x8df2bb0, {0x4ab1cec, 0x7914320}, 0x13a2da00)
      minio[2746]:         net/http/server.go:2047 +0x34
      minio[2746]: github.com/minio/minio/cmd.setCrossDomainPolicy.func1({0x4ab1cec, 0x7914320}, 0x13a2da00)
      minio[2746]:         github.com/minio/minio/cmd/crossdomain-xml-handler.go:43 +0x84
      minio[2746]: net/http.HandlerFunc.ServeHTTP(0x8df2bc0, {0x4ab1cec, 0x7914320}, 0x13a2da00)
      minio[2746]:         net/http/server.go:2047 +0x34
      minio[2746]: github.com/minio/minio/cmd.setBrowserRedirectHandler.func1({0x4ab1cec, 0x7914320}, 0x13a2da00)
      minio[2746]:         github.com/minio/minio/cmd/generic-handlers.go:140 +0x160
      minio[2746]: net/http.HandlerFunc.ServeHTTP(0x8df2bd0, {0x4ab1cec, 0x7914320}, 0x13a2da00)
      minio[2746]:         net/http/server.go:2047 +0x34
      minio[2746]: github.com/minio/minio/cmd.setAuthHandler.func1({0x4ab1cec, 0x7914320}, 0x13a2da00)
      minio[2746]:         github.com/minio/minio/cmd/auth-handler.go:498 +0x37c
      minio[2746]: net/http.HandlerFunc.ServeHTTP(0x8df2be0, {0x4ab1cec, 0x7914320}, 0x13a2da00)
      minio[2746]:         net/http/server.go:2047 +0x34
      minio[2746]: github.com/gorilla/mux.(*Router).ServeHTTP(0x7517200, {0x4ab1cec, 0x7914320}, 0x13a2d900)
      minio[2746]:         github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1dc
      minio[2746]: github.com/rs/cors.(*Cors).Handler.func1({0x4ab1cec, 0x7914320}, 0x13a2d900)
      minio[2746]:         github.com/rs/cors@v1.7.0/cors.go:219 +0x224
      minio[2746]: net/http.HandlerFunc.ServeHTTP(0x8109810, {0x4ab1cec, 0x7914320}, 0x13a2d900)
      minio[2746]:         net/http/server.go:2047 +0x34
      minio[2746]: github.com/minio/minio/cmd.setCriticalErrorHandler.func1({0x4ab1cec, 0x7914320}, 0x13a2d900)
      minio[2746]:         github.com/minio/minio/cmd/generic-handlers.go:476 +0x6c
      minio[2746]: net/http.HandlerFunc.ServeHTTP(0x8109820, {0x4ab1cec, 0x7914320}, 0x13a2d900)
      minio[2746]:         net/http/server.go:2047 +0x34
      minio[2746]: github.com/minio/minio/internal/http.(*Server).Start.func1({0x4ab1cec, 0x7914320}, 0x13a2d900)
      minio[2746]:         github.com/minio/minio/internal/http/server.go:97 +0x1f4
      minio[2746]: net/http.HandlerFunc.ServeHTTP(0x74e6010, {0x4ab1cec, 0x7914320}, 0x13a2d900)
      minio[2746]:         net/http/server.go:2047 +0x34
      minio[2746]: net/http.serverHandler.ServeHTTP({0x76a42c0}, {0x4ab1cec, 0x7914320}, 0x13a2d900)
      minio[2746]:         net/http/server.go:2879 +0x3f0
      minio[2746]: net/http.(*conn).serve(0x7979b00, {0x4ab2a48, 0x74ae048})
      minio[2746]:         net/http/server.go:1930 +0xcc4
      minio[2746]: created by net/http.(*Server).Serve
      minio[2746]:         net/http/server.go:3034 +0x47

After restarting the server, any command doesn't work, broken instance. 

Exemple:

    $ mc ls is3/
    mc: <ERROR> Unable to list folder. We encountered an internal error, please try again.

Log:

      panic: "GET /": unaligned 64-bit atomic operation
      goroutine 30644 [running]:
      runtime/debug.Stack()
              runtime/debug/stack.go:24 +0x78
      github.com/minio/minio/cmd.setCriticalErrorHandler.func1.1(0xa5ae300, {0x4ab1cec, 0x713e460})
              github.com/minio/minio/cmd/generic-handlers.go:469 +0x50
      panic({0x22eb6f0, 0x4a6f750})
              runtime/panic.go:1038 +0x23c
      runtime/internal/atomic.panicUnaligned()
              runtime/internal/atomic/unaligned.go:8 +0x24
      runtime/internal/atomic.Xadd64(0x73b60c4, 0x1)
              runtime/internal/atomic/atomic_arm.s:256 +0x14
      github.com/minio/minio/cmd.(*HTTPStats).incS3RequestsIncoming(...)
              github.com/minio/minio/cmd/http-stats.go:159
      github.com/minio/minio/cmd.maxClients.func1({0x4aab088, 0x16c7e420}, 0xa5ae400)
              github.com/minio/minio/cmd/handler-api.go:247 +0x58
      net/http.HandlerFunc.ServeHTTP(...)
              net/http/server.go:2047
      github.com/minio/minio/cmd.collectAPIStats.func1({0x4aab088, 0x16c7e3c0}, 0xa5ae400)
              github.com/minio/minio/cmd/handler-utils.go:387 +0xd0
      net/http.HandlerFunc.ServeHTTP(0x75adf30, {0x4aab088, 0x16c7e3c0}, 0xa5ae400)
              net/http/server.go:2047 +0x34
      github.com/minio/minio/cmd.setBucketForwardingHandler.func1({0x4aab088, 0x16c7e3c0}, 0xa5ae400)
              github.com/minio/minio/cmd/generic-handlers.go:377 +0x214
      net/http.HandlerFunc.ServeHTTP(0x8aae9f0, {0x4aab088, 0x16c7e3c0}, 0xa5ae400)
              net/http/server.go:2047 +0x34
      github.com/minio/minio/cmd.addCustomHeaders.func1({0x4aab050, 0x8aaea80}, 0xa5ae400)
              github.com/minio/minio/cmd/generic-handlers.go:452 +0x420
      net/http.HandlerFunc.ServeHTTP(0x8aaea00, {0x4aab050, 0x8aaea80}, 0xa5ae400)
              net/http/server.go:2047 +0x34
      github.com/minio/minio/cmd.setRequestValidityHandler.func1({0x4aab050, 0x8aaea80}, 0xa5ae400)
              github.com/minio/minio/cmd/generic-handlers.go:365 +0x8b0
      net/http.HandlerFunc.ServeHTTP(0x8aaea10, {0x4aab050, 0x8aaea80}, 0xa5ae400)
              net/http/server.go:2047 +0x34
      github.com/minio/minio/cmd.setHTTPStatsHandler.func1({0x4ab1cec, 0x713e460}, 0xa5ae400)
              github.com/minio/minio/cmd/generic-handlers.go:279 +0x110
      net/http.HandlerFunc.ServeHTTP(0x8aaea20, {0x4ab1cec, 0x713e460}, 0xa5ae400)
              net/http/server.go:2047 +0x34
      github.com/minio/minio/cmd.setRequestLimitHandler.func1({0x4ab1cec, 0x713e460}, 0xa5ae400)
              github.com/minio/minio/cmd/generic-handlers.go:111 +0x2e8
      net/http.HandlerFunc.ServeHTTP(0x8aaea30, {0x4ab1cec, 0x713e460}, 0xa5ae400)
              net/http/server.go:2047 +0x34
      github.com/minio/minio/cmd.setCrossDomainPolicy.func1({0x4ab1cec, 0x713e460}, 0xa5ae400)
              github.com/minio/minio/cmd/crossdomain-xml-handler.go:43 +0x84
      net/http.HandlerFunc.ServeHTTP(0x8aaea40, {0x4ab1cec, 0x713e460}, 0xa5ae400)
              net/http/server.go:2047 +0x34
      github.com/minio/minio/cmd.setBrowserRedirectHandler.func1({0x4ab1cec, 0x713e460}, 0xa5ae400)
              github.com/minio/minio/cmd/generic-handlers.go:140 +0x160
      net/http.HandlerFunc.ServeHTTP(0x8aaea50, {0x4ab1cec, 0x713e460}, 0xa5ae400)
              net/http/server.go:2047 +0x34
      github.com/minio/minio/cmd.setAuthHandler.func1({0x4ab1cec, 0x713e460}, 0xa5ae400)
              github.com/minio/minio/cmd/auth-handler.go:498 +0x37c
      net/http.HandlerFunc.ServeHTTP(0x8aaea60, {0x4ab1cec, 0x713e460}, 0xa5ae400)
              net/http/server.go:2047 +0x34
      github.com/gorilla/mux.(*Router).ServeHTTP(0x70416e0, {0x4ab1cec, 0x713e460}, 0xa5ae300)
              github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1dc
      github.com/rs/cors.(*Cors).Handler.func1({0x4ab1cec, 0x713e460}, 0xa5ae300)
              github.com/rs/cors@v1.7.0/cors.go:219 +0x224
      net/http.HandlerFunc.ServeHTTP(0x7df8020, {0x4ab1cec, 0x713e460}, 0xa5ae300)
              net/http/server.go:2047 +0x34
      github.com/minio/minio/cmd.setCriticalErrorHandler.func1({0x4ab1cec, 0x713e460}, 0xa5ae300)
              github.com/minio/minio/cmd/generic-handlers.go:476 +0x6c
      net/http.HandlerFunc.ServeHTTP(0x7df8030, {0x4ab1cec, 0x713e460}, 0xa5ae300)
              net/http/server.go:2047 +0x34
      github.com/minio/minio/internal/http.(*Server).Start.func1({0x4ab1cec, 0x713e460}, 0xa5ae300)
              github.com/minio/minio/internal/http/server.go:97 +0x1f4
      net/http.HandlerFunc.ServeHTTP(0x7440130, {0x4ab1cec, 0x713e460}, 0xa5ae300)
              net/http/server.go:2047 +0x34
      net/http.serverHandler.ServeHTTP({0x7d94630}, {0x4ab1cec, 0x713e460}, 0xa5ae300)
              net/http/server.go:2879 +0x3f0
      net/http.(*conn).serve(0x16c7e360, {0x4ab2a48, 0x7310648})
              net/http/server.go:1930 +0xcc4
      created by net/http.(*Server).Serve
              net/http/server.go:3034 +0x474


## Context

- Raspberry Pi 4 Model B Rev 1.4 / 8G,
- Raspbian GNU/Linux 11 (bullseye) 
- Lab with a 1To HDD.

## Regression
No

## Your Environment

* Minio version RELEASE.2022-02-16T00-35-27Z
* Server setup and configuration: minio server --anonymous --console-address :9001 --address :9000 /data/minio/disks/1 /data/minio/disks/2 /data/minio/disks/3 /data/minio/disks/4 /data/minio/disks/5
* Operating System : Linux lab17 5.10.92-v7l+ #1514 SMP Mon Jan 17 17:38:03 GMT 2022 armv7l GNU/Linux



Discussion / hints:
Yes this is a 32-bit problem @lionelg3 

Fix patch:
diff --git a/cmd/http-stats.go b/cmd/http-stats.go
index 980a6e828e4c0..9041d4bc964df 100644
--- a/cmd/http-stats.go
+++ b/cmd/http-stats.go
@@ -138,12 +138,13 @@ func (stats *HTTPAPIStats) Load() map[string]int {
 // HTTPStats holds statistics information about
 // HTTP requests made by all clients
 type HTTPStats struct {
+	s3RequestsInQueue       int32 // ref: https://golang.org/pkg/sync/atomic/#pkg-note-BUG
+	_                       int32 // For 64 bits alignment
+	s3RequestsIncoming      uint64
 	rejectedRequestsAuth    uint64
 	rejectedRequestsTime    uint64
 	rejectedRequestsHeader  uint64
 	rejectedRequestsInvalid uint64
-	s3RequestsInQueue       int32
-	s3RequestsIncoming      uint64
 	currentS3Requests       HTTPAPIStats
 	totalS3Requests         HTTPAPIStats
 	totalS3Errors           HTTPAPIStats
diff --git a/cmd/mrf.go b/cmd/mrf.go
index c7df7a20d6607..18fa6d193461d 100644
--- a/cmd/mrf.go
+++ b/cmd/mrf.go
@@ -50,7 +50,8 @@ type setInfo struct {
 // mrfState sncapsulates all the information
 // related to the global background MRF.
 type mrfState struct {
-	ready int32
+	ready int32 // ref: https://golang.org/pkg/sync/atomic/#pkg-note-BUG
+	_     int32 // For 64 bits alignment
 
 	ctx       context.Context
 	objectAPI ObjectLayer


Test patch: