Repository: etcd-io/etcd (language: go)

Issue:
Inconsistent WAL file permissions on broken files
### Bug report criteria

- [X] This bug report is not security related, security issues should be disclosed privately via [etcd maintainers](mailto:etcd-maintainers@googlegroups.com).
- [X] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [X] You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [X] Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.

### What happened?

When etcd recovers from a broken WAL file, it writes the backup broken file to different permissions (0644) than the rest of the files (0600).  This is a result of a system umask of 0022 at create time that does not appear to get changed/updated later.

Specific permissions that appear should be set:
https://github.com/etcd-io/etcd/blob/main/client/pkg/fileutil/fileutil.go#L31

Call to os.Create:
https://github.com/etcd-io/etcd/blob/main/server/storage/wal/repair.go#L70

A simple call to Chmod may fix the issue:

```
bf.Chmod(PrivateFileMode)
```

### What did you expect to happen?

When etcd recovers from a broken WAL file, it should write the backup broken file to the same permissions as the rest of the files.  

### How can we reproduce it (as minimally and precisely as possible)?

Create a scenario to corrupt a WAL file.  Alternatively, write a unit test that validates the permissions mode on the output from the repair.go code.

### Anything else we need to know?

Occurred in OpenShift 4.16.7
Kubernetes Version: v1.29.7+6abe8a1

### Etcd version (please run commands below)

<details>

```console
# etcd --version
etcd Version: 3.5.13
Git SHA: GitNotFound
Go Version: go1.21.11 (Red Hat 1.21.11-1.el9_4) X:strictfipsruntime
Go OS/Arch: linux/amd64

# etcdctl version
etcdctl version: 3.5.13
API version: 3.5
```

</details>


### Etcd configuration (command line flags or environment variables)

<details>

# paste your configuration here

</details>


### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

<details>

```console
# etcdctl member list -w table
+------------------+---------+-------------------------------+----------------------------+----------------------------+------------+
|        ID        | STATUS  |             NAME              |         PEER ADDRS         |        CLIENT ADDRS        | IS LEARNER |
+------------------+---------+-------------------------------+----------------------------+----------------------------+------------+
| 9af34339d56d737e | started | master1.example.com | https://192.168.10.91:2380 | https://192.168.10.91:2379 |      false |
| d10b5e4d21cb8025 | started | master0.example.com | https://192.168.10.90:2380 | https://192.168.10.90:2379 |      false |
| d51bdd1dfc1b1e42 | started | master2.example.com | https://192.168.10.92:2380 | https://192.168.10.92:2379 |      false |
+------------------+---------+-------------------------------+----------------------------+----------------------------+------------+


# etcdctl endpoint status -w table
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.10.90:2379 | d10b5e4d21cb8025 |  3.5.13 |  566 MB |     false |      false |       147 |  525018061 |          525018061 |        |
| https://192.168.10.91:2379 | 9af34339d56d737e |  3.5.13 |  432 MB |      true |      false |       147 |  525018062 |          525018062 |        |
| https://192.168.10.92:2379 | d51bdd1dfc1b1e42 |  3.5.13 |  601 MB |     false |      false |       147 |  525018063 |          525018063 |        |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
```

</details>


### Relevant log output

```Shell
[root@master2 ~]# ls -l /var/lib/etcd/member/wal/*
-rw-r--r--. 1 root root 64000000 Jun 15 20:14 /var/lib/etcd/member/wal/0000000000007d29-0000000012e90319.wal.broken
-rw-------. 1 root root 64016512 Sep 10 16:28 /var/lib/etcd/member/wal/000000000000cb6c-000000001f48458d.wal
-rw-------. 1 root root 64001752 Sep 10 16:34 /var/lib/etcd/member/wal/000000000000cb6d-000000001f487697.wal
-rw-------. 1 root root 64002720 Sep 10 16:41 /var/lib/etcd/member/wal/000000000000cb6e-000000001f48a32b.wal
-rw-------. 1 root root 64036568 Sep 10 16:47 /var/lib/etcd/member/wal/000000000000cb6f-000000001f48d276.wal
-rw-------. 1 root root 64015152 Sep 10 16:53 /var/lib/etcd/member/wal/000000000000cb70-000000001f490208.wal
-rw-------. 1 root root 64000000 Sep 10 16:53 /var/lib/etcd/member/wal/000000000000cb71-000000001f493093.wal
-rw-------. 1 root root 64000000 Sep 10 16:53 /var/lib/etcd/member/wal/1.tmp

[root@master1 ~]# ls -l /var/lib/etcd/member/wal/*
-rw-------. 1 root root 64000000 Sep 10 16:47 /var/lib/etcd/member/wal/0.tmp
-rw-r--r--. 1 root root 64000000 Jul 15 11:51 /var/lib/etcd/member/wal/0000000000001085-000000001695a9e4.wal.broken
-rw-------. 1 root root 64000128 Sep 10 16:28 /var/lib/etcd/member/wal/0000000000004698-000000001f48461f.wal
-rw-------. 1 root root 64007144 Sep 10 16:34 /var/lib/etcd/member/wal/0000000000004699-000000001f48772e.wal
-rw-------. 1 root root 64004480 Sep 10 16:41 /var/lib/etcd/member/wal/000000000000469a-000000001f48a3ab.wal
-rw-------. 1 root root 64025248 Sep 10 16:47 /var/lib/etcd/member/wal/000000000000469b-000000001f48d303.wal
-rw-------. 1 root root 64000000 Sep 10 16:52 /var/lib/etcd/member/wal/000000000000469c-000000001f4902cd.wal

```

This causes this known issue: https://access.redhat.com/solutions/7055147


Discussion / hints:

Fix patch:
diff --git a/server/storage/wal/repair.go b/server/storage/wal/repair.go
index 53734045167..d1a887835da 100644
--- a/server/storage/wal/repair.go
+++ b/server/storage/wal/repair.go
@@ -67,7 +67,7 @@ func Repair(lg *zap.Logger, dirpath string) bool {
 
 		case errors.Is(err, io.ErrUnexpectedEOF):
 			brokenName := f.Name() + ".broken"
-			bf, bferr := os.Create(brokenName)
+			bf, bferr := os.OpenFile(brokenName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, fileutil.PrivateFileMode)
 			if bferr != nil {
 				lg.Warn("failed to create backup file", zap.String("path", brokenName), zap.Error(bferr))
 				return false


Test patch:
diff --git a/server/storage/wal/repair_test.go b/server/storage/wal/repair_test.go
index b1fd9d25d39..38e3641bd7b 100644
--- a/server/storage/wal/repair_test.go
+++ b/server/storage/wal/repair_test.go
@@ -18,12 +18,14 @@ import (
 	"fmt"
 	"io"
 	"os"
+	"path/filepath"
 	"testing"
 
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 	"go.uber.org/zap/zaptest"
 
+	"go.etcd.io/etcd/client/pkg/v3/fileutil"
 	"go.etcd.io/etcd/server/v3/storage/wal/walpb"
 	"go.etcd.io/raft/v3/raftpb"
 )
@@ -77,6 +79,14 @@ func testRepair(t *testing.T, ents [][]raftpb.Entry, corrupt corruptFunc, expect
 	// repair the wal
 	require.True(t, Repair(lg, p), "'Repair' returned 'false', want 'true'")
 
+	// verify the broken wal has correct permissions
+	bf := filepath.Join(p, filepath.Base(w.tail().Name())+".broken")
+	fi, err := os.Stat(bf)
+	require.NoError(t, err)
+	expectedPerms := fmt.Sprintf("%o", os.FileMode(fileutil.PrivateFileMode))
+	actualPerms := fmt.Sprintf("%o", fi.Mode().Perm())
+	require.Equal(t, expectedPerms, actualPerms, "unexpected file permissions on .broken wal")
+
 	// read it back
 	w, err = Open(lg, p, walpb.Snapshot{})
 	require.NoError(t, err)