Repository: etcd-io/etcd (language: go)

Issue:
Setup a consistent way to manage go version
### What would you like to be added?

Currently there are multiple places setting the go versions, for examples,
- go.mod files, including both `go` and `toolchain` directives;
- .go-version file;
- environment variable, such as https://github.com/etcd-io/etcd/pull/17734

We should set up a consistent way to manage go version. There should be only one source of truth for the go&toolchain version.

One problem is that if a dependency sets a higher go or toolchain version, then when running scripts/fix.sh will automatically update the go & toolchain version in our go.mod files.  Refer to https://github.com/etcd-io/etcd/pull/17841/files#r1575344566

References:
- https://go.dev/doc/toolchain
- https://github.com/kubernetes/community/blob/master/contributors/devel/development.md#building-kubernetes-using-a-specific-version-of-go

cc @fuweid  @henrybear327  @ivanvc @jmhbnz  @MadhavJivrajani @serathius 

### Why is this needed?

Setup consistent way to manage go version


Discussion / hints:
> One problem is that if a dependency sets a higher go or toolchain version, then when running scripts/fix.sh will automatically update the go & toolchain version in our go.mod files.

We are trying to be more careful about this situation in Kubernetes (ref: https://github.com/kubernetes/kubernetes/issues/123744).

One thing we should keep in mind here also is that if dependabot bumps the `go` directive of the root `go.mod` and we don't reflect that change in the `.go-version` file then it could be problematic for reasons similar to https://github.com/kubernetes/kubernetes/issues/123744, if that happens, we are in a situation where:
- We build with a go version that is in .go-version, because the release GH action uses that as source of truth.
- However, if the `go` directive in `go.mod` is newer, `go` will default the values of `GODEBUG` based on the value of the `go` directive (see https://go.dev/doc/godebug). This will alter the runtime behaviour because the user might expect behaviour similar to whatever the `go` directive is, but the actual behaviour is that of `.go-version`.

In Kubernetes we added a pre-submit check to catch such accidental bumps if they occur: https://github.com/kubernetes/kube-openapi/pull/467 
> In Kubernetes we added a pre-submit check to catch such accidental bumps if they occur: [kubernetes/kube-openapi#467](https://github.com/kubernetes/kube-openapi/pull/467)

Thanks @MadhavJivrajani 

I think we should do similar check for all etcd branches: including `release-3.4`, `release-3.4` and `main`.

The other potential enhancement is to automatically populate the go versions for the `go` and `toolchain` lines in all go.mod (and go.sum) files based on the `.go-version` file.
I have some ideas on how we can go about doing this. Will create a PR and we can discuss there!
Followups:
- Update document: https://github.com/etcd-io/etcd/pull/17876#discussion_r1587360706
- Refactor the ./scripts/fix.sh as mentioned in https://github.com/etcd-io/etcd/pull/17876#discussion_r1593660044
I think we also need to backport the changes to both 3.5 and 3.4
We should follow the same rule to manage go version for all repositories under etcd-io. Note we only require users to depend on a minor version supported by golang team, i.e. **1.21**, but we don't force users to depend on a patch version, i.e. **1.21.10**.
Cross-posting and quoting my comment from https://github.com/etcd-io/etcd/pull/17876#issuecomment-2108783668:

> This implementation is great (referring to #17876) and simplifies what would be tedious if done by hand. We recently had the Go version updated by a new contributor, and I believe it wasn't very clear to him how to update the toolchain in the different go.mod files (refer to: #17975 (comment)). I think we would benefit from documenting this somewhere in the repository.
> A potential improvement regarding documentation would be to document how to update the Go version. In the main branch with this new script, I think it's as easy as updating the value in .go-version and running make fix after 🎉
Discussed during sig-etcd triage meeting, for the next step we just need to edit the following section of https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/dependency_management.md#golang-versions

```diff
- 3. Complete the upgrade locally in your development environment.
+ 3. Complete the upgrade locally in your development environment by editing `.go-version` and running `make fix.
```

Adding good-first-issue label.
@jmhbnz I will take it up

/assign @manthanguptaa 

Fix patch:
diff --git a/Documentation/contributor-guide/dependency_management.md b/Documentation/contributor-guide/dependency_management.md
index 374b3e8bdf9..8b022b424f8 100644
--- a/Documentation/contributor-guide/dependency_management.md
+++ b/Documentation/contributor-guide/dependency_management.md
@@ -131,7 +131,7 @@ Suggested steps for performing a minor version upgrade for the etcd development
 
 1. Carefully review new Go version release notes and potentially related blog posts for any deprecations, performance impacts, or other considerations.
 2. Create a GitHub issue to signal intent to upgrade and invite discussion, for example, <https://github.com/etcd-io/etcd/issues/16393>.
-3. Complete the upgrade locally in your development environment.
+3. Complete the upgrade locally in your development environment by editing `.go-version` and running `make fix`.
 4. Run performance benchmarks locally to compare before and after.
 5. Raise a pull request for the changes, for example, <https://github.com/etcd-io/etcd/pull/16394>.
 


Test patch: