Project: etcd (Orchestration & Management / Coordination & Service Discovery)
File: CHANGELOG-3.4.md

Previous change logs can be found at . <hr> Fix . Compile binaries using go . Upgrade . <hr> Fix . Fix . Update . . Fix Add . Fix . Compile binaries using . <hr> Add . Fix leases wrongly revoked by the leader by . Fix . Fix . Add `next-cluster-version-compatible` flag to . Add . Upgrade . Compile binaries using . Upgrade . Upgrade github.com/sirupsen/logrus to v1.9.3 to address . . <hr> Fix Compile binaries using go . Upgrade . <hr> Fix Fix Fix Compile binaries using go . <hr> Improve Use since golang.org/x/net@v0.11.0 started using round-robin scheduler. Add . Add . Fix . . . Fix . Fix . Compile binaries using . Upgrade . Upgrade gRPC to 1.58.3 in https://github.com/etcd-io/etcd/pull/16997 and https://github.com/etcd-io/etcd/pull/16999. Note that gRPC server will reject requests with connection header (refer to https://github.com/grpc/grpc-go/pull/4803). <hr> Fix Improve Fix Add . Compile binaries using . <hr> Fix . Compile binaries using <hr> Add to enable support for TLS 1.3. Add flag to support separating http server from grpc one, thus giving full immunity to . Change Fix Fix Fix . Fix by . Fix . Fix . Fix . Reverted the fix to . Recommend . Compile binaries using . Upgrade . Fix <hr> Fix . Improve Improve Improve Fix . Etcd now uses zap timestamp format: `2006-01-02T15:04:05.999999Z0700` (microsecond instead of milliseconds precision). Fix Fix . Upgrade . Upgrade from to . Updated . <hr> Fix . Fix . Bumped to address some HIGH Vulnerabilities. Fix . Recommend . Compile binaries using . Use to address critical Vulnerabilities. <hr> Fix Fix Fix Fix Fix Fix . Add flag to support adding configurable cipher list. <hr> Fix Fix Fix Fix <hr> Fix . Add flag to configure the max concurrent streams each client can open at a time, and defaults to math.MaxUint32. Add flag to enable checkpoint persisting. Fix , requires enabling checkpoint persisting. Fix Fix Fix Fix Fix Fix Fix <hr> See and for any breaking changes. Fix . Fix . Fix . Fix . Fix . Fix . Upgrade from to . Upgrade to v0.0.0-20220411220226-7b82a4e95df4 to address . Upgrade to v2.4.0 to address . Require . Compile with . etcd uses (instead of vendor dir) to track dependencies. <hr> See and for any breaking changes. See for all metrics per release. Add . Updated from `debian:buster-v1.4.0` to `debian:bullseye-20210927` to fix the following critical CVEs: : miscalculation of a buffer size in openssl's SM2 decryption : integer overflow flaw in glibc : incorrect syscall argument generation in libseccomp : libfetch in apk-tools mishandles numeric strings in FTP and HTTP protocols to allow out of bound reads. <hr> See and for any breaking changes. Fix to work with https endpoints. Add support for http client. Replace . Compile with . <hr> See and for any breaking changes. Add flag which allows apply duration threshold to be configurable. Fix to still write-out data avoiding corruption (most of the time). Reduce . Add . Fix back-ported from (https://github.com/etcd-io/etcd/pull/12196). Compile with . <hr> See and for any breaking changes. Log . Fix . Fix . Bump up . Compile with . <hr> See and for any breaking changes. Fix . Get AuthToken automatically when clientConn is ready. when force-new-cluster flag is enabled in a cluster which had learner node. Remove . These are not used anymore. They were only used for older versions of functional testing. Removed to adhere to best security practices, minimize arbitrary shell invocation. Implement

Compile with . <hr> See and for any breaking changes. A is added when etcd use any existing directory that has a permission different than 700 on Linux and 777 on Windows. Compile with . <hr> See and for any breaking changes. Fix . Fixed via . Compile with . <hr> See and for any breaking changes. Improve . Add flag to make watch progress notify interval configurable. Remove . See . Optimize . Add . Add . Compile with . <hr> See and for any breaking changes. Add flag. Setting the flag disables all uses of fsync, which is unsafe and will cause data loss. This flag makes it possible to run an etcd node for testing and development without placing lots of load on the file system. Add flag to customize `simpleTokenTTL` settings. Improve . Improve . Fix . Fix . Changed behavior on . Previously, the permission was not checked on existing data directory and the directory used for automatically generating self-signed certificates for TLS connections with clients. Now a check is added to make sure those directories, if already exist, has a desired permission of 700 on Linux and 777 on Windows. Compile with . <hr> See and for any breaking changes. Add . See https://github.com/etcd-io/etcd/issues/11918. Compile with . <hr> See and for any breaking changes. Make sure . Make sure . Improve logging around snapshot send and receive. . . Fix . Previously, server restore fails if it had crashed after persisting raft hard state but before saving snapshot. See https://github.com/etcd-io/etcd/issues/10219 for more. . Add . Compile with . <hr> See and for any breaking changes. Improve . Add . Add . Compile with . <hr> See and for any breaking changes. Fix . https://github.com/etcd-io/etcd/issues/11495 https://github.com/etcd-io/etcd/issues/11730 Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Log . Fix . Previously, `clientv3.WithRequireLeader(ctx)` was overwriting existing context keys. Fix command to prevent potential timeout. See for all metrics per release. Add . Fix for metrics handler. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Fix . Previously, during shutdown etcd could accidentally remove needed wal files, resulting in catastrophic error `etcdserver: open wal error: wal: file not found.` during startup. Now, etcd makes sure the purge file loop exits before server signals stop of the raft node. . Fix . Improve to work when peer mTLS is enabled. See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Add Prometheus metric. Fix bug where . Fix () Fix bug where <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Change Prometheus metrics to include only major and minor version. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Fix command to prevent potential timeout. Add to range, put and compact requests in etcdserver. Compile with including security fixes. Fix . Fix . Fix . Fix . <hr> See and for any breaking

Again, before running upgrades from any previous release, please make sure to read change logs below and . See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Add Prometheus metric. Add Prometheus metric. Fix . Remove . Add to allow creating a custom zap logger. Upgrade from to . Compile with including security fixes. <hr> See and for any breaking changes. (2019-08-30), see . (2019-08-29), see . (2019-08-27), see . (2019-08-23), see . (2019-08-15), see . (2019-08-12), see . Again, before running upgrades from any previous release, please make sure to read change logs below and . etcd now has a new website! Please visit https://etcd.io. Add Raft learner: , , . User guide: . API change: . More details on implementation: and . Rewrite with . Upgrade . Improve . Fix . Fix . uses an asynchronous resolver to pass endpoints to the gRPC dial function. To block until the underlying connection is up, pass `grpc.WithBlock()` to `clientv3.Config.DialOptions`. Add . Add to prevent . Improve , which indicates that local node might have slow network. Improve . e.g. `read-only range request "key:\"/a\" rangeend:\"/b\" " with result "rangeresponse_count:3 size:96" took too long (97.966s) to execute`. Redact . Provide . Improve , which indicates message send to a peer failed. Improve to help debug . Improve . Previously, periodic commit on pending writes blocks incoming read transactions, even if there is no pending write. Now, periodic commit operation does not block concurrent read transactions, thus improves long-running read transaction performance. Make . Previously, ongoing long-running read transactions block writes and future reads. With this change, write throughput is increased by 70% and P99 write latency is reduced by 90% in the presence of long-running reads. Improve . Adjust to reduce . Previously, etcd fast-forwards election ticks on server start, with only one tick left for leader election. This is to speed up start phase, without having to wait until all election ticks elapse. Advancing election ticks is useful for cross datacenter deployments with larger election timeouts. However, it was affecting cluster availability if the last tick elapses before leader contacts the restarted node. Now, when etcd restarts, it adjusts election ticks with more than one tick left, thus more time for leader to prevent disruptive restart. Add to reduce . For instance, a flaky(or rejoining) member may drop in and out, and start campaign. This member will end up with a higher term, and ignore all incoming messages with lower term. In this case, a new leader eventually need to get elected, thus disruptive to cluster availability. Raft implements Pre-Vote phase to prevent this kind of disruptions. If enabled, Raft runs an additional phase of election to check if pre-candidate can get enough votes to win an election. Adjust . e.g. `etcd --auto-compaction-mode=revision --auto-compaction-retention=1000` automatically `Compact` on `"latest revision" - 1000` every 5-minute (when latest revision is 30000, compact on revision 29000). e.g. Previously, `etcd --auto-compaction-mode=periodic --auto-compaction-retention=24h` automatically `Compact` with 24-hour retention windown for every 2.4-hour. Now, `Compact` happens for every 1-hour. e.g. Previously, `etcd --auto-compaction-mode=periodic --auto-compaction-retention=30m` automatically `Compact` with 30-minute retention windown for every 3-minute. Now, `Compact` happens for every 30-minute. Periodic compactor keeps recording latest revisions for every compaction period when given period is less than 1-hour, or for every 1-hour when given compaction period is greater than 1-hour (e.g. 1-hour when `etcd --auto-compaction-mode=periodic --auto-compaction-retention=24h`). For every compaction period or 1-hour, compactor uses the last revision that was fetched before compaction period, to discard historical

The retention window of compaction period moves for every given compaction period or hour. For instance, when hourly writes are 100 and `etcd --auto-compaction-mode=periodic --auto-compaction-retention=24h`, `v3.2.x`, `v3.3.0`, `v3.3.1`, and `v3.3.2` compact revision 2400, 2640, and 2880 for every 2.4-hour, while `v3.3.3` or later compacts revision 2400, 2500, 2600 for every 1-hour. Furthermore, when `etcd --auto-compaction-mode=periodic --auto-compaction-retention=30m` and writes per minute are about 1000, `v3.3.0`, `v3.3.1`, and `v3.3.2` compact revision 30000, 33000, and 36000, for every 3-minute, while `v3.3.3` or later compacts revision 30000, 60000, and 90000, for every 30-minute. Improve , address . Make . Make etcd server return `raft.ErrProposalDropped` on internal Raft proposal drop in and . e.g. a node is removed from cluster, or . Add package for easier snapshot workflow (see for more). Improve coverage: , , , , , , . Improve by using a copy on write clone to avoid holding the lock for the traversal of the entire index. Update to allow for use of any supported signature method/algorithm. Add to persist remaining TTLs to the consensus log periodically so that long lived leases progress toward expiry in the presence of leader elections and server restarts. Enabled by experimental flag "--experimental-enable-lease-checkpoint". Add ; enable `etcd --debug` flag to see per-request debug information. Add . If consistency check on snapshot file fails, `snapshot status` returns `"snapshot file integrity check failed..."` error. Add . Improve . Support for reducing security risk introduced by leaked password. The users can only be authenticated with `CommonName` based auth. Add `etcd --experimental-peer-skip-client-san-verification` to . Add `etcd --experimental-compaction-batch-limit` to . Reduced default compaction batch size from 10k revisions to 1k revisions to improve p99 latency during compactions and reduced wait between compactions from 100ms to 10ms. Rewrite with . Upgrade . Improve . Fix . Fix . uses an asynchronous resolver to pass endpoints to the gRPC dial function. To block until the underlying connection is up, pass `grpc.WithBlock()` to `clientv3.Config.DialOptions`. Require . Compile with including security fixes. Migrate dependency management tool from `glide` to . <= 3.3 puts `vendor` directory under `cmd/vendor` directory to . 3.4 moves `cmd/vendor` directory to `vendor` at repository root. Remove recursive symlinks in `cmd` directory. Now `go get/install/build` on `etcd` packages (e.g. `clientv3`, `tools/benchmark`) enforce builds with etcd `vendor` directory. Deprecated `latest` tag. `docker pull gcr.io/etcd-development/etcd:latest` would not be up-to-date. Deprecated version tags. `docker pull gcr.io/etcd-development/etcd:v3.3` would still work. `docker pull gcr.io/etcd-development/etcd:v3.4` would not work. Use `docker pull gcr.io/etcd-development/etcd:v3.4.x` instead, with the exact patch version. Deprecated . , as of late 2016. is not maintained anymore. `*.aci` files are not available from `v3.4` release. Move to . Change import path to `"go.etcd.io/etcd"`. e.g. `import "go.etcd.io/etcd/raft"`. Make . Now, `etcdctl set foo bar` must be `ETCDCTL_API=2 etcdctl set foo bar`. Now, `ETCDCTL_API=3 etcdctl put foo bar` could be just `etcdctl put foo bar`. Make . Make . Deprecated `etcd --ca-file` flag. Use instead (`etcd --ca-file` flag has been marked deprecated since v2.1). Deprecated `etcd --peer-ca-file` flag. Use instead (`etcd --peer-ca-file` flag has been marked deprecated since v2.1). Deprecated `pkg/transport.TLSInfo.CAFile` field. Use instead (`CAFile` field has been marked deprecated since v2.1). Exit on . Address . e.g. exit with error on `--advertise-client-urls=http://:2379`. e.g. exit with error on `--initial-advertise-peer-urls=http://:2380`. Exit on . Address . e.g. exit with error on `ETCD_NAME=abc etcd --name=def`. e.g. exit with error on `ETCDINITIALCLUSTER_TOKEN=abc etcd --initial-cluster-token=def`. e.g. exit with error on `ETCDCTLENDPOINTS=abc.com ETCDCTLAPI=3 etcdctl endpoint health --endpoints=def.com`. Change . Deprecating `etcddebuggingmvccdbtotalsizein_bytes` Prometheus metric (to be removed in v3.5). Use

Deprecating `etcddebuggingmvccputtotal` Prometheus metric (to be removed in v3.5). Use instead. Deprecating `etcddebuggingmvccdeletetotal` Prometheus metric (to be removed in v3.5). Use instead. Deprecating `etcddebuggingmvccrangetotal` Prometheus metric (to be removed in v3.5). Use instead. Deprecating `etcddebuggingmvcctxntotal`Prometheus metric (to be removed in v3.5). Use instead. Rename `etcdserver.ServerConfig.SnapCount` field to `etcdserver.ServerConfig.SnapshotCount`, to be consistent with the flag name `etcd --snapshot-count`. Rename `embed.Config.SnapCount` field to , to be consistent with the flag name `etcd --snapshot-count`. Change . Deprecated . Now logger is set up automatically based on . Rename to support multiple log outputs. `etcd --log-output` will be deprecated in v3.5. Rename to support multiple log outputs. Change to support multiple log outputs. Now that `etcd --log-outputs` accepts multiple writers, etcd configuration YAML file `log-outputs` field must be changed to `[]string` type. Previously, `etcd --config-file etcd.config.yaml` can have `log-outputs: default` field, now must be `log-outputs: [default]`. Deprecating flag. Use `etcd --log-level=debug` flag instead. v3.5 will deprecate `etcd --debug` flag in favor of `etcd --log-level=debug`. Change v3 `etcdctl snapshot` exit codes with . Exit on error with exit code 1 (no more exit code 5 or 6 on `snapshot save/restore` commands). Deprecated . `clientv3` and `proxy/grpcproxy` now does not return `grpc.ErrClientConnClosing`. `grpc.ErrClientConnClosing` has been . Use `clientv3.IsConnCanceled(error)` or `google.golang.org/grpc/status.FromError(error)` instead. Deprecated endpoint `/v3beta` with . Deprecated . To deprecate in v3.5. In v3.4, `curl -L http://localhost:2379/v3beta/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}'` still works as a fallback to `curl -L http://localhost:2379/v3/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}'`, but `curl -L http://localhost:2379/v3beta/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}'` won't work in v3.5. Use `curl -L http://localhost:2379/v3/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}'` instead. Change to support in server-side. Previously, `Open(dirpath string, snap walpb.Snapshot) (WAL, error)`, now `Open(lg zap.Logger, dirpath string, snap walpb.Snapshot) (*WAL, error)`. Previously, `OpenForRead(dirpath string, snap walpb.Snapshot) (WAL, error)`, now `OpenForRead(lg zap.Logger, dirpath string, snap walpb.Snapshot) (*WAL, error)`. Previously, `Repair(dirpath string) bool`, now `Repair(lg *zap.Logger, dirpath string) bool`. Previously, `Create(dirpath string, metadata []byte) (WAL, error)`, now `Create(lg zap.Logger, dirpath string, metadata []byte) (*WAL, error)`. Remove . Move internal packages to `etcdserver`. `"github.com/coreos/etcd/alarm"` to `"go.etcd.io/etcd/etcdserver/api/v3alarm"`. `"github.com/coreos/etcd/compactor"` to `"go.etcd.io/etcd/etcdserver/api/v3compactor"`. `"github.com/coreos/etcd/discovery"` to `"go.etcd.io/etcd/etcdserver/api/v2discovery"`. `"github.com/coreos/etcd/etcdserver/auth"` to `"go.etcd.io/etcd/etcdserver/api/v2auth"`. `"github.com/coreos/etcd/etcdserver/membership"` to `"go.etcd.io/etcd/etcdserver/api/membership"`. `"github.com/coreos/etcd/etcdserver/stats"` to `"go.etcd.io/etcd/etcdserver/api/v2stats"`. `"github.com/coreos/etcd/error"` to `"go.etcd.io/etcd/etcdserver/api/v2error"`. `"github.com/coreos/etcd/rafthttp"` to `"go.etcd.io/etcd/etcdserver/api/rafthttp"`. `"github.com/coreos/etcd/snap"` to `"go.etcd.io/etcd/etcdserver/api/snap"`. `"github.com/coreos/etcd/store"` to `"go.etcd.io/etcd/etcdserver/api/v2store"`. Change : On Linux, the snapshot file changes from readable by all (mode 0644) to readable by the user only (mode 0600). Change . See and . Release branch `/version` defines version `3.4.x-pre`, instead of `3.4.y+git`. Use `3.4.5-pre`, instead of `3.4.4+git`. Upgrade from to . Upgrade from to . Migrate to , to (See for more). Migrate to (See for more). Upgrade from to . Upgrade from to . Upgrade from to . Upgrade from to . Upgrade from to . Upgrade from to . Upgrade from to . Upgrade from to . Upgrade from to . Upgrade from to . Upgrade from to . Upgrade from to . Upgrade from to . Upgrade from to . Upgrade from to . Upgrade from to . Migrate to , as the later has replaced the original module. Upgrade from to . See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Let's say `"7339c4e5e833c029"` server `/metrics` returns `etcdnetworkactivepeers{Local="7339c4e5e833c029",Remote="729934363faa4a24"} 1` and `etcdnetworkactivepeers{Local="7339c4e5e833c029",Remote="b548c2511513015"} 1`. This indicates that the local node `"7339c4e5e833c029"` currently has two active remote peers `"729934363faa4a24"` and `"b548c2511513015"` in a 3-node

If the node `"b548c2511513015"` is down, the local node `"7339c4e5e833c029"` will show `etcdnetworkactivepeers{Local="7339c4e5e833c029",Remote="729934363faa4a24"} 1` and `etcdnetworkactivepeers{Local="7339c4e5e833c029",Remote="b548c2511513015"} 0`. Add Prometheus metric. If a remote peer `"b548c2511513015"` is down, the local node `"7339c4e5e833c029"` server `/metrics` would return `etcdnetworkdisconnectedpeerstotal{Local="7339c4e5e833c029",Remote="b548c2511513015"} 1`, while active peer metrics will show `etcdnetworkactivepeers{Local="7339c4e5e833c029",Remote="729934363faa4a24"} 1` and `etcdnetworkactivepeers{Local="7339c4e5e833c029",Remote="b548c2511513015"} 0`. Add Prometheus metric. e.g. `etcdnetworkserverstreamfailures_total{API="lease-keepalive",Type="receive"} 1` e.g. `etcdnetworkserverstreamfailures_total{API="watch",Type="receive"} 1` Improve Prometheus metric to track leader heartbeats. Previously, it only samples the TCP connection for snapshot messages. Increase Prometheus metric histogram upper-bound. Previously, highest bucket only collects requests taking 0.8192 seconds or more. Now, highest buckets collect 0.8192 seconds, 1.6384 seconds, and 3.2768 seconds or more. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. To replace . Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Use it with `etcdmvccdbtotalsizeinbytes` and `etcdmvccdbtotalsizeinuseinbytes`. `etcdserverquotabackendbytes 2.147483648e+09` means current quota size is 2 GB. `etcdmvccdbtotalsizeinbytes 20480` means current physically allocated DB size is 20 KB. `etcdmvccdbtotalsizeinuseinbytes 16384` means future DB size if defragment operation is complete. `etcdmvccdbtotalsizeinbytes - etcdmvccdbtotalsizeinuseinbytes` is the number of bytes that can be saved on disk with defragment operation. Add Prometheus metric. Use it with `etcdmvccdbtotalsizeinbytes` and `etcdmvccdbtotalsizeinuseinbytes`. `etcdserverquotabackendbytes 2.147483648e+09` means current quota size is 2 GB. `etcdmvccdbtotalsizeinbytes 20480` means current physically allocated DB size is 20 KB. `etcdmvccdbtotalsizeinuseinbytes 16384` means future DB size if defragment operation is complete. `etcdmvccdbtotalsizeinbytes - etcdmvccdbtotalsizeinuseinbytes` is the number of bytes that can be saved on disk with defragment operation. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Increase Prometheus metric histogram upper-bound. Previously, highest bucket only collects requests taking 1.024 seconds or more. Now, highest buckets collect 1.024 seconds, 2.048 seconds, and 4.096 seconds or more. Fix missing Prometheus metric count. Fix Prometheus metric. Fix . Change . The metrics that were exposed via the proxy were not etcd server members but instead the proxy itself. Fix bug where . Deprecating `etcddebuggingmvccdbtotalsizein_bytes` Prometheus metric (to be removed in v3.5). Use instead. Deprecating `etcddebuggingmvccputtotal` Prometheus metric (to be removed in v3.5). Use instead. Deprecating `etcddebuggingmvccdeletetotal` Prometheus metric (to be removed in v3.5). Use instead. Deprecating `etcddebuggingmvccrangetotal` Prometheus metric (to be removed in v3.5). Use instead. Deprecating `etcddebuggingmvcctxntotal`Prometheus metric (to be removed in v3.5). Use instead. See for more details. Support TLS cipher suite whitelisting. To block . TLS handshake fails when client hello is requested with invalid cipher suites. Add flag. If empty, Go auto-populates the list. Add flag, , and , to prevent attack. Any website can simply create an authorized DNS name, and direct DNS to `"localhost"` (or any other address). Then, all HTTP endpoints of etcd server listening on `"localhost"` becomes accessible, thus vulnerable to . Client origin enforce policy works as follow: If client connection is secure via HTTPS, allow any hostnames.. If client connection is not secure and `"HostWhitelist"` is not empty, only allow HTTP requests whose Host field is listed in whitelist. By default, `"HostWhitelist"` is `"*"`, which means insecure server allows all client HTTP requests. Note that the client origin policy is enforced whether authentication is enabled or not, for tighter controls. When specifying hostnames, loopback addresses are not added

To allow loopback interfaces, add them to whitelist manually (e.g. `"localhost"`, `"127.0.0.1"`, etc.). e.g. `etcd --host-whitelist example.com`, then the server will reject all HTTP requests whose Host field is not `example.com` (also rejects requests to `"localhost"`). Support in v3 HTTP requests (gRPC gateway). Support . e.g. `etcd --auth-token jwt,pub-key=<pub key path>,priv-key=<priv key path>,sign-method=<sign method>,ttl=5m`. Allow empty token provider in . Fix when . In Go, server calls `(tls.Config).GetCertificate` for TLS reload if and only if server's `(tls.Config).Certificates` field is not empty, or `(tls.ClientHelloInfo).ServerName` is not empty with a valid SNI from the client. Previously, etcd always populates `(tls.Config).Certificates` on the initial client TLS handshake, as non-empty. Thus, client was always expected to supply a matching SNI in order to pass the TLS verification and to trigger `(*tls.Config).GetCertificate` to reload TLS assets. However, a certificate whose SAN field does would request `*tls.ClientHelloInfo` with an empty `ServerName` field, thus failing to trigger the TLS reload on initial TLS handshake; this becomes a problem when expired certificates need to be replaced online. Now, `(tls.Config).Certificates` is created empty on initial TLS client handshake, first to trigger `(tls.Config).GetCertificate`, and then to populate rest of the certificates on every new TLS connection, even when client SNI is empty (e.g. cert only includes IPs). Add . Now linearizable requests with read index would fail fast when there is a leadership change, instead of waiting until context timeout. Add flag to configure initial election tick fast-forward. By default, `etcd --initial-election-tick-advance=true`, then local member fast-forwards election ticks to speed up "initial" leader election trigger. This benefits the case of larger election ticks. For instance, cross datacenter deployment may require longer election timeout of 10-second. If true, local node does not need wait up to 10-second. Instead, forwards its election ticks to 8-second, and have only 2-second left before leader election. Major assumptions are that: cluster has no active leader thus advancing ticks enables faster leader election. Or cluster already has an established leader, and rejoining follower is likely to receive heartbeats from the leader after tick advance and before election timeout. However, when network from leader to rejoining follower is congested, and the follower does not receive leader heartbeat within left election ticks, disruptive election has to happen thus affecting cluster availabilities. Now, this can be disabled by setting `etcd --initial-election-tick-advance=false`. Disabling this would slow down initial bootstrap process for cross datacenter deployments. Make tradeoffs by configuring `etcd --initial-election-tick-advance` at the cost of slow initial bootstrap. If single-node, it advances ticks regardless. Address . Add flag to enable to run an additional Raft election phase. For instance, a flaky(or rejoining) member may drop in and out, and start campaign. This member will end up with a higher term, and ignore all incoming messages with lower term. In this case, a new leader eventually need to get elected, thus disruptive to cluster availability. Raft implements Pre-Vote phase to prevent this kind of disruptions. If enabled, Raft runs an additional phase of election to check if pre-candidate can get enough votes to win an election. `etcd --pre-vote=false` by default. v3.5 will enable `etcd --pre-vote=true` by default. Add `etcd --experimental-compaction-batch-limit` to . Reduced default compaction batch size from 10k revisions to 1k revisions to improve p99 latency during compactions and reduced wait between compactions from 100ms to 10ms. Add flag to support custom DNS SRV name with discovery. If not given, etcd queries `etcd-server-ssl.tcp.[YOURHOST]` and `etcd-server.tcp.[YOURHOST]`. If `etcd --discovery-srv-name="foo"`, then query `etcd-server-ssl-foo.tcp.[YOURHOST]` and `etcd-server-foo.tcp.[YOURHOST]`. Useful for operating multiple etcd clusters under the same

Support TLS cipher suite whitelisting. To block . TLS handshake fails when client hello is requested with invalid cipher suites. Add flag. If empty, Go auto-populates the list. Support in v3 HTTP requests (gRPC gateway). Rename to support multiple log outputs. `etcd --log-output` will be deprecated in v3.5. Add flag to support in server-side. `etcd --logger=capnslog` will be deprecated in v3.5. Main motivation is to promote automated etcd monitoring, rather than looking back server logs when it starts breaking. Future development will make etcd log as few as possible, and make etcd easier to monitor with metrics and alerts. `etcd --logger=capnslog --log-outputs=default` is the default setting and same as previous etcd server logging format. `etcd --logger=zap --log-outputs=default` is not supported when `etcd --logger=zap`. Use `etcd --logger=zap --log-outputs=stderr` instead. Or, use `etcd --logger=zap --log-outputs=systemd/journal` to send logs to the local systemd journal. Previously, if etcd parent process ID (PPID) is 1 (e.g. run with systemd), `etcd --logger=capnslog --log-outputs=default` redirects server logs to local systemd journal. And if write to journald fails, it writes to `os.Stderr` as a fallback. However, even with PPID 1, it can fail to dial systemd journal (e.g. run embedded etcd with Docker container). Then, and fall back to `os.Stderr`, which is inefficient. To avoid this problem, systemd journal logging must be configured manually. `etcd --logger=zap --log-outputs=stderr` will log server operations in and writes logs to `os.Stderr`. Use this to override journald log redirects. `etcd --logger=zap --log-outputs=stdout` will log server operations in and writes logs to `os.Stdout` Use this to override journald log redirects. `etcd --logger=zap --log-outputs=a.log` will log server operations in and writes logs to the specified file `a.log`. `etcd --logger=zap --log-outputs=a.log,b.log,c.log,stdout` and outputs to `os.Stderr`, in . `etcd --logger=zap --log-outputs=/dev/null` will discard all server logs. Add flag to support log level. v3.5 will deprecate `etcd --debug` flag in favor of `etcd --log-level=debug`. Add flag. Add flag. Fix . "unsynced" watcher is watcher that needs to be in sync with events that have happened. That is, "unsynced" watcher is the slow watcher that was requested on old revision. "unsynced" watcher restore operation was not correctly populating its underlying watcher group. Which possibly causes . A node gets network partitioned with a watcher on a future revision, and falls behind receiving a leader snapshot after partition gets removed. When applying this snapshot, etcd watch storage moves current synced watchers to unsynced since sync watchers might have become stale during network partition. And reset synced watcher group to restart watcher routines. Previously, there was a bug when moving from synced watcher group to unsynced, thus client would miss events when the watcher was requested to the network-partitioned node. Fix . Let's assume that a watcher had been requested with a future revision X and sent to node A that became network-partitioned thereafter. Meanwhile, cluster makes progress. Then when the partition gets removed, the leader sends a snapshot to node A. Previously if the snapshot's latest revision is still lower than the watch revision X, etcd server panicked during snapshot restore operation. Now, this server-side panic has been fixed. Fix . Previously, wrong-formatted HTTP requests to Election API could trigger panic in etcd server. e.g. `curl -L http://localhost:2379/v3/election/proclaim -X POST -d '{"value":""}'`, `curl -L http://localhost:2379/v3/election/resign -X POST -d '{"value":""}'`. Fix . Previously, `etcd --auto-compaction-mode revision --auto-compaction-retention 1` was . Now, `etcd --auto-compaction-mode revision --auto-compaction-retention 1` is correctly parsed as revision retention 1. Prevent . `TTL` parameter to `Grant` request is unit of second. Leases with too large `TTL` values exceeding

. Server now returns `rpctypes.ErrLeaseTTLTooLarge` to client, when the requested `TTL` is larger than 9,000,000,000 seconds (which is >285 years). Again, etcd `Lease` is meant for short-periodic keepalives or sessions, in the range of seconds or minutes. Not for hours or days! Fix . Fix . Enable etcd server . Allow . Previously, existing files such as in WAL directory prevent etcd server boot. Now, WAL directory that contains only `lost+found` or a file that's not suffixed with `.wal` is considered non-initialized. Fix . Fix . Fix . Previously, creating a role with an empty name crashed etcd server with an error code `Unavailable`. Now, creating a role with an empty name is not allowed with an error code `InvalidArgument`. Add `isLearner` field to `etcdserverpb.Member`, `etcdserverpb.MemberAddRequest` and `etcdserverpb.StatusResponse` as part of . Add `MemberPromote` rpc to `etcdserverpb.Cluster` interface and the corresponding `MemberPromoteRequest` and `MemberPromoteResponse` as part of . Add package for snapshot restore/save operations (see for more). Add to allow user-provided watch ID to `mvcc`. Corresponding `watch_id` is returned via `etcdserverpb.WatchResponse`, if any. Add to request etcd server to when the total size of events exceeds `etcd --max-request-bytes` flag value plus gRPC-overhead 512 bytes. The default server-side request bytes limit is `embed.DefaultMaxRequestBytes` which is 1.5 MiB plus gRPC-overhead 512 bytes. If watch response events exceed this server-side request limit and watch request is created with `fragment` field `true`, the server will split watch events into a set of chunks, each of which is a subset of watch events below server-side request limit. Useful when client-side has limited bandwidths. For example, watch response contains 10 events, where each event is 1 MiB. And server `etcd --max-request-bytes` flag value is 1 MiB. Then, server will send 10 separate fragmented events to the client. For example, watch response contains 5 events, where each event is 2 MiB. And server `etcd --max-recv-bytes` flag value is 1 MiB and `clientv3.Config.MaxCallRecvMsgSize` is 1 MiB. Then, server will try to send 5 separate fragmented events to the client, and the client will error with `"code = ResourceExhausted desc = grpc: received message larger than max (...)"`. Client must implement fragmented watch event merge (which `clientv3` does in etcd v3.4). Add for current Raft applied index. Add for server-side error. e.g. `"etcdserver: no leader", "NOSPACE", "CORRUPT"` Add for actual DB size after compaction. Add . To manually trigger broadcasting watch progress event (empty watch response with latest header) to all associated watch streams. Think of it as `WithProgressNotify` that can be triggered manually. Note: v3.5 will deprecate `etcd --log-package-levels` flag for `capnslog`; `etcd --logger=zap --log-outputs=stderr` will the default. v3.5 will deprecate `[CLIENT-URL]/config/local/log` endpoint. Add to specify a list of supported cipher suites for TLS handshake between client/server and peers. If empty, Go auto-populates the list. Both `embed.Config.ClientTLSInfo.CipherSuites` and `embed.Config.CipherSuites` cannot be non-empty at the same time. If not empty, specify either `embed.Config.ClientTLSInfo.CipherSuites` or `embed.Config.CipherSuites`. Add to enable/disable initial election tick fast-forward. `embed.NewConfig()` would return `*embed.Config` with `InitialElectionTickAdvance` as true by default. Define for `compactor.ModePeriodic`. Define for `compactor.ModeRevision`. Change . Remove . Now logger is set up automatically based on . Add to support in server-side. Add . Rename `embed.Config.SnapCount` field to , to be consistent with the flag name `etcd --snapshot-count`. Rename to support multiple log outputs. Change to support multiple log outputs. Add field. Add field. Make . Change . See and . Improve . See and . Fix . Previously, delete operation violates . Add . Deprecated `capnslog` in integration tests. Add to `Clientv3.Cluster`

This API is used to add a learner member to etcd cluster. Add to `Clientv3.Cluster` interface. This API is used to promote a learner member in etcd cluster. Client may receive from server. Now linearizable requests with read index would fail fast when there is a leadership change, instead of waiting until context timeout. Add to support when the total size of events exceeds `etcd --max-request-bytes` flag value plus gRPC-overhead 512 bytes. Watch fragmentation is disabled by default. The default server-side request bytes limit is `embed.DefaultMaxRequestBytes` which is 1.5 MiB plus gRPC-overhead 512 bytes. If watch response events exceed this server-side request limit and watch request is created with `fragment` field `true`, the server will split watch events into a set of chunks, each of which is a subset of watch events below server-side request limit. Useful when client-side has limited bandwidths. For example, watch response contains 10 events, where each event is 1 MiB. And server `etcd --max-request-bytes` flag value is 1 MiB. Then, server will send 10 separate fragmented events to the client. For example, watch response contains 5 events, where each event is 2 MiB. And server `etcd --max-request-bytes` flag value is 1 MiB and `clientv3.Config.MaxCallRecvMsgSize` is 1 MiB. Then, server will try to send 5 separate fragmented events to the client, and the client will error with `"code = ResourceExhausted desc = grpc: received message larger than max (...)"`. Add . To manually trigger broadcasting watch progress event (empty watch response with latest header) to all associated watch streams. Think of it as `WithProgressNotify` that can be triggered manually. Fix . If `<-chan *clientv3LeaseKeepAliveResponse` from `clientv3.Lease.KeepAlive` was never consumed or channel is full, client was instead of expected rate of every "TTL / 3" duration. Change : On Linux, the snapshot file changes from readable by all (mode 0644) to readable by the user only (mode 0600). Client may choose to send keepalive pings to server using . By setting `PermitWithoutStream` to true, client can send keepalive pings to server without any active streams(RPCs). In other words, it allows sending keepalive pings with unary or simple RPC calls. `PermitWithoutStream` is set to false by default. Fix logic on in `clientv3/concurrency` package. Fix . Deprecated . `clientv3` and `proxy/grpcproxy` now does not return `grpc.ErrClientConnClosing`. `grpc.ErrClientConnClosing` has been . Use `clientv3.IsConnCanceled(error)` or `google.golang.org/grpc/status.FromError(error)` instead. Make . Now, `etcdctl set foo bar` must be `ETCDCTL_API=2 etcdctl set foo bar`. Now, `ETCDCTL_API=3 etcdctl put foo bar` could be just `etcdctl put foo bar`. Add to add and promote raft learner member in etcd cluster. Add flag. To support . e.g. `etcdctl --user user --password password get foo` Add flag. Add command. Add flags. Add flags. Add flag. Add . Add . Add . Previously, . Add . Fix parsing. Previously, `ETCDCTL_API=3 etcdctl watch foo -- echo watch event received` panicked. Fix . Add . To manually trigger broadcasting watch progress event (empty watch response with latest header) to all associated watch streams. Think of it as `WithProgressNotify` that can be triggered manually. Add to `etcdctl snapshot save`. User can specify timeout of `etcdctl snapshot save` command using flag `--command-timeout`. Fix etcdctl to Fix . Let's assume that a watcher had been requested with a future revision X and sent to node A that became network-partitioned thereafter. Meanwhile, cluster makes progress. Then when the partition gets removed, the leader sends a snapshot to node

Previously if the snapshot's latest revision is still lower than the watch revision X, etcd server panicked during snapshot restore operation. Especially, gRPC proxy was affected, since it detects a leader loss with a key `"proxy-namespacelostleader"` and a watch revision `"int64(math.MaxInt64 - 2)"`. Now, this server-side panic has been fixed. Fix . Change . The metrics that were exposed via the proxy were not etcd server members but instead the proxy itself. Replace endpoint `/v3beta` with . Deprecated . To deprecate in v3.5. In v3.4, `curl -L http://localhost:2379/v3beta/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}'` still works as a fallback to `curl -L http://localhost:2379/v3/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}'`, but `curl -L http://localhost:2379/v3beta/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}'` won't work in v3.5. Use `curl -L http://localhost:2379/v3/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}'` instead. Add API endpoints . To deprecate in v3.5. Support in v3 HTTP requests (gRPC gateway). Fix . Add . Now if a proposal has been ignored. e.g. a node is removed from cluster, or . Improve by keeping track of latest `pb.EntryConfChange` index. Previously record `pendingConf` boolean field scanning the entire tail of the log, which can delay heartbeat send. Fix . Add to limit the total size of the uncommitted entries in bytes. Once exceeded, raft returns `raft.ErrProposalDropped` error. Prevent . There was a bug in but fixed via . Add . This prevents out-of-memory errors if the raft log has become very large and commits all at once. Fix . Optimize . Leader now sends more append entries if it has more non-empty entries to send after updating flow control information. Now, Raft allows multiple in-flight append messages. Optimize . By boxing a heap-allocated slice header instead of the slice header on the stack, we can avoid an allocation when passing through the sort.Interface interface. Avoid . Avoid . Extract . Add . Optimize . Make . Prevent . Add . Bundle . Use . Implement . Refactor . Allow . Add . Use . Add . Add . `raftpb.ConfChange` continues to work as today: it allows carrying out a single configuration change. A `pb.ConfChange` proposal gets added to the Raft log as such and is thus also observed by the app during Ready handling, and fed back to ApplyConfChange. `raftpb.ConfChangeV2` allows joint configuration changes but will continue to carry out configuration changes in "one phase" (i.e. without ever entering a joint config) when this is possible. `raftpb.ConfChangeV2` messages initiate configuration changes. They support both the simple "one at a time" membership change protocol and full Joint Consensus allowing for arbitrary changes in membership. Change . Allow . necessary in the situation in which a learner has been promoted (i.e. is now a voter) but has not learned about this yet. Fix . Visit . Proactively . The general expectation in `tracker.Progress.Next == c.LastIndex` is that the follower has no log at all (and will thus likely need a snapshot), though the app may have applied a snapshot out of band before adding the replica (thus making the first index the better choice). Previously, when the leader applied a new configuration that added voters, it would not immediately probe these voters, delaying when they would be caught up. Add . Fix . Add flag to support WAL log filtering by entry type. Add flag to support custom decoder. Add file to release assets. etcd maintainers are a distributed team, this change allows for releases to be cut and validation provided without requiring a signing key. Require . Compile with including security fixes. to improve security and maintenance effort for etcd