Project: etcd (Orchestration & Management / Coordination & Service Discovery)
File: CHANGELOG-3.2.md

Previous change logs can be found at . <hr> See and for any breaking changes. add wal slice bound check to make sure entry index is not greater than the number of entries check slice size in decodeRecord fix panic when decoder not set fix constant for linux locking Compile with . <hr> See and for any breaking changes. Improve . . fix a data corruption bug caused by lease expiration when authentication is enabled and upgrading cluster from etcd-3.2 to etcd-3.3 Optimize . Add . Compile with . <hr> See and for any breaking changes. Add . Add . Compile with . <hr> See and for any breaking changes. . Log . Fix . Previously, `clientv3.WithRequireLeader(ctx)` was overwriting existing context keys. See for all metrics per release. Add . Compile with . <hr> Add `etcd --experimental-peer-skip-client-san-verification` to . See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Add Prometheus metric. Fix . Previously, during shutdown etcd could accidentally remove needed wal files, resulting in catastrophic error `etcdserver: open wal error: wal: file not found.` during startup. Now, etcd makes sure the purge file loop exits before server signals stop of the raft node. Compile with . <hr> with etcdctl v2 Add . Previously, . The command output is changed. Previously, if endpoint is unreachable, the command output is "\<endpoint\> is unhealthy: failed to connect: \<error message\>". This change unified the error message, all error types now have the same output "\<endpoint\> is unhealthy: failed to commit proposal: \<error message\>". Fix . For example, start etcd `v3.3.10` Write some data Use etcdctl `v3.3.10` to save snapshot Somehow, upgrading Kubernetes fails, thus rolling back to previous version etcd `v3.2.24` Run etcdctl `v3.2.24` `snapshot status` against the snapshot file saved from `v3.3.10` server Run etcdctl `v3.2.24` `snapshot restore` fails with `"expected sha256 [12..."` See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Fix bug where . Add Prometheus metric. Add Prometheus metric. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Fix . Disable gRPC-gateway proxy requests to etcd server use the etcd client server TLS certificate. If that certificate contains CommonName we do not want to use that for authentication as it could lead to permission escalation. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Improve , which indicates message send to a peer failed. Improve , which indicates that local node might have slow network. Add ; enable `etcd --debug` flag to see per-request debug information. Add . If consistency check on snapshot file fails, `snapshot status` returns `"snapshot file integrity check failed..."` error. See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Improve Prometheus metric to track leader heartbeats. Previously, it only samples the TCP connection for snapshot messages. Display all registered

Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Fix logic on in `clientv3/concurrency` package. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Improve . See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Use it with `etcdmvccdbtotalsizeinbytes` and `etcdmvccdbtotalsizeinuseinbytes`. `etcdserverquotabackendbytes 2.147483648e+09` means current quota size is 2 GB. `etcdmvccdbtotalsizeinbytes 20480` means current physically allocated DB size is 20 KB. `etcdmvccdbtotalsizeinuseinbytes 16384` means future DB size if defragment operation is complete. `etcdmvccdbtotalsizeinbytes - etcdmvccdbtotalsizeinuseinbytes` is the number of bytes that can be saved on disk with defragment operation. Add Prometheus metric. In addition to . Add Prometheus metric. Use it with `etcdmvccdbtotalsizeinbytes` and `etcdserverquotabackendbytes`. `etcdserverquotabackendbytes 2.147483648e+09` means current quota size is 2 GB. `etcdmvccdbtotalsizeinbytes 20480` means current physically allocated DB size is 20 KB. `etcdmvccdbtotalsizeinuseinbytes 16384` means future DB size if defragment operation is complete. `etcdmvccdbtotalsizeinbytes - etcdmvccdbtotalsizeinuseinbytes` is the number of bytes that can be saved on disk with defragment operation. Add : Add `grpc-proxy start --cert-file`, `grpc-proxy start --key-file` and `grpc-proxy start --trusted-ca-file` flags. Add . Fix . If `<-chan *clientv3LeaseKeepAliveResponse` from `clientv3.Lease.KeepAlive` was never consumed or channel is full, client was instead of expected rate of every "TTL / 3" duration. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Improve . e.g. `read-only range request "key:\"/a\" rangeend:\"/b\" " with result "rangeresponse_count:3 size:96" took too long (97.966s) to execute`. Redact . Provide . Add . See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Add Prometheus metric. To replace . Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Support TLS cipher suite whitelisting. To block . TLS handshake fails when client hello is requested with invalid cipher suites. Add flag. If empty, Go auto-populates the list. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Fix . Fix . Let's assume that a watcher had been requested with a future revision X and sent to node A that became network-partitioned thereafter. Meanwhile, cluster makes progress. Then when the partition gets removed, the leader sends a snapshot to node A. Previously if the snapshot's latest revision is still lower than the watch revision X, etcd server panicked during snapshot restore operation. Now, this server-side panic has been fixed. Compile with . <hr> See and for any breaking

Again, before running upgrades from any previous release, please make sure to read change logs below and . Purge old . Previously, etcd did not respect `--max-snapshots` flag to purge old `*.snap.db` files. Now, etcd purges old `*.snap.db` files to keep maximum `--max-snapshots` number of files on disk. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Fix Prometheus metric. Fix . Add Prometheus metric. Fix when . In Go, server calls `(tls.Config).GetCertificate` for TLS reload if and only if server's `(tls.Config).Certificates` field is not empty, or `(tls.ClientHelloInfo).ServerName` is not empty with a valid SNI from the client. Previously, etcd always populates `(tls.Config).Certificates` on the initial client TLS handshake, as non-empty. Thus, client was always expected to supply a matching SNI in order to pass the TLS verification and to trigger `(*tls.Config).GetCertificate` to reload TLS assets. However, a certificate whose SAN field does would request `*tls.ClientHelloInfo` with an empty `ServerName` field, thus failing to trigger the TLS reload on initial TLS handshake; this becomes a problem when expired certificates need to be replaced online. Now, `(tls.Config).Certificates` is created empty on initial TLS client handshake, first to trigger `(tls.Config).GetCertificate`, and then to populate rest of the certificates on every new TLS connection, even when client SNI is empty (e.g. cert only includes IPs). Add flag to configure initial election tick fast-forward. By default, `etcd --initial-election-tick-advance=true`, then local member fast-forwards election ticks to speed up "initial" leader election trigger. This benefits the case of larger election ticks. For instance, cross datacenter deployment may require longer election timeout of 10-second. If true, local node does not need wait up to 10-second. Instead, forwards its election ticks to 8-second, and have only 2-second left before leader election. Major assumptions are that: cluster has no active leader thus advancing ticks enables faster leader election. Or cluster already has an established leader, and rejoining follower is likely to receive heartbeats from the leader after tick advance and before election timeout. However, when network from leader to rejoining follower is congested, and the follower does not receive leader heartbeat within left election ticks, disruptive election has to happen thus affecting cluster availabilities. Now, this can be disabled by setting `--initial-election-tick-advance=false`. Disabling this would slow down initial bootstrap process for cross datacenter deployments. Make tradeoffs by configuring `--initial-election-tick-advance` at the cost of slow initial bootstrap. If single-node, it advances ticks regardless. Address . Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Adjust to reduce . Previously, etcd fast-forwards election ticks on server start, with only one tick left for leader election. This is to speed up start phase, without having to wait until all election ticks elapse. Advancing election ticks is useful for cross datacenter deployments with larger election

However, it was affecting cluster availability if the last tick elapses before leader contacts the restarted node. Now, when etcd restarts, it adjusts election ticks with more than one tick left, thus more time for leader to prevent disruptive restart. See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Add missing . Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Fix . Previously, wrong-formatted HTTP requests to Election API could trigger panic in etcd server. e.g. `curl -L http://localhost:2379/v3/election/proclaim -X POST -d '{"value":""}'`, `curl -L http://localhost:2379/v3/election/resign -X POST -d '{"value":""}'`. Prevent . `TTL` parameter to `Grant` request is unit of second. Leases with too large `TTL` values exceeding `math.MaxInt64` . Server now returns `rpctypes.ErrLeaseTTLTooLarge` to client, when the requested `TTL` is larger than 9,000,000,000 seconds (which is >285 years). Again, etcd `Lease` is meant for short-periodic keepalives or sessions, in the range of seconds or minutes. Not for hours or days! Enable etcd server . Fix . Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Fix . "unsynced" watcher is watcher that needs to be in sync with events that have happened. That is, "unsynced" watcher is the slow watcher that was requested on old revision. "unsynced" watcher restore operation was not correctly populating its underlying watcher group. Which possibly causes . A node gets network partitioned with a watcher on a future revision, and falls behind receiving a leader snapshot after partition gets removed. When applying this snapshot, etcd watch storage moves current synced watchers to unsynced since sync watchers might have become stale during network partition. And reset synced watcher group to restart watcher routines. Previously, there was a bug when moving from synced watcher group to unsynced, thus client would miss events when the watcher was requested to the network-partitioned node. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Prevent with . Log . Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Log . Fix . Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Remove in server-side. Fix . Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Upgrade from to . Upgrade from to . Fix in server-side. Add fields to . Fix . Address . In previous versions(v3.2.10, v3.2.11), client response size was limited to only 4 MiB. `MaxCallSendMsgSize` default value is 2 MiB, if not configured. `MaxCallRecvMsgSize` default value is `math.MaxInt32`, if not configured. Compile with . <hr> See and for any breaking

Again, before running upgrades from any previous release, please make sure to read change logs below and . Upgrade from to . See for more details. Log . Fix racey grpc-go's server handler transport `WriteStatus` call to prevent . Add to help debug such issues in the future. Remove `--listen-metrics-urls` flag in monitoring document (non-released in `v3.2.x`, planned for `v3.3.x`). Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Upgrade from to . Upgrade from to . See for more details. Revert to support non-wildcard subject alternative names in the certs (see for more contexts). For instance, `etcd --discovery-srv=etcd.local` will only authenticate peers/clients when the provided certs have root domain `etcd.local` (not `*.etcd.local`) as an entry in Subject Alternative Name (SAN) field. Replace backend key-value database `boltdb/bolt` with to address . Rewrite balancer to handle . Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . See for more details. Update `golang.org/x/crypto/bcrypt` (see ). Fix discovery SRV bootstrapping to , in order to support sub-domain wildcard matching (see for more contexts). For instance, `etcd --discovery-srv=etcd.local` will only authenticate peers/clients when the provided certs have root domain `*.etcd.local` as an entry in Subject Alternative Name (SAN) field. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Fix v2 client failover to next endpoint on mutable operation. Handle . Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Fix . Fix . Use store revision from first fetch to resolve write conflicts instead of modified revision. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Fix watch restore from snapshot. Fix multiple URLs for `--listen-peer-urls` flag. Add `--enable-pprof` flag to etcd configuration file format. See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Fix `etcddebuggingmvcckeystotal` inconsistency. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Return non-zero exit code on unhealthy `endpoint health`. See for more details. . For instance, if peer cert contains only DNS names (no IP addresses) in Subject Alternative Name (SAN) field, server first reverse-lookups the remote IP address to get a list of names mapping to that address (e.g. `nslookup IPADDR`). Then accepts the connection if those names have a matching name with peer cert's DNS names (either by exact or wildcard match). If none is matched, server forward-lookups each DNS entry in peer cert (e.g. look up `example.default.svc` when the entry is

and accepts connection only when the host's resolved addresses have the matching IP address with the peer's remote IP address. For example, peer B's CSR (with `cfssl`) SAN field is ` for more detail. See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Fix unreachable `/metrics` endpoint when `--enable-v2=false`. Handle . Add container registry `gcr.io/etcd-development/etcd`. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Do not block on active client stream when stopping server Fix gRPC proxy Snapshot RPC error handling Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Let clients establish unlimited streams Tag docker images with minor versions e.g. `docker pull quay.io/coreos/etcd:v3.2` to fetch latest v3.2 versions Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Rate-limit lease revoke on expiration. Extend leases on promote to avoid queueing effect on lease expiration. See for more details. . For instance, if peer cert contains IP addresses and DNS names in Subject Alternative Name (SAN) field, and the remote IP address matches one of those IP addresses, server just accepts connection without further checking the DNS names. For example, peer B's CSR (with `cfssl`) SAN field is ` for more detail. Accept connection with matched IP SAN but no DNS match. Don't check DNS entries in certs if there's a matching IP. Use user-provided listen address to connect to gRPC gateway. `net.Listener` rewrites IPv4 0.0.0.0 to IPv6 [::], breaking IPv6 disabled hosts. Only v3.2.0, v3.2.1 are affected. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Fix backend database in-memory index corruption issue on restore (only 3.2.0 is affected). Fix Txn marshaling. See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Fix backend database size debugging metrics. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Improve backend read concurrency. Increased . Higher snapshot count means it holds Raft entries in memory for longer before discarding old entries. It is a trade-off between less frequent snapshotting and . User lower `--snapshot-count` value for lower memory usage. User higher `--snapshot-count` value for better availabilities of slow followers (less frequent snapshots from leader). `clientv3.Lease.TimeToLive` returns `LeaseTimeToLiveResponse.TTL == -1` on lease not found. `clientv3.NewFromConfigFile` is moved to `clientv3/yaml.NewConfig`. `embed.Etcd.Peers` field is now `[]*peerListener`. Rejects domains names for `--listen-peer-urls` and `--listen-client-urls` (3.1 only prints out warnings), since . Upgrade from to . Upgrade to . See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Add Add `etcddebuggingserverleaseexpired_total` metrics. See for more details.

This is useful when replacing expiry certs without stopping etcd servers; it can be done by overwriting old certs with new ones. Refreshing certs for every connection should not have too much overhead, but can be improved in the future, with caching layer. Example tests can be found . . For instance, if peer cert contains any IP addresses in Subject Alternative Name (SAN) field, server authenticates a peer only when the remote IP address matches one of those IP addresses. This is to prevent unauthorized endpoints from joining the cluster. For example, peer B's CSR (with `cfssl`) SAN field is `[".example.default.svc", ".example.default.svc.cluster.local", "10.138.0.27"]` when peer B's actual IP address is `10.138.0.2`, not `10.138.0.27`. When peer B tries to join the cluster, peer A will reject B with the error `x509: certificate is valid for 10.138.0.27, not 10.138.0.2`, because B's remote IP address does not match the one in Subject Alternative Name (SAN) field. . For instance, if peer cert contains only DNS names (no IP addresses) in Subject Alternative Name (SAN) field, server authenticates a peer only when forward-lookups (`dig b.com`) on those DNS names have matching IP with the remote IP address. For example, peer B's CSR (with `cfssl`) SAN field is `["b.com"]` when peer B's remote IP address is `10.138.0.2`. When peer B tries to join the cluster, peer A looks up the incoming host `b.com` to get the list of IP addresses (e.g. `dig b.com`). And rejects B if the list does not contain the IP `10.138.0.2`, with the error `tls: 10.138.0.2 does not match any of DNSNames ["b.com"]`. Auth support JWT token. RPCs Add Election, Lock service. Native client `etcdserver/api/v3client` client "embedded" in the server. Logging, monitoring Server warns large snapshot operations. Add `etcd --enable-v2` flag to enable v2 API server. `etcd --enable-v2=true` by default. Add `etcd --auth-token` flag. v3.2 compactor runs . Compactor only supports periodic compaction. Compactor continues to record latest revisions every 5-minute. For every hour, it uses the last revision that was fetched before compaction period, from the revision records that were collected every 5-minute. That is, for every hour, compactor discards historical data created before compaction period. The retention window of compaction period moves to next hour. For instance, when hourly writes are 100 and `--auto-compaction-retention=10`, v3.1 compacts revision 1000, 2000, and 3000 for every 10-hour, while v3.2 compacts revision 1000, 1100, and 1200 for every 1-hour. If compaction succeeds or requested revision has already been compacted, it resets period timer and removes used compacted revision from historical revision records (e.g. start next revision collect and compaction from previously collected revisions). If compaction fails, it retries in 5 minutes. Allow snapshot over 512MB. STM prefetching. Add namespace feature. Add `ErrOldCluster` with server version checking. Translate `WithPrefix()` into `WithFromKey()` for empty key. Add `check perf` command. Add `etcdctl --from-key` flag to role grant-permission command. `lock` command takes an optional command to execute. Proxy endpoint discovery. Namespaces. Coalesce lease requests. Support for . v3 client concurrency package's elections updated to match RPC interfaces. let client dial endpoints not in the balancer. Release Annotate acbuild with supports-systemd-notify. Add `nsswitch.conf` to Docker container image. Add ppc64le, arm64(experimental) builds. Compile with . <hr>