Project: etcd (Orchestration & Management / Coordination & Service Discovery)
File: CHANGELOG-3.1.md

Previous change logs can be found at . <hr> with etcdctl v2 Add . Previously, . The command output is changed. Previously, if endpoint is unreachable, the command output is "\<endpoint\> is unhealthy: failed to connect: \<error message\>". This change unified the error message, all error types now have the same output "\<endpoint\> is unhealthy: failed to commit proposal: \<error message\>". See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Fix bug where . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Improve , which indicates message send to a peer failed. Improve , which indicates that local node might have slow network. Add ; enable `etcd --debug` flag to see per-request debug information. Add . If consistency check on snapshot file fails, `snapshot status` returns `"snapshot file integrity check failed..."` error. See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Improve Prometheus metric to track leader heartbeats. Previously, it only samples the TCP connection for snapshot messages. Display all registered . Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Fix logic on in `clientv3/concurrency` package. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Improve . See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Add Prometheus metric. Add Prometheus metric. Add Prometheus metric. Use it with `etcdmvccdbtotalsizeinbytes` and `etcdmvccdbtotalsizeinuseinbytes`. `etcdserverquotabackendbytes 2.147483648e+09` means current quota size is 2 GB. `etcdmvccdbtotalsizeinbytes 20480` means current physically allocated DB size is 20 KB. `etcdmvccdbtotalsizeinuseinbytes 16384` means future DB size if defragment operation is complete. `etcdmvccdbtotalsizeinbytes - etcdmvccdbtotalsizeinuseinbytes` is the number of bytes that can be saved on disk with defragment operation. Add Prometheus metric. In addition to . Add Prometheus metric. Use it with `etcdmvccdbtotalsizeinbytes` and `etcdmvccdbtotalsizeinuseinbytes`. `etcdserverquotabackendbytes 2.147483648e+09` means current quota size is 2 GB. `etcdmvccdbtotalsizeinbytes 20480` means current physically allocated DB size is 20 KB. `etcdmvccdbtotalsizeinuseinbytes 16384` means future DB size if defragment operation is complete. `etcdmvccdbtotalsizeinbytes - etcdmvccdbtotalsizeinuseinbytes` is the number of bytes that can be saved on disk with defragment operation. Fix . If `<-chan *clientv3LeaseKeepAliveResponse` from `clientv3.Lease.KeepAlive` was never consumed or channel is full, client was instead of expected rate of every "TTL / 3" duration. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Add Prometheus metric. To replace . Compile with . <hr> See and for any breaking

Again, before running upgrades from any previous release, please make sure to read change logs below and . Fix . A follower receives a leader snapshot to be persisted as a `[SNAPSHOT-INDEX].snap.db` file on disk. Now, server . Otherwise, index mismatch happens and triggers server-side panic (e.g. newer WAL entry with outdated snapshot index). Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Fix . Let's assume that a watcher had been requested with a future revision X and sent to node A that became network-partitioned thereafter. Meanwhile, cluster makes progress. Then when the partition gets removed, the leader sends a snapshot to node A. Previously if the snapshot's latest revision is still lower than the watch revision X, etcd server panicked during snapshot restore operation. Now, this server-side panic has been fixed. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Purge old . Previously, etcd did not respect `--max-snapshots` flag to purge old `*.snap.db` files. Now, etcd purges old `*.snap.db` files to keep maximum `--max-snapshots` number of files on disk. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Add Prometheus metric. Add flag to configure initial election tick fast-forward. By default, `--initial-election-tick-advance=true`, then local member fast-forwards election ticks to speed up "initial" leader election trigger. This benefits the case of larger election ticks. For instance, cross datacenter deployment may require longer election timeout of 10-second. If true, local node does not need wait up to 10-second. Instead, forwards its election ticks to 8-second, and have only 2-second left before leader election. Major assumptions are that: cluster has no active leader thus advancing ticks enables faster leader election. Or cluster already has an established leader, and rejoining follower is likely to receive heartbeats from the leader after tick advance and before election timeout. However, when network from leader to rejoining follower is congested, and the follower does not receive leader heartbeat within left election ticks, disruptive election has to happen thus affecting cluster availabilities. Now, this can be disabled by setting `--initial-election-tick-advance=false`. Disabling this would slow down initial bootstrap process for cross datacenter deployments. Make tradeoffs by configuring `--initial-election-tick-advance` at the cost of slow initial bootstrap. If single-node, it advances ticks regardless. Address . Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Adjust to reduce . Previously, etcd fast-forwards election ticks on server start, with only one tick left for leader election. This is to speed up start phase, without having to wait until all election ticks elapse. Advancing election ticks is useful for cross datacenter deployments with larger election

However, it was affecting cluster availability if the last tick elapses before leader contacts the restarted node. Now, when etcd restarts, it adjusts election ticks with more than one tick left, thus more time for leader to prevent disruptive restart. See for all metrics per release. Note that any `etcddebugging*` metrics are experimental and subject to change. Add missing . Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Fix . "unsynced" watcher is watcher that needs to be in sync with events that have happened. That is, "unsynced" watcher is the slow watcher that was requested on old revision. "unsynced" watcher restore operation was not correctly populating its underlying watcher group. Which possibly causes . A node gets network partitioned with a watcher on a future revision, and falls behind receiving a leader snapshot after partition gets removed. When applying this snapshot, etcd watch storage moves current synced watchers to unsynced since sync watchers might have become stale during network partition. And reset synced watcher group to restart watcher routines. Previously, there was a bug when moving from synced watcher group to unsynced, thus client would miss events when the watcher was requested to the network-partitioned node. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . , backport "mvcc: sending events after restore" , backport coreos/bbolt v1.3.1-coreos.5 Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Tag docker images with minor versions. e.g. `docker pull quay.io/coreos/etcd:v3.1` to fetch latest v3.1 versions. Compile with . Fix panic on `net/http.CloseNotify` <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Allow v2 snapshot over 512MB. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Fill in Auth API response header. Remove auth check in Status API. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Fix raft memory leak issue. Fix Windows file path issues. Add `/etc/nsswitch.conf` file to alpine-based Docker image. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Compile with . <hr> See and for any breaking

Again, before running upgrades from any previous release, please make sure to read change logs below and . Fix `etcd gateway` schema handling in DNS discovery. Fix sd_notify behaviors in `gateway`, `grpc-proxy`. Fix sd_notify behaviors in `gateway`, `grpc-proxy`. Use machine default host when advertise URLs are default values(`localhost:2379,2380`) AND if listen URL is `0.0.0.0`. Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Fix `etcd gateway` with multiple endpoints. Use IPv4 default host, by default (when IPv4 and IPv6 are available). Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Compile with . <hr> See and for any breaking changes. Again, before running upgrades from any previous release, please make sure to read change logs below and . Faster linearizable reads (implements Raft ). v3 authentication API is now stable. Deprecated following gRPC metrics in favor of . `etcdgrpcrequests_total` `etcdgrpcrequestsfailedtotal` `etcdgrpcactive_streams` `etcdgrpcunaryrequestsduration_seconds` Upgrade to , and . See for more details. SRV records (e.g., infra1.example.com) must match the discovery domain (i.e., example.com) if no custom certificate authority is given. `TLSConfig.ServerName` is ignored with user-provided certificates for backwards compatibility; to be deprecated. For example, `etcd --discovery-srv=example.com` will only authenticate peers/clients when the provided certs have root domain `example.com` as an entry in Subject Alternative Name (SAN) field. Automatic leadership transfer when leader steps down. etcd flags `--strict-reconfig-check` flag is set by default. Add `--log-output` flag. Add `--metrics` flag. etcd uses default route IP if advertise URL is not given. Cluster rejects removing members if quorum will be lost. Discovery now has upper limit for waiting on retries. Warn on binding listeners through domain names; to be deprecated. v3.0 and v3.1 with `--auto-compaction-retention=10` run periodic compaction on v3 key-value store for every 10-hour. Compactor only supports periodic compaction. Compactor records latest revisions every 5-minute, until it reaches the first compaction period (e.g. 10-hour). In order to retain key-value history of last compaction period, it uses the last revision that was fetched before compaction period, from the revision records that were collected every 5-minute. When `--auto-compaction-retention=10`, compactor uses revision 100 for compact revision where revision 100 is the latest revision fetched from 10 hours ago. If compaction succeeds or requested revision has already been compacted, it resets period timer and starts over with new historical revision records (e.g. restart revision collect and compact for the next 10-hour period). If compaction fails, it retries in 5 minutes. Add `SetEndpoints` method; update endpoints at runtime. Add `Sync` method; auto-update endpoints at runtime. Add `Lease TimeToLive` API; fetch lease information. replace Config.Logger field with global logger. Get API responses are sorted in ascending order by default. Add `lease timetolive` command. Add `--print-value-only` flag to get command. Add `--dest-prefix` flag to make-mirror command. `get` command responses are sorted in ascending order by default. Experimental gRPC proxy feature. `recipes` now conform to sessions defined in `clientv3/concurrency`. ACI has symlinks to `/usr/local/bin/etcd*`. Compile with . <hr>