Repository: traefik/traefik (language: go)

Issue:
redis Provider will not update rules when "unexpected EOF" error happened
### Welcome!

- [X] Yes, I've searched similar issues on [GitHub](https://github.com/traefik/traefik/issues) and didn't find any.
- [X] Yes, I've searched similar issues on the [Traefik community forum](https://community.traefik.io) and didn't find any.

### What did you do?

Hi, current we use redis as Provider and update redis rule dynamic. But we found a problem is that, when we update route rule(or any other redis keys) very frenquently, may caused redis "unexpected EOF" error happend. This will cause traefik won't update rules any more.

### What did you see instead?

So I digest the problem from [kvtools/valkerie](https://github.com/kvtools/valkeyrie/blob/0bf7d50195f55baef38afda33940b17248dbd990/store/redis/redis.go#L520;)'s code here. 
```golang
func (s *subscribe) receiveLoop(ctx context.Context, msgCh chan *redis.Message) {
	defer close(msgCh)

	for {
		select {
		case <-s.closeCh:
			return
		case <-ctx.Done():
			return
		default:
			msg, err := s.pubsub.ReceiveMessage(ctx)
			if err != nil {
				return
			}
			if msg != nil {
				msgCh <- msg
			}
		}
	}
}
```
When s.pubsub.ReceiveMessage(ctx) returns error, the goroutine will exited and close the message channel: msgCh.
However the error will not notify the upper user [code here](https://github.com/kvtools/valkeyrie/blob/0bf7d50195f55baef38afda33940b17248dbd990/store/redis/redis.go#L214).
Finally, treafik still waiting the notify from the "events" [channel](https://github.com/traefik/traefik/blob/c9520480c2c70935817bedac1e8a3e9eaacf2f86/pkg/provider/kv/kv.go#L117).

```golang
func (p *Provider) watchKv(ctx context.Context, configurationChan chan<- dynamic.Message) error {
	operation := func() error {
		events, err := p.kvClient.WatchTree(p.RootKey, ctx.Done(), nil)
		if err != nil {
			return fmt.Errorf("failed to watch KV: %w", err)
		}

		for {
			select {
			case <-ctx.Done():
				return nil
			case _, ok := <-events:
				if !ok {
					return errors.New("the WatchTree channel is closed")
				}

				configuration, errC := p.buildConfiguration()
				if errC != nil {
					return errC
				}

				if configuration != nil {
					configurationChan <- dynamic.Message{
						ProviderName:  p.name,
						Configuration: configuration,
					}
				}
			}
		}
	}

	notify := func(err error, time time.Duration) {
		log.FromContext(ctx).Errorf("KV connection error: %+v, retrying in %s", err, time)
	}

	err := backoff.RetryNotify(safe.OperationWithRecover(operation),
		backoff.WithContext(job.NewBackOff(backoff.NewExponentialBackOff()), ctx), notify)
	if err != nil {
		return fmt.Errorf("cannot connect to KV server: %w", err)
	}
	return nil
```

Anyone can fix the problem, or just I create a pull request to fix the problem.

### What version of Traefik are you using?

v2.8.1

### What is your environment & configuration?

```
global:
  checkNewVersion: false
  sendAnonymousUsage: false
api:
  insecure: true
  dashboard: true
entryPoints:
  portal:
    address: ":30888"
porviders:
  providersThrottleDuration: 0.5s
  docker:
    swarmMode: false
    network: true
    exposedByDefault: false
  redis:
    rootKey: 'TRAEFIK_CONFIG'
    endpoints:
      - 'xx.xx.xx.xx:6379'
log:
    level: DEBUG
    format: json
```

Add more configuration information here.


### If applicable, please paste the log output in DEBUG level

_No response_


Discussion / hints:
Hello @bstr9,

Thanks for reporting this!

After taking a look at the code you pointed out, we think that the bug is related to valkeyrie and not Traefik.
It looks like the `watchCh` chan should be close here https://github.com/kvtools/valkeyrie/blob/0bf7d50195f55baef38afda33940b17248dbd990/store/redis/redis.go#L211.

> Anyone can fix the problem, or just I create a pull request to fix the problem.

As this is not related to Traefik, we are closing this issue, but we gladly welcome any contribution, so please open a pull request on the kvtools/valkeyrie side to fix the issue.
After a second thought, we are reopening this issue to keep track of that bug, until of the update of the valkeyrie dependency once https://github.com/kvtools/valkeyrie/pull/70 will be merged.
Thanks @rtribotte. It seems fixed now.
Hello @bstr9,

I am reopening the issue to keep track of this because until the update of the dependency to valkeyrie the problem is not yet fixed for Traefik.

Fix patch:
diff --git a/go.mod b/go.mod
index 0a867b8bf6..0e7ce8eed6 100644
--- a/go.mod
+++ b/go.mod
@@ -7,7 +7,7 @@ require (
 	github.com/ExpediaDotCom/haystack-client-go v0.0.0-20190315171017-e7edbdf53a61
 	github.com/Masterminds/sprig/v3 v3.2.2
 	github.com/abbot/go-http-auth v0.0.0-00010101000000-000000000000
-	github.com/aws/aws-sdk-go v1.39.0
+	github.com/aws/aws-sdk-go v1.44.47
 	github.com/cenkalti/backoff/v4 v4.1.1
 	github.com/compose-spec/compose-go v1.0.3
 	github.com/containous/alice v0.0.0-20181107144136-d83ebdd94cbd
@@ -36,7 +36,7 @@ require (
 	github.com/influxdata/influxdb1-client v0.0.0-20191209144304-8bf82d3c094d
 	github.com/instana/go-sensor v1.38.3
 	github.com/klauspost/compress v1.14.2
-	github.com/kvtools/valkeyrie v0.4.1
+	github.com/kvtools/valkeyrie v0.4.2-0.20220810161836-a9a70ee3f199
 	github.com/lucas-clemente/quic-go v0.28.1
 	github.com/mailgun/ttlmap v0.0.0-20170619185759-c1c17f74874f
 	github.com/miekg/dns v1.1.47
@@ -135,6 +135,7 @@ require (
 	github.com/cpu/goacmedns v0.1.1 // indirect
 	github.com/deepmap/oapi-codegen v1.8.2 // indirect
 	github.com/dgraph-io/ristretto v0.1.0 // indirect
+	github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
 	github.com/dimchansky/utfbom v1.1.1 // indirect
 	github.com/distribution/distribution/v3 v3.0.0-20210316161203-a01c71e2477e // indirect
 	github.com/dnsimple/dnsimple-go v0.70.1 // indirect
@@ -158,6 +159,7 @@ require (
 	github.com/go-errors/errors v1.0.1 // indirect
 	github.com/go-logfmt/logfmt v0.5.1 // indirect
 	github.com/go-logr/logr v0.4.0 // indirect
+	github.com/go-redis/redis/v8 v8.11.5 // indirect
 	github.com/go-resty/resty/v2 v2.1.1-0.20191201195748-d7b97669fe48 // indirect
 	github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0 // indirect
 	github.com/go-zookeeper/zk v1.0.2 // indirect
@@ -258,7 +260,7 @@ require (
 	github.com/nrdcg/namesilo v0.2.1 // indirect
 	github.com/nrdcg/porkbun v0.1.1 // indirect
 	github.com/nxadm/tail v1.4.8 // indirect
-	github.com/onsi/ginkgo v1.16.4 // indirect
+	github.com/onsi/ginkgo v1.16.5 // indirect
 	github.com/opencontainers/go-digest v1.0.0 // indirect
 	github.com/opencontainers/image-spec v1.0.2 // indirect
 	github.com/opencontainers/runc v1.0.2 // indirect
@@ -299,9 +301,9 @@ require (
 	github.com/xeipuuv/gojsonschema v1.2.0 // indirect
 	go.elastic.co/apm/module/apmhttp v1.13.1 // indirect
 	go.elastic.co/fastjson v1.1.0 // indirect
-	go.etcd.io/etcd/api/v3 v3.5.0 // indirect
-	go.etcd.io/etcd/client/pkg/v3 v3.5.0 // indirect
-	go.etcd.io/etcd/client/v3 v3.5.0 // indirect
+	go.etcd.io/etcd/api/v3 v3.5.4 // indirect
+	go.etcd.io/etcd/client/pkg/v3 v3.5.4 // indirect
+	go.etcd.io/etcd/client/v3 v3.5.4 // indirect
 	go.opencensus.io v0.23.0 // indirect
 	go.uber.org/atomic v1.7.0 // indirect
 	go.uber.org/multierr v1.6.0 // indirect
@@ -311,7 +313,7 @@ require (
 	golang.org/x/lint v0.0.0-20210508222113-6edffad5e616 // indirect
 	golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b // indirect
 	golang.org/x/sync v0.0.0-20210220032951-036812b2e83c // indirect
-	golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a // indirect
+	golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e // indirect
 	golang.org/x/term v0.0.0-20210927222741-03fcf44c2211 // indirect
 	golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
 	google.golang.org/api v0.44.0 // indirect
@@ -321,7 +323,6 @@ require (
 	gopkg.in/inf.v0 v0.9.1 // indirect
 	gopkg.in/ini.v1 v1.62.0 // indirect
 	gopkg.in/ns1/ns1-go.v2 v2.6.2 // indirect
-	gopkg.in/redis.v5 v5.2.9 // indirect
 	gopkg.in/square/go-jose.v2 v2.6.0 // indirect
 	gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect
 	gopkg.in/yaml.v2 v2.4.0 // indirect
diff --git a/go.sum b/go.sum
index f05106f2e5..4187850350 100644
--- a/go.sum
+++ b/go.sum
@@ -238,8 +238,8 @@ github.com/aws/aws-sdk-go v1.25.41/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpi
 github.com/aws/aws-sdk-go v1.27.0/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
 github.com/aws/aws-sdk-go v1.34.9/go.mod h1:5zCpMtNQVjRREroY7sYe8lOMRSxkhG6MZveU8YkpAk0=
 github.com/aws/aws-sdk-go v1.34.28/go.mod h1:H7NKnBqNVzoTJpGfLrQkkD+ytBA93eiDYi/+8rV9s48=
-github.com/aws/aws-sdk-go v1.39.0 h1:74BBwkEmiqBbi2CGflEh34l0YNtIibTjZsibGarkNjo=
-github.com/aws/aws-sdk-go v1.39.0/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro=
+github.com/aws/aws-sdk-go v1.44.47 h1:uyiNvoR4wfZ8Bp4ghgbyzGFIg5knjZMUAd5S9ba9qNU=
+github.com/aws/aws-sdk-go v1.44.47/go.mod h1:y4AeaBuwd2Lk+GepC1E9v0qOiTws0MIWAX4oIKwKHZo=
 github.com/aws/aws-sdk-go-v2 v0.18.0/go.mod h1:JWVYvqSMppoMJC0x5wdwiImzgXTI9FuZwxzkQq9wy+g=
 github.com/aws/aws-sdk-go-v2 v1.0.0/go.mod h1:smfAbmpW+tcRVuNUjo3MOArSZmW72t62rkCzc2i0TWM=
 github.com/aws/aws-sdk-go-v2/config v1.0.0/go.mod h1:WysE/OpUgE37tjtmtJd8GXgT8s1euilE5XtUkRNUQ1w=
@@ -514,6 +514,7 @@ github.com/dgrijalva/jwt-go v0.0.0-20170104182250-a601269ab70c/go.mod h1:E3ru+11
 github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
 github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2 h1:tdlZCpZ/P9DhczCTSixgIKmwPv6+wP5DGjqLYw5SUiA=
 github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw=
+github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
 github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
 github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
 github.com/digitalocean/godo v1.1.1/go.mod h1:h6faOIcZ8lWIwNQ+DN7b3CgX4Kwby5T+nbpNqkUIozU=
@@ -719,6 +720,8 @@ github.com/go-playground/validator/v10 v10.4.1/go.mod h1:nlOn6nFhuKACm19sB/8EGNn
 github.com/go-redis/redis v6.15.9+incompatible/go.mod h1:NAIEuMOZ/fxfXJIrKDQDz8wamY7mA7PouImQ2Jvg6kA=
 github.com/go-redis/redis/v7 v7.1.0/go.mod h1:JDNMw23GTyLNC4GZu9njt15ctBQVn7xjRfnwdHj/Dcg=
 github.com/go-redis/redis/v8 v8.0.0/go.mod h1:isLoQT/NFSP7V67lyvM9GmdvLdyZ7pEhsXvvyQtnQTo=
+github.com/go-redis/redis/v8 v8.11.5 h1:AcZZR7igkdvfVmQTPnu9WE37LRrO/YrBH5zWyjDC0oI=
+github.com/go-redis/redis/v8 v8.11.5/go.mod h1:gREzHqY1hg6oD9ngVRbLStwAWKhA0FEgq8Jd4h5lpwo=
 github.com/go-resty/resty/v2 v2.1.1-0.20191201195748-d7b97669fe48 h1:JVrqSeQfdhYRFk24TvhTZWU0q8lfCojxZQFi3Ou7+uY=
 github.com/go-resty/resty/v2 v2.1.1-0.20191201195748-d7b97669fe48/go.mod h1:dZGr0i9PLlaaTD4H/hoZIDjQ+r6xq8mgbRzHZf7f2J8=
 github.com/go-sql-driver/mysql v1.3.0/go.mod h1:zAC/RDZ24gD3HViQzih4MyKcchzm+sOG5ZlKdlhCg5w=
@@ -1269,8 +1272,8 @@ github.com/kr/pty v1.1.8/go.mod h1:O1sed60cT9XZ5uDucP5qwvh+TE3NnUj51EiZO/lmSfw=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
 github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
 github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
-github.com/kvtools/valkeyrie v0.4.1 h1:S0lOF4OOmPFd1i37vHmgCVr7/2ygwL4grKbHT5vGQ6M=
-github.com/kvtools/valkeyrie v0.4.1/go.mod h1:E34+bty7IqLoFkOqGD9AHDE4Bw4APJtoKmj0cqJJ7ug=
+github.com/kvtools/valkeyrie v0.4.2-0.20220810161836-a9a70ee3f199 h1:SF+ZjtFyUPxkkCLYowtWzTeEMeziEKdW0Qhg9siq9b8=
+github.com/kvtools/valkeyrie v0.4.2-0.20220810161836-a9a70ee3f199/go.mod h1:6gM2lOTGIFQxcIioo3/EEeOS0knFB8+AFKydVIN/nAU=
 github.com/kylelemons/go-gypsy v0.0.0-20160905020020-08cad365cd28/go.mod h1:T/T7jsxVqf9k/zYOqbgNAsANsjxTd1Yq3htjDhQ1H0c=
 github.com/kylelemons/godebug v0.0.0-20170820004349-d65d576e9348/go.mod h1:B69LEHPfb2qLo0BaaOLcbitczOKLWTsrBG9LczfCD4k=
 github.com/labbsr0x/bindman-dns-webhook v1.0.2 h1:I7ITbmQPAVwrDdhd6dHKi+MYJTJqPCK0jE6YNBAevnk=
@@ -1520,8 +1523,9 @@ github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108
 github.com/onsi/ginkgo v1.14.0/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY=
 github.com/onsi/ginkgo v1.14.1/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY=
 github.com/onsi/ginkgo v1.16.2/go.mod h1:CObGmKUOKaSC0RjmoAK7tKyn4Azo5P2IWuoMnvwxz1E=
-github.com/onsi/ginkgo v1.16.4 h1:29JGrr5oVBm5ulCWet69zQkzWipVXIol6ygQUe/EzNc=
 github.com/onsi/ginkgo v1.16.4/go.mod h1:dX+/inL/fNMqNlz0e9LfyB9TswhZpCVdJM/Z6Vvnwo0=
+github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE=
+github.com/onsi/ginkgo v1.16.5/go.mod h1:+E8gABHa3K6zRBolWtd+ROzc/U5bkGt0FwiG042wbpU=
 github.com/onsi/ginkgo/v2 v2.0.0/go.mod h1:vw5CSIxN1JObi/U8gcbwft7ZxR2dgaR70JSE3/PpL4c=
 github.com/onsi/ginkgo/v2 v2.1.3/go.mod h1:vw5CSIxN1JObi/U8gcbwft7ZxR2dgaR70JSE3/PpL4c=
 github.com/onsi/gomega v0.0.0-20151007035656-2152b45fa28a/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA=
@@ -1649,6 +1653,7 @@ github.com/prometheus/client_golang v1.3.0/go.mod h1:hJaj2vgQTGQmVCsAACORcieXFeD
 github.com/prometheus/client_golang v1.4.0/go.mod h1:e9GMxYsXl05ICDXkRhurwBS4Q3OK1iX/F2sw+iXX5zU=
 github.com/prometheus/client_golang v1.7.1/go.mod h1:PY5Wy2awLA44sXw4AOSfFBetzPP4j5+D6mVACh+pe2M=
 github.com/prometheus/client_golang v1.11.0/go.mod h1:Z6t4BnS23TR94PD6BsDNk8yVqroYurpAkEiz0P2BEV0=
+github.com/prometheus/client_golang v1.11.1/go.mod h1:Z6t4BnS23TR94PD6BsDNk8yVqroYurpAkEiz0P2BEV0=
 github.com/prometheus/client_golang v1.12.1/go.mod h1:3Z9XVyYiZYEO+YQWt3RD2R3jrbd179Rt297l4aS6nDY=
 github.com/prometheus/client_golang v1.12.2-0.20220704083116-e8f91604d835 h1:sYuFGkrz0PtewSFk0Bg7p7jjiiklc6FUIWz+mFGQfD0=
 github.com/prometheus/client_golang v1.12.2-0.20220704083116-e8f91604d835/go.mod h1:RjnYTcBFM8s+WRft6oBqj4p5OgXJASPw5UFiI7w+GSs=
@@ -2018,13 +2023,15 @@ go.etcd.io/bbolt v1.3.3/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
 go.etcd.io/bbolt v1.3.5/go.mod h1:G5EMThwa9y8QZGBClrRx5EY+Yw9kAhnjy3bSjsnlVTQ=
 go.etcd.io/etcd v0.0.0-20191023171146-3cf2f69b5738/go.mod h1:dnLIgRNXwCJa5e+c6mIZCrds/GIG4ncV9HhK5PX7jPg=
 go.etcd.io/etcd v0.5.0-alpha.5.0.20200910180754-dd1b699fc489/go.mod h1:yVHk9ub3CSBatqGNg7GRmsnfLWtoW60w4eDYfh7vHDg=
-go.etcd.io/etcd/api/v3 v3.5.0 h1:GsV3S+OfZEOCNXdtNkBSR7kgLobAa/SO6tCxRa0GAYw=
 go.etcd.io/etcd/api/v3 v3.5.0/go.mod h1:cbVKeC6lCfl7j/8jBhAK6aIYO9XOjdptoxU/nLQcPvs=
-go.etcd.io/etcd/client/pkg/v3 v3.5.0 h1:2aQv6F436YnN7I4VbI8PPYrBhu+SmrTaADcf8Mi/6PU=
+go.etcd.io/etcd/api/v3 v3.5.4 h1:OHVyt3TopwtUQ2GKdd5wu3PmmipR4FTwCqoEjSyRdIc=
+go.etcd.io/etcd/api/v3 v3.5.4/go.mod h1:5GB2vv4A4AOn3yk7MftYGHkUfGtDHnEraIjym4dYz5A=
 go.etcd.io/etcd/client/pkg/v3 v3.5.0/go.mod h1:IJHfcCEKxYu1Os13ZdwCwIUTUVGYTSAM3YSwc9/Ac1g=
+go.etcd.io/etcd/client/pkg/v3 v3.5.4 h1:lrneYvz923dvC14R54XcA7FXoZ3mlGZAgmwhfm7HqOg=
+go.etcd.io/etcd/client/pkg/v3 v3.5.4/go.mod h1:IJHfcCEKxYu1Os13ZdwCwIUTUVGYTSAM3YSwc9/Ac1g=
 go.etcd.io/etcd/client/v2 v2.305.0/go.mod h1:h9puh54ZTgAKtEbut2oe9P4L/oqKCVB6xsXlzd7alYQ=
-go.etcd.io/etcd/client/v3 v3.5.0 h1:62Eh0XOro+rDwkrypAGDfgmNh5Joq+z+W9HZdlXMzek=
-go.etcd.io/etcd/client/v3 v3.5.0/go.mod h1:AIKXXVX/DQXtfTEqBryiLTUXwON+GuvO6Z7lLS/oTh0=
+go.etcd.io/etcd/client/v3 v3.5.4 h1:p83BUL3tAYS0OT/r0qglgc3M1JjhM0diV8DSWAhVXv4=
+go.etcd.io/etcd/client/v3 v3.5.4/go.mod h1:ZaRkVgBZC+L+dLCjTcF1hRXpgZXQPOvnA/Ak/gq3kiY=
 go.mongodb.org/mongo-driver v1.5.1/go.mod h1:gRXCHX4Jo7J0IJ1oDQyUxF7jfy19UfxniMS4xxMmUqw=
 go.mozilla.org/pkcs7 v0.0.0-20200128120323-432b2356ecb1/go.mod h1:SNgMg+EgDFwmvSmLRTNKC5fegJjB7v23qTQ0XLGUNHk=
 go.opencensus.io v0.18.0/go.mod h1:vKdFvxhtzZ9onBp9VKHK8z/sRpBMnKAsufL7wlDrCOA=
@@ -2393,8 +2400,9 @@ golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBc
 golang.org/x/sys v0.0.0-20220114195835-da31bd327af9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220209214540-3681064d5158/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220227234510-4e6760a101f9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a h1:dGzPydgVsqGcTRVwiLJ1jVbufYwmzD3LfVPLKsKg+0k=
 golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e h1:CsOuNlbOuf0mzxJIefr6Q4uAUetRUwZE4qt7VfzP+xo=
+golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
@@ -2708,8 +2716,6 @@ gopkg.in/ns1/ns1-go.v2 v2.6.2 h1:tC+gRSN6fmnb9l9cVrIysXyuRO0wV6cZrjDqlMB0gGc=
 gopkg.in/ns1/ns1-go.v2 v2.6.2/go.mod h1:GMnKY+ZuoJ+lVLL+78uSTjwTz2jMazq6AfGKQOYhsPk=
 gopkg.in/olivere/elastic.v3 v3.0.75/go.mod h1:yDEuSnrM51Pc8dM5ov7U8aI/ToR3PG0llA8aRv2qmw0=
 gopkg.in/olivere/elastic.v5 v5.0.84/go.mod h1:LXF6q9XNBxpMqrcgax95C6xyARXWbbCXUrtTxrNrxJI=
-gopkg.in/redis.v5 v5.2.9 h1:MNZYOLPomQzZMfpN3ZtD1uyJ2IDonTTlxYiV/pEApiw=
-gopkg.in/redis.v5 v5.2.9/go.mod h1:6gtv0/+A4iM08kdRfocWYB3bLX2tebpNtfKlFT6H4mY=
 gopkg.in/resty.v1 v1.12.0/go.mod h1:mDo4pnntr5jdWRML875a/NmxYqAlA73dVijT2AXvQQo=
 gopkg.in/square/go-jose.v2 v2.2.2/go.mod h1:M9dMgbHiYLoDGQrXy7OpJDJWiKiU//h+vD76mk0e1AI=
 gopkg.in/square/go-jose.v2 v2.3.1/go.mod h1:M9dMgbHiYLoDGQrXy7OpJDJWiKiU//h+vD76mk0e1AI=
diff --git a/integration/try/condition.go b/integration/try/condition.go
index 7325360a91..66c70a899d 100644
--- a/integration/try/condition.go
+++ b/integration/try/condition.go
@@ -1,6 +1,7 @@
 package try
 
 import (
+	"context"
 	"errors"
 	"fmt"
 	"io"
@@ -185,7 +186,7 @@ type DoCondition func() error
 // Verify if a Key exists in the store.
 func KVExists(kv store.Store, key string) DoCondition {
 	return func() error {
-		_, err := kv.Exists(key, nil)
+		_, err := kv.Exists(context.Background(), key, nil)
 		return err
 	}
 }
diff --git a/pkg/provider/kv/kv.go b/pkg/provider/kv/kv.go
index 173c49698a..4ced1362a1 100644
--- a/pkg/provider/kv/kv.go
+++ b/pkg/provider/kv/kv.go
@@ -67,7 +67,7 @@ func (p *Provider) Provide(configurationChan chan<- dynamic.Message, pool *safe.
 	logger := log.FromContext(ctx)
 
 	operation := func() error {
-		if _, err := p.kvClient.Exists(path.Join(p.RootKey, "qmslkjdfmqlskdjfmqlksjazçueznbvbwzlkajzebvkwjdcqmlsfj"), nil); err != nil {
+		if _, err := p.kvClient.Exists(ctx, path.Join(p.RootKey, "qmslkjdfmqlskdjfmqlksjazçueznbvbwzlkajzebvkwjdcqmlsfj"), nil); err != nil {
 			return fmt.Errorf("KV store connection error: %w", err)
 		}
 		return nil
@@ -76,12 +76,12 @@ func (p *Provider) Provide(configurationChan chan<- dynamic.Message, pool *safe.
 	notify := func(err error, time time.Duration) {
 		logger.Errorf("KV connection error: %+v, retrying in %s", err, time)
 	}
-	err := backoff.RetryNotify(safe.OperationWithRecover(operation), job.NewBackOff(backoff.NewExponentialBackOff()), notify)
+	err := backoff.RetryNotify(safe.OperationWithRecover(operation), backoff.WithContext(job.NewBackOff(backoff.NewExponentialBackOff()), ctx), notify)
 	if err != nil {
 		return fmt.Errorf("cannot connect to KV server: %w", err)
 	}
 
-	configuration, err := p.buildConfiguration()
+	configuration, err := p.buildConfiguration(ctx)
 	if err != nil {
 		logger.Errorf("Cannot build the configuration: %v", err)
 	} else {
@@ -105,7 +105,7 @@ func (p *Provider) Provide(configurationChan chan<- dynamic.Message, pool *safe.
 
 func (p *Provider) watchKv(ctx context.Context, configurationChan chan<- dynamic.Message) error {
 	operation := func() error {
-		events, err := p.kvClient.WatchTree(p.RootKey, ctx.Done(), nil)
+		events, err := p.kvClient.WatchTree(ctx, p.RootKey, nil)
 		if err != nil {
 			return fmt.Errorf("failed to watch KV: %w", err)
 		}
@@ -119,7 +119,7 @@ func (p *Provider) watchKv(ctx context.Context, configurationChan chan<- dynamic
 					return errors.New("the WatchTree channel is closed")
 				}
 
-				configuration, errC := p.buildConfiguration()
+				configuration, errC := p.buildConfiguration(ctx)
 				if errC != nil {
 					return errC
 				}
@@ -146,8 +146,8 @@ func (p *Provider) watchKv(ctx context.Context, configurationChan chan<- dynamic
 	return nil
 }
 
-func (p *Provider) buildConfiguration() (*dynamic.Configuration, error) {
-	pairs, err := p.kvClient.List(p.RootKey, nil)
+func (p *Provider) buildConfiguration(ctx context.Context) (*dynamic.Configuration, error) {
+	pairs, err := p.kvClient.List(ctx, p.RootKey, nil)
 	if err != nil {
 		return nil, err
 	}
@@ -190,7 +190,7 @@ func (p *Provider) createKVClient(ctx context.Context) (store.Store, error) {
 		redis.Register()
 	}
 
-	kvStore, err := valkeyrie.NewStore(p.storeType, p.Endpoints, storeConfig)
+	kvStore, err := valkeyrie.NewStore(ctx, p.storeType, p.Endpoints, storeConfig)
 	if err != nil {
 		return nil, err
 	}
diff --git a/pkg/provider/kv/storewrapper.go b/pkg/provider/kv/storewrapper.go
index 7f46c878c0..2a7b9aad71 100644
--- a/pkg/provider/kv/storewrapper.go
+++ b/pkg/provider/kv/storewrapper.go
@@ -1,6 +1,8 @@
 package kv
 
 import (
+	"context"
+
 	"github.com/kvtools/valkeyrie/store"
 	"github.com/traefik/traefik/v2/pkg/log"
 )
@@ -9,110 +11,110 @@ type storeWrapper struct {
 	store.Store
 }
 
-func (s *storeWrapper) Put(key string, value []byte, options *store.WriteOptions) error {
+func (s *storeWrapper) Put(ctx context.Context, key string, value []byte, options *store.WriteOptions) error {
 	log.WithoutContext().Debugf("Put: %s", key, string(value))
 
 	if s.Store == nil {
 		return nil
 	}
-	return s.Store.Put(key, value, options)
+	return s.Store.Put(ctx, key, value, options)
 }
 
-func (s *storeWrapper) Get(key string, options *store.ReadOptions) (*store.KVPair, error) {
+func (s *storeWrapper) Get(ctx context.Context, key string, options *store.ReadOptions) (*store.KVPair, error) {
 	log.WithoutContext().Debugf("Get: %s", key)
 
 	if s.Store == nil {
 		return nil, nil
 	}
-	return s.Store.Get(key, options)
+	return s.Store.Get(ctx, key, options)
 }
 
-func (s *storeWrapper) Delete(key string) error {
+func (s *storeWrapper) Delete(ctx context.Context, key string) error {
 	log.WithoutContext().Debugf("Delete: %s", key)
 
 	if s.Store == nil {
 		return nil
 	}
-	return s.Store.Delete(key)
+	return s.Store.Delete(ctx, key)
 }
 
-func (s *storeWrapper) Exists(key string, options *store.ReadOptions) (bool, error) {
+func (s *storeWrapper) Exists(ctx context.Context, key string, options *store.ReadOptions) (bool, error) {
 	log.WithoutContext().Debugf("Exists: %s", key)
 
 	if s.Store == nil {
 		return true, nil
 	}
-	return s.Store.Exists(key, options)
+	return s.Store.Exists(ctx, key, options)
 }
 
-func (s *storeWrapper) Watch(key string, stopCh <-chan struct{}, options *store.ReadOptions) (<-chan *store.KVPair, error) {
+func (s *storeWrapper) Watch(ctx context.Context, key string, options *store.ReadOptions) (<-chan *store.KVPair, error) {
 	log.WithoutContext().Debugf("Watch: %s", key)
 
 	if s.Store == nil {
 		return nil, nil
 	}
-	return s.Store.Watch(key, stopCh, options)
+	return s.Store.Watch(ctx, key, options)
 }
 
-func (s *storeWrapper) WatchTree(directory string, stopCh <-chan struct{}, options *store.ReadOptions) (<-chan []*store.KVPair, error) {
+func (s *storeWrapper) WatchTree(ctx context.Context, directory string, options *store.ReadOptions) (<-chan []*store.KVPair, error) {
 	log.WithoutContext().Debugf("WatchTree: %s", directory)
 
 	if s.Store == nil {
 		return nil, nil
 	}
-	return s.Store.WatchTree(directory, stopCh, options)
+	return s.Store.WatchTree(ctx, directory, options)
 }
 
-func (s *storeWrapper) NewLock(key string, options *store.LockOptions) (store.Locker, error) {
+func (s *storeWrapper) NewLock(ctx context.Context, key string, options *store.LockOptions) (store.Locker, error) {
 	log.WithoutContext().Debugf("NewLock: %s", key)
 
 	if s.Store == nil {
 		return nil, nil
 	}
-	return s.Store.NewLock(key, options)
+	return s.Store.NewLock(ctx, key, options)
 }
 
-func (s *storeWrapper) List(directory string, options *store.ReadOptions) ([]*store.KVPair, error) {
+func (s *storeWrapper) List(ctx context.Context, directory string, options *store.ReadOptions) ([]*store.KVPair, error) {
 	log.WithoutContext().Debugf("List: %s", directory)
 
 	if s.Store == nil {
 		return nil, nil
 	}
-	return s.Store.List(directory, options)
+	return s.Store.List(ctx, directory, options)
 }
 
-func (s *storeWrapper) DeleteTree(directory string) error {
+func (s *storeWrapper) DeleteTree(ctx context.Context, directory string) error {
 	log.WithoutContext().Debugf("DeleteTree: %s", directory)
 
 	if s.Store == nil {
 		return nil
 	}
-	return s.Store.DeleteTree(directory)
+	return s.Store.DeleteTree(ctx, directory)
 }
 
-func (s *storeWrapper) AtomicPut(key string, value []byte, previous *store.KVPair, options *store.WriteOptions) (bool, *store.KVPair, error) {
+func (s *storeWrapper) AtomicPut(ctx context.Context, key string, value []byte, previous *store.KVPair, options *store.WriteOptions) (bool, *store.KVPair, error) {
 	log.WithoutContext().Debugf("AtomicPut: %s", key, string(value), previous)
 
 	if s.Store == nil {
 		return true, nil, nil
 	}
-	return s.Store.AtomicPut(key, value, previous, options)
+	return s.Store.AtomicPut(ctx, key, value, previous, options)
 }
 
-func (s *storeWrapper) AtomicDelete(key string, previous *store.KVPair) (bool, error) {
+func (s *storeWrapper) AtomicDelete(ctx context.Context, key string, previous *store.KVPair) (bool, error) {
 	log.WithoutContext().Debugf("AtomicDelete: %s", key, previous)
 
 	if s.Store == nil {
 		return true, nil
 	}
-	return s.Store.AtomicDelete(key, previous)
+	return s.Store.AtomicDelete(ctx, key, previous)
 }
 
-func (s *storeWrapper) Close() {
+func (s *storeWrapper) Close() error {
 	log.WithoutContext().Debugf("Close")
 
 	if s.Store == nil {
-		return
+		return nil
 	}
-	s.Store.Close()
+	return s.Store.Close()
 }


Test patch:
diff --git a/integration/consul_test.go b/integration/consul_test.go
index 59c661a573..7037ab46c5 100644
--- a/integration/consul_test.go
+++ b/integration/consul_test.go
@@ -2,6 +2,7 @@ package integration
 
 import (
 	"bytes"
+	"context"
 	"encoding/json"
 	"fmt"
 	"net"
@@ -36,6 +37,7 @@ func (s *ConsulSuite) setupStore(c *check.C) {
 
 	consul.Register()
 	kv, err := valkeyrie.NewStore(
+		context.Background(),
 		store.CONSUL,
 		[]string{consulAddr},
 		&store.Config{
@@ -105,7 +107,7 @@ func (s *ConsulSuite) TestSimpleConfiguration(c *check.C) {
 	}
 
 	for k, v := range data {
-		err := s.kvClient.Put(k, []byte(v), nil)
+		err := s.kvClient.Put(context.Background(), k, []byte(v), nil)
 		c.Assert(err, checker.IsNil)
 	}
 
diff --git a/integration/etcd_test.go b/integration/etcd_test.go
index 3b7b2bca2e..8f0630181a 100644
--- a/integration/etcd_test.go
+++ b/integration/etcd_test.go
@@ -2,6 +2,7 @@ package integration
 
 import (
 	"bytes"
+	"context"
 	"encoding/json"
 	"net"
 	"net/http"
@@ -35,6 +36,7 @@ func (s *EtcdSuite) SetUpSuite(c *check.C) {
 	var err error
 	s.etcdAddr = net.JoinHostPort(s.getComposeServiceIP(c, "etcd"), "2379")
 	s.kvClient, err = valkeyrie.NewStore(
+		context.Background(),
 		store.ETCDV3,
 		[]string{s.etcdAddr},
 		&store.Config{
@@ -101,7 +103,7 @@ func (s *EtcdSuite) TestSimpleConfiguration(c *check.C) {
 	}
 
 	for k, v := range data {
-		err := s.kvClient.Put(k, []byte(v), nil)
+		err := s.kvClient.Put(context.Background(), k, []byte(v), nil)
 		c.Assert(err, checker.IsNil)
 	}
 
diff --git a/integration/redis_test.go b/integration/redis_test.go
index 520fefdf33..859d5498a1 100644
--- a/integration/redis_test.go
+++ b/integration/redis_test.go
@@ -2,6 +2,7 @@ package integration
 
 import (
 	"bytes"
+	"context"
 	"encoding/json"
 	"net"
 	"net/http"
@@ -33,6 +34,7 @@ func (s *RedisSuite) setupStore(c *check.C) {
 	s.redisAddr = net.JoinHostPort(s.getComposeServiceIP(c, "redis"), "6379")
 	redis.Register()
 	kv, err := valkeyrie.NewStore(
+		context.Background(),
 		store.REDIS,
 		[]string{s.redisAddr},
 		&store.Config{
@@ -102,7 +104,7 @@ func (s *RedisSuite) TestSimpleConfiguration(c *check.C) {
 	}
 
 	for k, v := range data {
-		err := s.kvClient.Put(k, []byte(v), nil)
+		err := s.kvClient.Put(context.Background(), k, []byte(v), nil)
 		c.Assert(err, checker.IsNil)
 	}
 
diff --git a/integration/zk_test.go b/integration/zk_test.go
index 77f904e2ee..04ebbe5b2b 100644
--- a/integration/zk_test.go
+++ b/integration/zk_test.go
@@ -2,6 +2,7 @@ package integration
 
 import (
 	"bytes"
+	"context"
 	"encoding/json"
 	"net"
 	"net/http"
@@ -36,6 +37,7 @@ func (s *ZookeeperSuite) setupStore(c *check.C) {
 
 	var err error
 	s.kvClient, err = valkeyrie.NewStore(
+		context.Background(),
 		store.ZK,
 		[]string{s.zookeeperAddr},
 		&store.Config{
@@ -104,7 +106,7 @@ func (s *ZookeeperSuite) TestSimpleConfiguration(c *check.C) {
 	}
 
 	for k, v := range data {
-		err := s.kvClient.Put(k, []byte(v), nil)
+		err := s.kvClient.Put(context.Background(), k, []byte(v), nil)
 		c.Assert(err, checker.IsNil)
 	}
 
diff --git a/pkg/provider/kv/kv_mock_test.go b/pkg/provider/kv/kv_mock_test.go
index aa74f7c6fc..14c45a6c86 100644
--- a/pkg/provider/kv/kv_mock_test.go
+++ b/pkg/provider/kv/kv_mock_test.go
@@ -1,6 +1,7 @@
 package kv
 
 import (
+	"context"
 	"errors"
 	"strings"
 
@@ -41,11 +42,11 @@ func newKvClientMock(kvPairs []*store.KVPair, err error) *Mock {
 	return mock
 }
 
-func (s *Mock) Put(key string, value []byte, opts *store.WriteOptions) error {
+func (s *Mock) Put(ctx context.Context, key string, value []byte, opts *store.WriteOptions) error {
 	return errors.New("method Put not supported")
 }
 
-func (s *Mock) Get(key string, options *store.ReadOptions) (*store.KVPair, error) {
+func (s *Mock) Get(ctx context.Context, key string, options *store.ReadOptions) (*store.KVPair, error) {
 	if err := s.Error.Get; err != nil {
 		return nil, err
 	}
@@ -57,12 +58,12 @@ func (s *Mock) Get(key string, options *store.ReadOptions) (*store.KVPair, error
 	return nil, store.ErrKeyNotFound
 }
 
-func (s *Mock) Delete(key string) error {
+func (s *Mock) Delete(ctx context.Context, key string) error {
 	return errors.New("method Delete not supported")
 }
 
 // Exists mock.
-func (s *Mock) Exists(key string, options *store.ReadOptions) (bool, error) {
+func (s *Mock) Exists(ctx context.Context, key string, options *store.ReadOptions) (bool, error) {
 	if err := s.Error.Get; err != nil {
 		return false, err
 	}
@@ -75,22 +76,22 @@ func (s *Mock) Exists(key string, options *store.ReadOptions) (bool, error) {
 }
 
 // Watch mock.
-func (s *Mock) Watch(key string, stopCh <-chan struct{}, options *store.ReadOptions) (<-chan *store.KVPair, error) {
+func (s *Mock) Watch(ctx context.Context, key string, options *store.ReadOptions) (<-chan *store.KVPair, error) {
 	return nil, errors.New("method Watch not supported")
 }
 
 // WatchTree mock.
-func (s *Mock) WatchTree(prefix string, stopCh <-chan struct{}, options *store.ReadOptions) (<-chan []*store.KVPair, error) {
+func (s *Mock) WatchTree(ctx context.Context, prefix string, options *store.ReadOptions) (<-chan []*store.KVPair, error) {
 	return s.WatchTreeMethod(), nil
 }
 
 // NewLock mock.
-func (s *Mock) NewLock(key string, options *store.LockOptions) (store.Locker, error) {
+func (s *Mock) NewLock(ctx context.Context, key string, options *store.LockOptions) (store.Locker, error) {
 	return nil, errors.New("method NewLock not supported")
 }
 
 // List mock.
-func (s *Mock) List(prefix string, options *store.ReadOptions) ([]*store.KVPair, error) {
+func (s *Mock) List(ctx context.Context, prefix string, options *store.ReadOptions) ([]*store.KVPair, error) {
 	if err := s.Error.List; err != nil {
 		return nil, err
 	}
@@ -104,19 +105,21 @@ func (s *Mock) List(prefix string, options *store.ReadOptions) ([]*store.KVPair,
 }
 
 // DeleteTree mock.
-func (s *Mock) DeleteTree(prefix string) error {
+func (s *Mock) DeleteTree(ctx context.Context, prefix string) error {
 	return errors.New("method DeleteTree not supported")
 }
 
 // AtomicPut mock.
-func (s *Mock) AtomicPut(key string, value []byte, previous *store.KVPair, opts *store.WriteOptions) (bool, *store.KVPair, error) {
+func (s *Mock) AtomicPut(ctx context.Context, key string, value []byte, previous *store.KVPair, opts *store.WriteOptions) (bool, *store.KVPair, error) {
 	return false, nil, errors.New("method AtomicPut not supported")
 }
 
 // AtomicDelete mock.
-func (s *Mock) AtomicDelete(key string, previous *store.KVPair) (bool, error) {
+func (s *Mock) AtomicDelete(ctx context.Context, key string, previous *store.KVPair) (bool, error) {
 	return false, errors.New("method AtomicDelete not supported")
 }
 
 // Close mock.
-func (s *Mock) Close() {}
+func (s *Mock) Close() error {
+	return nil
+}
diff --git a/pkg/provider/kv/kv_test.go b/pkg/provider/kv/kv_test.go
index a7097755ff..535a48bab6 100644
--- a/pkg/provider/kv/kv_test.go
+++ b/pkg/provider/kv/kv_test.go
@@ -283,7 +283,7 @@ func Test_buildConfiguration(t *testing.T) {
 		"traefik/tls/certificates/1/stores/1":                                                        "foobar",
 	}))
 
-	cfg, err := provider.buildConfiguration()
+	cfg, err := provider.buildConfiguration(context.Background())
 	require.NoError(t, err)
 
 	expected := &dynamic.Configuration{
@@ -929,7 +929,7 @@ func Test_buildConfiguration_KV_error(t *testing.T) {
 		},
 	}
 
-	cfg, err := provider.buildConfiguration()
+	cfg, err := provider.buildConfiguration(context.Background())
 	require.Error(t, err)
 	assert.Nil(t, cfg)
 }