Repository: traefik/traefik (language: go)

Issue:
KV provider returns empty configuration meaning deleting last router/service etc is ignored
### Do you want to request a *feature* or report a *bug*?
Bug

The KV configuration provider will generate what's considered an "empty" configuration when no keys are returned by, e.g, Consul. Configurations that are "empty" are skipped. This means that if the last router or service provided by a KV provider is deleted, the deletion is ignored.

### What did you do?

Define a router and service using the consul provider, or other KV provider. Use the dashboard to see that it is registered in Traefik. Delete the keys from consul so that there are no further keys in consul. Traefik produced a log message about skipping the empty configuration, the route/service remains in traefik


### What did you expect to see?

The router/service should be deleted

### What did you see instead?

The router/service stayed in traefik

### Output of `traefik version`: (_What version of Traefik are you using?_)
```
Version:      dev
Codename:     cheddar
Go version:   go1.16.3
Built:        I don't remember exactly
OS/Arch:      linux/amd64
```

### What is your environment & configuration (arguments, toml, provider, platform, ...)?
Running on Centos 7 under docker

### If applicable, please paste the log output in DEBUG level (`--log.level=DEBUG` switch)

If you use the file provider and create a router/service by putting a file in the configuration directory, and then you delete the file, you get:
```
msg="Configuration received from provider file: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=file
```

If you do the same with consul, instead you get:
```
msg="Configuration received from provider consul: {}" providerName=consul
msg="Skipping empty Configuration for provider consul" providerName=consul
```

The issue appears to be that in the file provider all of the fields in the Configuration are initialised (pkg/provider/file/file.go loadFileConfigFromDirectory). In the KV provider a zero value Configuration is used.



Discussion / hints:
Hello @tomqwpl 

Thanks a lot for using Traefik and reporting the issue. 
Would you please provide the reproducible use case of the issue?  We reproduce each of the issues locally, so having the example will be helpful. 

Thank you, 
I'm not sure what form a reproducible test case would consist of.

You need traefik configured to use, say, consul as a dynamic provider.
You need, say, a router, service or middleware defined in consul such that traefik has picked that up.
You need to delete the router/service/middleware from consul such that there are no further KV pairs in consul under the "traefik/" key. This is the key part, there need to be no further KV pairs under the "traefik/" key.
Traefik ignores the config change and the thing you've deleted from consul remains in traefik.


Would it possible to share a simple docker-compose file with all services and its dependencies?

Thank you, 
I've just been trying to run the integration tests to create a test case that fails, but I don't seem to be able to on Windows, or under WSL2 on windows.

What I was going to try was changing the test case in integration/consul_test.go such that at the end it did something like:

```
	for k := range data {
		err := s.kvClient.Delete(k)
		c.Assert(err, checker.IsNil)
	}
```

It would then somehow have to wait until the configuration change happened. I guess you'd just have to get the rawdata repeatedly until it matches the content of a new file, rawdata-consul-empty.json, where that was rawdata-consul.json with all the consul provided routers, middlewares and services removed. So something like:

```
	err = try.GetRequest("http://127.0.0.1:8080/api/rawdata", 10*time.Second, func(r *http.Response) error {
		var obtained api.RunTimeRepresentation
		err = json.NewDecoder(r.Body).Decode(&obtained)
		c.Assert(err, checker.IsNil)
		got, err := json.MarshalIndent(obtained, "", "  ")
		c.Assert(err, checker.IsNil)

		fmt.Println(string(got))

		expectedJSON := filepath.FromSlash("testdata/rawdata-consul-empty.json")
		expected, err := os.ReadFile(expectedJSON)
		c.Assert(err, checker.IsNil)

		if !bytes.Equal(expected, got) {
			return errors.New("configuration currently differs from expected")
		}
		return nil
	})
	c.Assert(err, checker.IsNil)
```

The file rawdata-consul-empty.json would just be rawdata-consul.json with all of the consul provided services, routers and middlewares removed, so what you ought to get if traefik was started with an empty consul, or any dynamic provider providing no configuration.

As I say, if I could run the integration tests in my environment I would try this out and see if it worked (or rather didn't).

That's the best I can do right now. I don't have access to a "proper" linux environment with suitable go, docker etc on it, I only have Windows and WSL2.

Attempting to run the tests under WSL2 gives me:
```
$ make test-integration
mkdir dist
docker build  -t "traefik-dev:master" -f build.Dockerfile .
[+] Building 238.9s (18/18) FINISHED
 => [internal] load build definition from build.Dockerfile                                                         0.1s
 => => transferring dockerfile: 1.53kB                                                                             0.0s
 => [internal] load .dockerignore                                                                                  0.1s
 => => transferring context: 86B                                                                                   0.0s
 => [internal] load metadata for docker.io/library/golang:1.16-alpine                                              3.0s
 => [auth] library/golang:pull token for registry-1.docker.io                                                      0.0s
 => [internal] load build context                                                                                  4.9s
 => => transferring context: 74.18MB                                                                               4.9s
 => [ 1/12] FROM docker.io/library/golang:1.16-alpine@sha256:49c07aa83790aca732250c2258b5912659df31b6bfa2ab42866  31.5s
 => => resolve docker.io/library/golang:1.16-alpine@sha256:49c07aa83790aca732250c2258b5912659df31b6bfa2ab428661bc  0.0s
 => => sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba 2.81MB / 2.81MB                     1.0s
 => => sha256:adcc1eea9eeabb6de296adb3e0c1b0722cf13251ff3e4e2d0a5f7ed8e3d48342 281.27kB / 281.27kB                 0.8s
 => => sha256:4c4ab2625f07be8d5c6e48046a05ff3ecc7f374b794a926fb62247b66b511909 154B / 154B                         0.8s
 => => sha256:49c07aa83790aca732250c2258b5912659df31b6bfa2ab428661bc66833769e1 1.65kB / 1.65kB                     0.0s
 => => sha256:6600d9933c681cb38c13c2218b474050e6a9a288ac62bdb23aee13bc6dedce18 1.36kB / 1.36kB                     0.0s
 => => sha256:270727b8fd0f89bdcca21bc85c0105b94245b87a2d05436d27ff8d038936dbef 5.16kB / 5.16kB                     0.0s
 => => sha256:0510c868ecb4537a06149c7336217ecc57426cbade1e78dc5f5b9214ce925dab 105.70MB / 105.70MB                26.8s
 => => sha256:afea3b2eda06482098bc605cb2ee7e3170dea8e719423cd084ebb4b8b97fcafc 155B / 155B                         1.0s
 => => extracting sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba                          0.2s
 => => extracting sha256:adcc1eea9eeabb6de296adb3e0c1b0722cf13251ff3e4e2d0a5f7ed8e3d48342                          0.1s
 => => extracting sha256:4c4ab2625f07be8d5c6e48046a05ff3ecc7f374b794a926fb62247b66b511909                          0.0s
 => => extracting sha256:0510c868ecb4537a06149c7336217ecc57426cbade1e78dc5f5b9214ce925dab                          4.2s
 => => extracting sha256:afea3b2eda06482098bc605cb2ee7e3170dea8e719423cd084ebb4b8b97fcafc                          0.0s
 => [ 2/12] RUN apk --update upgrade     && apk --no-cache --no-progress add git mercurial bash gcc musl-dev cur  22.5s
 => [ 3/12] RUN mkdir -p /usr/local/bin     && curl -fL https://download.docker.com/linux/static/stable/x86_64/d  14.0s
 => [ 4/12] RUN mkdir -p /usr/local/bin     && curl -fsSL -o /usr/local/bin/go-bindata https://github.com/contain  2.0s
 => [ 5/12] RUN curl -sfL https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | bash -s -- -b /go  4.5s
 => [ 6/12] RUN  curl -sfL https://raw.githubusercontent.com/client9/misspell/master/install-misspell.sh | bash -  3.2s
 => [ 7/12] RUN curl -sfL https://install.goreleaser.com/github.com/goreleaser/goreleaser.sh | sh                  5.2s
 => [ 8/12] WORKDIR /go/src/github.com/traefik/traefik                                                             0.1s
 => [ 9/12] COPY go.mod .                                                                                          0.1s
 => [10/12] COPY go.sum .                                                                                          0.1s
 => [11/12] RUN GO111MODULE=on GOPROXY=https://proxy.golang.org go mod download                                  125.3s
 => [12/12] COPY . /go/src/github.com/traefik/traefik                                                              0.7s
 => exporting to image                                                                                            26.4s
 => => exporting layers                                                                                           26.3s
 => => writing image sha256:5f4ffc022f5998a920658816474bbe18a0e97fc60e58f436cd924b4474635777                       0.0s
 => => naming to docker.io/library/traefik-dev:master                                                              0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
docker run --add-host=host.docker.internal:127.0.0.1  -e "TEST_CONTAINER=1" -v "/var/run/docker.sock:/var/run/docker.sock"   -e OS_ARCH_ARG -e OS_PLATFORM_ARG -e TESTFLAGS -e VERBOSE -e VERSION -e CODENAME -e TESTDIRS -e CI -e CONTAINER=DOCKER              -v "/mnt/d/work/thirdparty/traefik/"dist":/go/src/github.com/traefik/traefik/"dist"" "traefik-dev:master" ./script/make.sh generate binary test-integration
': No such file or directory
make: *** [Makefile:96: test-integration] Error 127
```
Hello @tomqwpl 

I was able to reproduce the issue that you describe. If you delete the rootKey Traefik and leave consul empty, routers and services that have been created will not be wiped out from Traefik.
Traefik will throw the following log:
```
traefik | time="2021-04-28T14:42:04Z" level=debug msg="WatchTree: traefik"
traefik | time="2021-04-28T14:42:04Z" level=debug msg="List: traefik"
traefik | time="2021-04-28T14:42:05Z" level=error msg="KV connection error: Key not found in store, retrying in 334.339834ms" providerName=consul
```
However, the existing routing rules work correctly and as end-user, you can reach your applications exposed through Traefik. 

We discussed that internally and this is expected behavior because in specific circumstances e.g. network connectivity issue to Consul, removing by mistake rootKey and the entire configuration, the routing works correctly and Traefik is able to do its job. 

Please let us know if you have any other questions concerning that topic, I will be happy to answer you. 
Deleting the root key isn't the circumstance I have in mind. The query to consul is returning the single "traefik/" key and nothing else. It's not returning a "not found". 

Or perhaps I should pose the problem as a question.

I have consul set up as a dynamic provider. It starts out with just the traefik/ key defined, nothing else. This is a valid configuration. No routes, services are defined in traefik.
I add a route and a service by adding appropriate keys in consul. Traefik picks them up. 

How do I return back to the starting state?

Naively, I delete the subkeys from consul, returning it to the initial state where I've just got the "traefik/" root key defined. Traefik ignores the configuration and ignores the deletion change.

I can do this with the "file" provider. 
If I use a directory as my dynamic configuration source and the directory is empty, I have no routes/services defined in traefik. This is the equivalent of just having the "traefik/" key defined.
I put a file in the directory that defines something and traefik picks it up.
I then delete the file and those definitions are removed from traefik, I go back to the starting state of an empty configuration. 
This behaves as I would expect. I don't appear to be able to do the same with consul as the dynamic provider.

To ensure that I can always delete my routes and services I have to ensure that I always have a "dummy" entry in the configuration. I use a middleware for this, as they can exist and don't "do" anything. With this dummy entry in the configuration I can guarantee that if I delete routes and services etc, traefik will always honour those changes. Without that dummy configuration entry in place, I don't believe that there's a way of being able to delete the "last" configuration entry. Traefik treats the resulting configuration returned from the dynamic provider as "empty" and produces the "Skipping empty configuration" message.


Hello @tomqwpl 

Thanks a lot for providing a comprehensive description of the issue. After deeper analysis, we changed the label for `bug/analysis`, currently we are investigating the issue and we will back to you with more details shortly. 

Thanks for using Traefik!



OK, thanks. Sorry it's taken me a while to explain the issue with sufficient clarity. As I said earlier, I know what I'd do to fix it I think, but can't get into a position to be able to build and run tests in order to test it out.

My assumption is that the line:
```
cfg := &dynamic.Configuration{}
```
in the buildConfiguration function in pkg\provider\pk\pv.go should be more like the lines in for example the decodeConfiguration function in pkg\provider\file\file.go, and initialise all of the fields. That way, the isEmptyConfiguration function won't return true and the configuration will be taken notice of.


> Hello @tomqwpl
> 
> Thanks a lot for providing a comprehensive description of the issue. After deeper analysis, we changed the label for `bug/analysis`, currently we are investigating the issue and we will back to you with more details shortly.
> 
> Thanks for using Traefik!

After 2 year are there some news to share? Will this be fixed in Traefik v3?
Hello @flopana,

We have not been able to move forward on it and it looks like time to get more eyes on it. 
If any community member can help us find verify steps to reproduce, we would love the help. 
Hello @nmengin,

I've subscribed this issue, because I'm also affected by this problem. So here is my point of view (maybe it helps):

For me, reproducing this issue is very simple. If you use for example Redis as a provider for the routers, you can simply check the traefik-dashboard for when a router is added or deleted and simply add or remove data from redis. If you now have an empty Redis DB, you can add the first router:
```
traefik/http/routers/router1/rule: 'Host(`example.com`)');
traefik/http/routers/router1/tls/certresolver: 'letsencrypt'
traefik/http/routers/router1/entrypoints: [...]
traefik/http/routers/router1/service: [...]
```

If you now refresh the traefik dashboard you should immediately see router1 poping up.

If you now add a second router to redis:
```
traefik/http/routers/router2/rule: 'Host(`example2.com`)');
traefik/http/routers/router2/tls/certresolver: 'letsencrypt'
traefik/http/routers/router2/entrypoints: [...]
traefik/http/routers/router2/service: [...]
```
and refresh the dashboard you should also see router2 now.

Now if you start moving backward, you can simply delete router2 from redis. So that there is no entry anymore about it in Redis. If you now refresh the dashboard you will immediately see that router2 is vanished from traefik. Which is exactly what we wanted and expected.

But if you now remove router1 from redis too, it is still remaining on the dashboard and also in traefik. Because traefik is still routing correctly, when we use the domain. Even though there is no configuration left in Redis. So everything is deleted, but traefik is still believing that router1 still exists in Redis.

I hope this simple test case is helping you out.

I've also tested several scenarios and come to the same conclusion as @tomqwpl that the problem is based on modifying the last router of the provider. So deleting the last router from the provider is not noticed by traefik.

Kind Regards
Hello @ALaDST,

Thanks for reaching out.
I moved this issue as a bug confirmed.

If any community member can help us fix it, we would love the help.

Fix patch:
diff --git a/integration/resources/compose/consul.yml b/integration/resources/compose/consul.yml
index 068dc9a36f..96860e10c1 100644
--- a/integration/resources/compose/consul.yml
+++ b/integration/resources/compose/consul.yml
@@ -2,6 +2,8 @@ version: "3.8"
 services:
   consul:
     image: consul:1.6
+  whoami:
+    image: traefik/whoami
 
 networks:
   default:
diff --git a/pkg/provider/kv/kv.go b/pkg/provider/kv/kv.go
index 3bbf201f9a..36b4ecdbaa 100644
--- a/pkg/provider/kv/kv.go
+++ b/pkg/provider/kv/kv.go
@@ -136,6 +136,16 @@ func (p *Provider) watchKv(ctx context.Context, configurationChan chan<- dynamic
 func (p *Provider) buildConfiguration(ctx context.Context) (*dynamic.Configuration, error) {
 	pairs, err := p.kvClient.List(ctx, p.RootKey, nil)
 	if err != nil {
+		if errors.Is(err, store.ErrKeyNotFound) {
+			// This empty configuration satisfies the pkg/server/configurationwatcher.go isEmptyConfiguration func constraints,
+			// and will not be discarded by the configuration watcher.
+			return &dynamic.Configuration{
+				HTTP: &dynamic.HTTPConfiguration{
+					Routers: make(map[string]*dynamic.Router),
+				},
+			}, nil
+		}
+
 		return nil, err
 	}
 


Test patch:
diff --git a/integration/consul_test.go b/integration/consul_test.go
index 27c9171d84..b1028d2bb7 100644
--- a/integration/consul_test.go
+++ b/integration/consul_test.go
@@ -4,6 +4,7 @@ import (
 	"bytes"
 	"context"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"net"
 	"net/http"
@@ -28,6 +29,13 @@ type ConsulSuite struct {
 	consulURL string
 }
 
+func (s *ConsulSuite) resetStore(c *check.C) {
+	err := s.kvClient.DeleteTree(context.Background(), "traefik")
+	if err != nil && !errors.Is(err, store.ErrKeyNotFound) {
+		c.Fatal(err)
+	}
+}
+
 func (s *ConsulSuite) setupStore(c *check.C) {
 	s.createComposeProject(c, "consul")
 	s.composeUp(c)
@@ -155,3 +163,71 @@ func (s *ConsulSuite) TestSimpleConfiguration(c *check.C) {
 		c.Error(text)
 	}
 }
+
+func (s *ConsulSuite) assertWhoami(c *check.C, host string, expectedStatusCode int) {
+	req, err := http.NewRequest(http.MethodGet, "http://127.0.0.1:8000", nil)
+	if err != nil {
+		c.Fatal(err)
+	}
+	req.Host = host
+
+	resp, err := try.ResponseUntilStatusCode(req, 15*time.Second, expectedStatusCode)
+	resp.Body.Close()
+	c.Assert(err, checker.IsNil)
+}
+
+func (s *ConsulSuite) TestDeleteRootKey(c *check.C) {
+	// This test case reproduce the issue: https://github.com/traefik/traefik/issues/8092
+	s.setupStore(c)
+	s.resetStore(c)
+
+	file := s.adaptFile(c, "fixtures/consul/simple.toml", struct{ ConsulAddress string }{s.consulURL})
+	defer os.Remove(file)
+
+	ctx := context.Background()
+	svcaddr := net.JoinHostPort(s.getComposeServiceIP(c, "whoami"), "80")
+
+	data := map[string]string{
+		"traefik/http/routers/Router0/entryPoints/0": "web",
+		"traefik/http/routers/Router0/rule":          "Host(`kv1.localhost`)",
+		"traefik/http/routers/Router0/service":       "simplesvc0",
+
+		"traefik/http/routers/Router1/entryPoints/0": "web",
+		"traefik/http/routers/Router1/rule":          "Host(`kv2.localhost`)",
+		"traefik/http/routers/Router1/service":       "simplesvc1",
+
+		"traefik/http/services/simplesvc0/loadBalancer/servers/0/url": "http://" + svcaddr,
+		"traefik/http/services/simplesvc1/loadBalancer/servers/0/url": "http://" + svcaddr,
+	}
+
+	for k, v := range data {
+		err := s.kvClient.Put(ctx, k, []byte(v), nil)
+		c.Assert(err, checker.IsNil)
+	}
+
+	cmd, display := s.traefikCmd(withConfigFile(file))
+	defer display(c)
+	err := cmd.Start()
+	c.Assert(err, checker.IsNil)
+	defer s.killCmd(cmd)
+
+	// wait for traefik
+	err = try.GetRequest("http://127.0.0.1:8080/api/rawdata", 2*time.Second,
+		try.BodyContains(`"Router0@consul":`, `"Router1@consul":`, `"simplesvc0@consul":`, `"simplesvc1@consul":`),
+	)
+	c.Assert(err, checker.IsNil)
+	s.assertWhoami(c, "kv1.localhost", http.StatusOK)
+	s.assertWhoami(c, "kv2.localhost", http.StatusOK)
+
+	// delete router1
+	err = s.kvClient.DeleteTree(ctx, "traefik/http/routers/Router1")
+	c.Assert(err, checker.IsNil)
+	s.assertWhoami(c, "kv1.localhost", http.StatusOK)
+	s.assertWhoami(c, "kv2.localhost", http.StatusNotFound)
+
+	// delete simple services and router0
+	err = s.kvClient.DeleteTree(ctx, "traefik")
+	c.Assert(err, checker.IsNil)
+	s.assertWhoami(c, "kv1.localhost", http.StatusNotFound)
+	s.assertWhoami(c, "kv2.localhost", http.StatusNotFound)
+}