Repository: traefik/traefik (language: go)

Issue:
Provide a hook to have Traefik live-reload its configuration
Based from [this](https://github.com/containous/traefik/issues/7#issuecomment-281486988) post, this issue is a feature request for the ability to utilize [ConfigMap](https://kubernetes.io/docs/user-guide/configmap/) in k8s for Traefik configuration with the added ability to support live reloading on changes.

This ask is to help not require users to painfully interact with the toml file but instead use a k8s native construct in which they can merely write the ConfigMap with the needed options, and deploy it as part of their already existent manifests (or in a [helm](https://github.com/kubernetes/helm) chart etc.)


Discussion / hints:
As far as I am aware Kube doesn't push changes made to config maps to containers. I recently read over a proposal for Kube to support reloading containers when changes happen to things like ConfigMaps.
Hmm I thought I had seen a few applications that already supported this. I can't think of them atm but if I come across them again, I'll share.

Could you post a link to the proposal you're talking about; I'd like to follow the updates.
Here is the main one: https://github.com/kubernetes/kubernetes/issues/22368

It's not actually the one I was thinking of but covers the same thing. It spawns from the idea of just trying to do a rolling update of a config map. Lots of the comments though talk just about wanting to update a config map, don't necessarily need rolling updates.

The one I was thinking of, after finding it, was actually about daemonsets: https://github.com/kubernetes/features/issues/124
If you do know of something else that does update on ConfigMap updates I would love to see how they do it.
iirc, it depends in part on how the configmap is consumed. If your manifest injects the values of specific configmap keys into environment variables, those are _not_ updated when the configmap is altered, however, if the configmap in question is mounted into the file system, the file(s) contents do change as the configmap is altered. If a process running inside such a container watched those files for changes, it could reload configuration each time a configmap is altered.
I was unaware of that. If Kube is updating the file then that is very different.
FWIW, I use https://github.com/fabric8io/configmapcontroller to do a rolling update of the Deployment when I update my config. So far, so good.

Thanks for sharing that. Looks like they monitor the kube event stream and take action based on that.
There's also https://github.com/jimmidyson/configmap-reload which watches mounted volume directories and notifies target processes through a web-hook.

If Traefik supported an HTTP-based trigger (which could be handy outside of Kubernetes too I imagine), this would be another route to go, presumably less heavy-weight than a complete Pod restart.
Somewhat related: #1183.
The Kube provider already watches the kube events. If Traefik had a way to reload the config file, and Kube actually does update the config files when config maps change, then that would be the best way to trigger it I think. Lots of "ifs" in there though
An argument speaking in favor of implementing a native, built-in file watcher is that multiple providers would potentially be able to benefit, not just Kube.
Yea, very true.
Been looking and I don't see anything that indicates Kube updates files based on ConfigMap changes.
@Regner [This response](https://github.com/kubernetes/kubernetes/issues/25418#issuecomment-231388101)  to an upstream bug report seems to indicate that volumes backed by ConfigMaps are indeed updated.
@timoreimann huh, awesome! The docs don't seem to hint at that in any way. Not complaining that it seems to work.

So then yea, seems the best thing would be to have traefik just natively support the ability to watch its config file and reload it on change.
Can confirm that kube will update files on ConfigMap changes (usually takes 30-60s). This is from experience working with kube.
If this is in fact the case, I will personally assist on this challenge. This has been one of the biggest issues I have had to date, having to restart pods when a simple configmap change is made.

I will fire up minikube tomorrow and start testing.
There's also a bit of [a workaround](https://github.com/kubernetes/helm/blob/master/docs/charts_tips_and_tricks.md#automatically-roll-deployments-when-configmaps-or-secrets-change) if using Helm although live reload is generally going to be better than bumping the entire Deployment.
Right, usually the configmap gets mounted as volume and AFAIK k8s doesn't support signaling processes in that case. https://github.com/jimmidyson/configmap-reload uses inotify to watch for changes which could be done in traefik as well: https://github.com/jimmidyson/configmap-reload/blob/master/configmap-reload.go#L46
FWIW, I prefer the approach that uses an HTTP endpoint to trigger the reload. For certificates, etc, I may want to write several files to a directory, then trigger the restart.
@dtomcej What's the status of this issue? Have you done any progress?
Has anyone tried SIGHUP? It works when reloading rsyslog config, but I think an API hook would be better, because some people use Træfik in HA.
@psalaberria002 no one is actively working on this issue. Feel free to take ownership if you'd like to drive it further.

@gaui as of today, SIGHUP simply kills Traefik. See also #1660, which is somewhat related to this issue.
Hot-reloading of the static configuration requires a good understanding of the system, and it doesn't sound like an easy task. Not sure I can help at the moment. Hopefully someone else can take it. It would be a very nice feature.
Signal Use Case:
Backend File configurations on NFS, GlusterFS or any other system that does not support inotify.

@gaui Pardon my ignorance, are there instances of Træfik in HA without a Key-Value Store?  If that's the case, should sending a signal/api call to a worker 'fik (my shorthand for a Træfik process in an Træfik cluster, CC with attribution) attempt to contact the manager 'fik and distribute the reload command cluster-wide?
@jnovack Good question, probably... Docker configs and Swarm or ConfigMap in k8s.
Would love to have this feature - we have Traefik v1.7 set up in Azure Service Fabric and we want Traefik to force reload and to discover services immediately when we set traefik.enable to true. Are there any further updates to this?
A very useful feature ,hope it can be implemented
Is there any progress on this?

It would really be convenient to send a SIGHUP or SIGUSR to traefik to reload its dynamic conf.

A restart is quick, but with a large enough userbase you still get page loads dropouts and short certificate errors which cause a lot of confusion without need.
it's really a pity, that this flaw doesn't see any improvements. its a fundamental limitation, which makes it very hard to integrate traffic in an acceptable clean manner an more k8s friendly setups. if you look e.g. at [gloo](https://github.com/solo-io/gloo) and some other envoy based control planes, you'll soon get an idea, how this can be solved in much more suitable manner -- although this other approaches also come with obvious drawbacks (missing ACME integration, complicated Ingress/Gatway incompatibilities etc.) which are handled in much more user friendly fashion by traefik. nevertheless i would see this particular deficit as such a grave issue, that i can hardly value and recommend traefik as more then just a highly unsatisfying compromise on this platform.
@mash-graz i'am not sure what you are ranting about :)

 - toml files are automatically live reloaded on change as long as you keep them flat in one folder. This is valid for router, middleware and  services
 - if you are on K8s you might want to use etcd/confd/consul or similar kv storages supported by traefik to solve the issue even more elegantly

So to be honest, i'am not even sure what this feature is all about or why it is open, since configuration reload for router/middleware/services is already part of traefik core and done automatically for you, why even overcomplicate by using a hook at all?

If it is about using Filesystems without inotify support, not willing to use a KV backend for dynamic setups like K8s i really ask how much technical flaws you want to enforce on a system to then handle for you.

Conceptually Traefik offers both, inotify based storage (FS) or if you cannot use it for live reload since you picked a cluster-storage which does not support inotify, then _use a cluster storage_ for values which have been build for exactly confiiguration, which is a KV of any sorts, which Traefik does again, support natively with all the reloads you have.

Maybe i just do not get the point here - for me this seems like the classic "if you use the right tool for the job, you solve other issues automatically". In this case it seems like using a KV would solve the other issues right on spot

i see two main issues, which are inextricably linked to this flaw:

1. if you use k8s, configmaps and other file based beside the common resource describtion and exchange formats are always more or less unsatisfying and just a compromise to support legacy software. this becomes very obvious if you e.g. want to handle a system just by minimalist GitOps principles and utilize a small batch of helm and yaml manifests to roll out new instances e.g. by the [k3s helm/manifest support](https://rancher.com/docs/k3s/latest/en/helm/#deploying-manifests-and-helm-charts). sure, you can wrap and deploy configmap content also by this kind of exchange file format, but there will always be a gap resp. duplication and error prone redundancy between this import helpers and a system state, which actually operates by utilizing different means and mechanisms.

this brings me to:

2. one of the most obvious consequences of this design can be seen its unavoidable centralized paradigm. it's nearly impossible to e.g. deploy any workload, which need additional entry points to serve TCP or UDP on some service specific well known ports, just by applying some application specific manifests to the  cluster -- no, you also have to modify a kind of config file by other means, because it's simply not properly integrated into the sounding ecosystem and its specific mode of operation. this also concerns the question of delegation of setup tasks and access privileges. 

don't get me wrong. for sure i see the positive advantages of traefik as well. it's a really nice and very user friendly solution, which doesn't waste to much computing resources and tries to make many challenges very simple. but it's nevertheless a pity, when grave flaws like this do not become solved or significantly improved over a very long period of time.
You consider a config-map change on a cluster-wide filessystem storage as a decentralized way of dealing with the problem? You call this a "non centralized paradigm" where Traefik reads it's configuration from a storage which is centralized in the entire ecosystem?

IMHO this centralized storage makes your setup obviously centralized (as usual, since it is the single point of truth for configuration), as every load balancer is - by design. The LB is the centralized POF which needs to be as redundant as possible, since it is the edge.

That said, it is just consequential and consistent to require a consul ( or alikes ) to be the cluster-based storage of a Traefik edge setup - not only because the below ( easy decoupled changes ) but also to the nature on how such a persistence layer can be properly clustered to be safe of single point of failure itself.

What you are talking about is decoupling service-configuration FROM Traefik, so passing configuration downstream from a new service itself,  to "something/some storage"(which seems to be FS only in your idea) and then Traefik picks it up, not knowing who stored it or why, Traefik just follows the "order". This is perfectly fine and possible with FS based storages using toml when FS events are present and much more even using alikes like consul, which have been specifically build for such scenarios in the first place ( at least in consul the KV part, we could also talk about the service registry in this regard in addition, but that's just more offtopic )

Usually i tend to say, every one picks his weapons and as far as you solve your issue, it's all up to you to do it your way. But in this case i at least see, that your enforced paradigm requires a feature, which is not there, and you define that this is the "core flaw of Traefik" - did you consider how many people are using traefik for k3/k8, swarm, rancher, custom setups, legacy setups and so on, without complaining about this issue being somewhat of a deal breaker? 

Constructive suggestion: 
Maybe instead of waiting for this feature ensure your concept cannot be changed to be still convenient for you and solve this issue right along the way.

Or of course, if it is not possible to be done, maybe consider a different LB, e.g. based on nginx where this kind of reload is possible (by the way, nginx -s reload does by no means reload the entire configuration, only parts. As Traefik, it will not allow specific parts to get hot-swap changed ).

Third option is to wait for this feature - one that is open for more then 3 years and usually this means, it will never happen - but those are just bets.

The last option is, writing your own provider, which uses a FS storage  and lets you reload manually.

> IMHO this centralized storage makes your setup obviously centralized (as usual, since it is the single point of truth for configuration), as every load balancer is - by design. The LB is the centralized POF which needs to be as redundant as possible, since it is the edge.

just take a look, how this is actually solved by a bunch of more advanced solutions:

https://docs.solo.io/gloo/latest/guides/traffic_management/listener_configuration/
it'd be sooooo useful to be able to reload the config after adding an entrypoint without killing pods - nginx has supported this for about a squillion years
[File provider are live reloaded](https://docs.traefik.io/providers/file/) and the Docker provider also support live reload for Traefiks side. A Container itself can't be live changed which is a limitation of Docker.
file provider allows live reloading of dynamic configuration, but not static :-(
> file provider allows live reloading of dynamic configuration, but not static :-(

What do you mean? What's dynamic configuration and what's static configuration?
> What do you mean? What's dynamic configuration and what's static configuration?

see:
https://docs.traefik.io/v2.2/getting-started/configuration-overview/

the main issue concerning this fundamental split between staic vs. dynamic config options resp. the limited capabilities of the latter ones in traefik has to be seen in the fact, that you are simply not able to set some very important routing settings without restarting traefik -- e.g. a  RAW TCP/UDP port forwarding instruction, as it will be necessary to host non-HTTP internet services behind your gateway, can not be realized in traefik right now without editing this central static configuration and a restart of the whole ingress system. that's a very unsatisfying resp. unacceptable behavior, if you want to utilize traefik in front of a kubernetes clusters and provide/configure such services in a more suitable manner (i.e. without any restart and and in a more decentralized/delegated form of configuration handling).
Thank you for the full explanation: I get it now.
I've achieved this by setting an annotation on the deployment which contains a hash of my static config. The changing annotation forces Kubernetes to perform a roll-out of the Traefik deployment, which it does safely for HA environments. I deploy Kubernetes resources using Terraform, so it's easy to chain the ConfigMap containing my static configuration together with the Deployment in order to achieve this.

This is an environment-specific workaround, but I've done similar things in Docker swarm, and there could be tricks to achieve this in other environments too.
In Kubernetes and Docker clusters multiple replicas and proxies can make dynamic reloads less critical. Traefik works well for load balancing non-container workloads as well, but needs fewer restarts to work for more critical things.

We're trying to build an on-prem load balancer that works similar to something you'd normally use purpose built hardware for (including automatic IP failover). The services behind it will include lots of TCP and UDP services and multiple, dynamic IP addresses. We need to minimize restarts since a lot of critical traffic will flow through whatever we build as part of the traffic here includes keeping our client network operational (at a university with lots of clients).

The only feature that Traefik is missing to fit in this use case is the ability to modify entrypoints without hard restarts (binding and unbinding). The nginx method of reloading works perfectly, but we'd love to use Traefik since we're already using it so many other places in our environment.
@pcfens take a look at https://github.com/traefik/traefik/pull/7370
Thanks @ldez - that's something we need to consider too. In some cases we'll be mapping an IP/port combination directly to a server on the backend (one entrypoint, one router, one service with one server). When we need to add a new mapping (including a new IP address) we don't want to have to restart everything.
Live-reload through handling some signal (either SIGHUP or SIGUSR) would make this generally useful.

Fix patch:
diff --git a/pkg/provider/file/file.go b/pkg/provider/file/file.go
index ad19f573ee..6ddc310848 100644
--- a/pkg/provider/file/file.go
+++ b/pkg/provider/file/file.go
@@ -6,8 +6,10 @@ import (
 	"errors"
 	"fmt"
 	"os"
+	"os/signal"
 	"path/filepath"
 	"strings"
+	"syscall"
 	"text/template"
 
 	"github.com/Masterminds/sprig/v3"
@@ -48,6 +50,8 @@ func (p *Provider) Init() error {
 // Provide allows the file provider to provide configurations to traefik
 // using the given configuration channel.
 func (p *Provider) Provide(configurationChan chan<- dynamic.Message, pool *safe.Pool) error {
+	logger := log.With().Str(logs.ProviderName, providerName).Logger()
+
 	if p.Watch {
 		var watchItem string
 
@@ -57,48 +61,44 @@ func (p *Provider) Provide(configurationChan chan<- dynamic.Message, pool *safe.
 		case len(p.Filename) > 0:
 			watchItem = filepath.Dir(p.Filename)
 		default:
-			return errors.New("error using file configuration provider, neither filename or directory defined")
+			return errors.New("error using file configuration provider, neither filename nor directory is defined")
 		}
 
-		if err := p.addWatcher(pool, watchItem, configurationChan, p.watcherCallback); err != nil {
+		if err := p.addWatcher(pool, watchItem, configurationChan, p.applyConfiguration); err != nil {
 			return err
 		}
 	}
 
-	configuration, err := p.BuildConfiguration()
-	if err != nil {
-		if p.Watch {
-			log.Error().
-				Str(logs.ProviderName, providerName).
-				Err(err).
-				Msg("Error while building configuration (for the first time)")
+	pool.GoCtx(func(ctx context.Context) {
+		signals := make(chan os.Signal, 1)
+		signal.Notify(signals, syscall.SIGHUP)
+
+		for {
+			select {
+			case <-ctx.Done():
+				return
+			// signals only receives SIGHUP events.
+			case <-signals:
+				if err := p.applyConfiguration(configurationChan); err != nil {
+					logger.Error().Err(err).Msg("Error while building configuration")
+				}
+			}
+		}
+	})
 
+	if err := p.applyConfiguration(configurationChan); err != nil {
+		if p.Watch {
+			logger.Err(err).Msg("Error while building configuration (for the first time)")
 			return nil
 		}
+
 		return err
 	}
 
-	sendConfigToChannel(configurationChan, configuration)
 	return nil
 }
 
-// BuildConfiguration loads configuration either from file or a directory
-// specified by 'Filename'/'Directory' and returns a 'Configuration' object.
-func (p *Provider) BuildConfiguration() (*dynamic.Configuration, error) {
-	ctx := log.With().Str(logs.ProviderName, providerName).Logger().WithContext(context.Background())
-
-	if len(p.Directory) > 0 {
-		return p.loadFileConfigFromDirectory(ctx, p.Directory, nil)
-	}
-
-	if len(p.Filename) > 0 {
-		return p.loadFileConfig(ctx, p.Filename, true)
-	}
-
-	return nil, errors.New("error using file configuration provider, neither filename or directory defined")
-}
-
-func (p *Provider) addWatcher(pool *safe.Pool, directory string, configurationChan chan<- dynamic.Message, callback func(chan<- dynamic.Message, fsnotify.Event)) error {
+func (p *Provider) addWatcher(pool *safe.Pool, directory string, configurationChan chan<- dynamic.Message, callback func(chan<- dynamic.Message) error) error {
 	watcher, err := fsnotify.NewWatcher()
 	if err != nil {
 		return fmt.Errorf("error creating file watcher: %w", err)
@@ -111,6 +111,7 @@ func (p *Provider) addWatcher(pool *safe.Pool, directory string, configurationCh
 
 	// Process events
 	pool.GoCtx(func(ctx context.Context) {
+		logger := log.With().Str(logs.ProviderName, providerName).Logger()
 		defer watcher.Close()
 		for {
 			select {
@@ -121,39 +122,50 @@ func (p *Provider) addWatcher(pool *safe.Pool, directory string, configurationCh
 					_, evtFileName := filepath.Split(evt.Name)
 					_, confFileName := filepath.Split(p.Filename)
 					if evtFileName == confFileName {
-						callback(configurationChan, evt)
+						err := callback(configurationChan)
+						if err != nil {
+							logger.Error().Err(err).Msg("Error occurred during watcher callback")
+						}
 					}
 				} else {
-					callback(configurationChan, evt)
+					err := callback(configurationChan)
+					if err != nil {
+						logger.Error().Err(err).Msg("Error occurred during watcher callback")
+					}
 				}
 			case err := <-watcher.Errors:
-				log.Error().Str(logs.ProviderName, providerName).Err(err).Msg("Watcher event error")
+				logger.Error().Err(err).Msg("Watcher event error")
 			}
 		}
 	})
 	return nil
 }
 
-func (p *Provider) watcherCallback(configurationChan chan<- dynamic.Message, event fsnotify.Event) {
-	watchItem := p.Filename
-	if len(p.Directory) > 0 {
-		watchItem = p.Directory
+// applyConfiguration builds the configuration and sends it to the given configurationChan.
+func (p *Provider) applyConfiguration(configurationChan chan<- dynamic.Message) error {
+	configuration, err := p.buildConfiguration()
+	if err != nil {
+		return err
 	}
 
-	logger := log.With().Str(logs.ProviderName, providerName).Logger()
+	sendConfigToChannel(configurationChan, configuration)
+	return nil
+}
+
+// buildConfiguration loads configuration either from file or a directory
+// specified by 'Filename'/'Directory' and returns a 'Configuration' object.
+func (p *Provider) buildConfiguration() (*dynamic.Configuration, error) {
+	ctx := log.With().Str(logs.ProviderName, providerName).Logger().WithContext(context.Background())
 
-	if _, err := os.Stat(watchItem); err != nil {
-		logger.Error().Err(err).Msgf("Unable to watch %s", watchItem)
-		return
+	if len(p.Directory) > 0 {
+		return p.loadFileConfigFromDirectory(ctx, p.Directory, nil)
 	}
 
-	configuration, err := p.BuildConfiguration()
-	if err != nil {
-		logger.Error().Err(err).Msg("Error occurred during watcher callback")
-		return
+	if len(p.Filename) > 0 {
+		return p.loadFileConfig(ctx, p.Filename, true)
 	}
 
-	sendConfigToChannel(configurationChan, configuration)
+	return nil, errors.New("error using file configuration provider, neither filename nor directory is defined")
 }
 
 func sendConfigToChannel(configurationChan chan<- dynamic.Message, configuration *dynamic.Configuration) {


Test patch: