Repository: traefik/traefik (language: go)

Issue:
Need a way to define or alter the traefik `service name` based on consul tags
### Welcome!

- [X] Yes, I've searched similar issues on [GitHub](https://github.com/traefik/traefik/issues) and didn't find any.
- [X] Yes, I've searched similar issues on the [Traefik community forum](https://community.traefik.io) and didn't find any.

### What did you expect to see?

This is a new description of an older problem originally discussed here: https://github.com/traefik/traefik/issues/3882

That issue was originally opened in such a way that it appeared related to Traefik v1 -- however the issue affects Traefik v2 as well and @ddtmachado requested a new issue be created to describe the problem.

## Problem

When using the nomad <-> consul <-> traefik integration, it is not possible to use the canary deployment workflow that is supported by nomad.

An introduction to the nomad canary roll-out strategy can be found [here](https://learn.hashicorp.com/tutorials/nomad/job-blue-green-and-canary-deployments).  The crux of the problem is that nomad will register the non-canary version of the service and the canary version of the service with the same consul service-name such that traefik routing policy cannot distinguish between the canary and non-canary instances of the service.  Both service types are presented in traefik as one 'service' representing a pool of instances of both instance types.  This means you can't use traefik routing policy to direct traffic to only one or the other instance type independently.

## Background

Nomad supports a service deployment lifecycle that includes a canary deployment process.  When a new version of a service that uses the canary deployment feature is deployed, nomad will spin up `n` instances of the canary service 'alongside' the existing service instances and allow the canary instances to be health-checked and verified as desired before they are 'promoted'.  After promotion, the normal deployment process is followed and the running older instances are replaced with the new version.

Nomad supports defining consul parameters specific for the canary instances via the [canary_tags](https://www.nomadproject.io/docs/job-specification/service#canary_tags) and [canary_meta](https://www.nomadproject.io/docs/job-specification/service#canary_meta) nomad parameters.  These parameters parallel the [tags](https://www.nomadproject.io/docs/job-specification/service#tags) and [meta](https://www.nomadproject.io/docs/job-specification/service#meta) parameters that are used to populate consul for regular instances of the service.  When nomad spins up canary instances it populates consul based on canary_tags/canary_meta and when nomad spins up regular instances it populates consul based on tags/meta.

For both canary and non-canary instances, nomad registers the service with the same consul service-name.  Nomad only allows canary/non-canary instances to differ in their presentation to consul via the canary_tags and canary_meta parameters.

Traefik on the hand, considers all consul registrations with the same name to be the 'same service'.  Its not currently possible to define a traefik routing policy that will route traffic to canary only or non-canary only nomad instances.  As far as traefik is concerned the canary and non-canary instances are the same pool of instances and these instances can't be independently addressed..

If we could customize the presentation of the service in traefik via the nomad `canary_tags` or `canary_meta` properties in such a way as to persuade traefik to consider these instance types as two distinct services, then we would be able to take advantage of the full flexibility of traefik routing behaviors to decide how we would like to route to these now distinct-from-the-perspective-of-traefik services.

## Possible Solution

I suggest allowing to define a consul parameter that allows deriving an alternate name by combining with a suffix with the base consul service name.

For example -- suppose support is added for a consul tag named `traefik.consulnamesuffix` and suppose a nomad service declaration like this:

```
service {
  name = "health-check-tool"
  port = "http"

  tags = [
    "traefik.enable=true",
    "traefik.http.middlewares.strip_canary_header.headers.customrequestheaders.Canary=",
    "traefik.http.routers.deploy_automation_tester.rule=Host(`${var.domain}`)",
    "traefik.http.routers.deploy_automation_tester.tls=true",
    "traefik.http.routers.deploy_automation_tester.entrypoints=https",
    "traefik.http.routers.deploy_automation_tester.middlewares=strip_canary_header@consulcatalog"
  ]

  canary_tags = [
    "traefik.consulnamesuffix=`canary`",
    "traefik.enable=true",
    "traefik.http.routers.deploy_automation_tester_canary.rule=Host(`${var.domain}`) && Headers(`canary`, `true`)",
    "traefik.http.routers.deploy_automation_tester_canary.tls=true",
    "traefik.http.routers.deploy_automation_tester_canary.entrypoints=https",
  ]

  connect {
    sidecar_service {}
  }

  check {
    name     = "alive"
    type     = "http"
    path     = "/ping"
    interval = "10s"
    timeout  = "2s"
  }
}
```

While an active nomad deployment is in progress we would see two services in `traefik` -- a service named `health_check_tool` which would represent the 'old' version of the service and a service named `health_check_tool_canary` which would represent the 'new' desired version of the service that is being evaluated.  In this example, there would also be two traefik routers -- one named `deploy_automation_tester`  would route traffic for `${var.domain}` to `health_check_tool` and one named `deploy_automation_tester_canary` which would route traffic for `${var.domain}` that contains the `canary: true` header to the `health_check_tool_canary` service.

A mechanism like this would integrate well with the nomad deployment lifecycle and would allow using the normal traefik routing policy mechanisms to send traffic to the canary/non-canary instances of the service in whatever manner is desired during deployment of a new service version.


Discussion / hints:
It is really nice idea to get final traefik service name by modifying consul service name with the suffix from the tag, or even replacing full service name with the value from the tag. This will simplify configuration. Also, please see my note about router names with similar configuration https://github.com/hashicorp/nomad/issues/8073#issuecomment-1119532106
Great write up @breathe, pretty much the same way that [Fabio works](https://github.com/fabiolb/fabio) basically.
Btw while I do agree that something like this would be nice you can achieve the same today by running two traefik instances/job and pointing them to different tag prefixes (https://doc.traefik.io/traefik/providers/consul-catalog/#prefix)

The service would then look like this:
```
service {
  name = "health-check-tool"
  port = "http"

  tags = [
    "traefik.enable=true",
    "traefik.http.routers.deploy_automation_tester.rule=Host(`${var.domain}`)",
    "traefik.http.routers.deploy_automation_tester.tls=true",
    "traefik.http.routers.deploy_automation_tester.entrypoints=https",
  ]

  canary_tags = [
    "traefik-canary.enable=true",
    "traefik-canary.http.routers.deploy_automation_tester.rule=Host(`${var.domain}`)"",
    "traefik-canary.http.routers.deploy_automation_tester.tls=true",
    "traefik-canary.http.routers.deploy_automation_tester.entrypoints=https",
  ]

  connect {
    sidecar_service {}
  }

  check {
    name     = "alive"
    type     = "http"
    path     = "/ping"
    interval = "10s"
    timeout  = "2s"
  }
}
```

While this might feel cumbersome it also prevents one from introducing invalid configuration into the system via canary tags :) [And one needs to route canary requests with yet another loadbalancer to traefik first, so might only be feasible if one has a canary domain or similar].

Fix patch:
diff --git a/docs/content/routing/providers/consul-catalog.md b/docs/content/routing/providers/consul-catalog.md
index 7115726ce7..e1d5b1c65e 100644
--- a/docs/content/routing/providers/consul-catalog.md
+++ b/docs/content/routing/providers/consul-catalog.md
@@ -470,6 +470,20 @@ You can tell Traefik to consider (or not) the service as a Connect capable one b
 
 This option overrides the value of `connectByDefault`.
 
+#### `traefik.consulcatalog.canary`
+
+```yaml
+traefik.consulcatalog.canary=true
+```
+
+When ConsulCatalog, in the context of a Nomad orchestrator,
+is a provider (of service registration) for Traefik,
+one might have the need to distinguish within Traefik between a [Canary](https://learn.hashicorp.com/tutorials/nomad/job-blue-green-and-canary-deployments#deploy-with-canaries) instance of a service, or a production one.
+For example if one does not want them to be part of the same load-balancer.
+
+Therefore, this option, which is meant to be provided as one of the values of the `canary_tags` field in the Nomad [service stanza](https://www.nomadproject.io/docs/job-specification/service#canary_tags),
+allows Traefik to identify that the associated instance is a canary one.
+
 #### Port Lookup
 
 Traefik is capable of detecting the port to use, by following the default consul Catalog flow.
diff --git a/docs/content/routing/providers/nomad.md b/docs/content/routing/providers/nomad.md
index fcba195238..ccc6316e77 100644
--- a/docs/content/routing/providers/nomad.md
+++ b/docs/content/routing/providers/nomad.md
@@ -460,6 +460,19 @@ You can tell Traefik to consider (or not) the service by setting `traefik.enable
 
 This option overrides the value of `exposedByDefault`.
 
+#### `traefik.nomad.canary`
+
+```yaml
+traefik.nomad.canary=true
+```
+
+When Nomad orchestrator is a provider (of service registration) for Traefik,
+one might have the need to distinguish within Traefik between a [Canary](https://learn.hashicorp.com/tutorials/nomad/job-blue-green-and-canary-deployments#deploy-with-canaries) instance of a service, or a production one.
+For example if one does not want them to be part of the same load-balancer.
+
+Therefore, this option, which is meant to be provided as one of the values of the `canary_tags` field in the Nomad [service stanza](https://www.nomadproject.io/docs/job-specification/service#canary_tags),
+allows Traefik to identify that the associated instance is a canary one.
+
 #### Port Lookup
 
 Traefik is capable of detecting the port to use, by following the default Nomad Service Discovery flow.
diff --git a/pkg/provider/consulcatalog/config.go b/pkg/provider/consulcatalog/config.go
index 19fe33084d..bf0072f2e0 100644
--- a/pkg/provider/consulcatalog/config.go
+++ b/pkg/provider/consulcatalog/config.go
@@ -4,7 +4,10 @@ import (
 	"context"
 	"errors"
 	"fmt"
+	"hash/fnv"
 	"net"
+	"sort"
+	"strings"
 
 	"github.com/hashicorp/consul/api"
 	"github.com/traefik/traefik/v2/pkg/config/dynamic"
@@ -37,8 +40,7 @@ func (p *Provider) buildConfiguration(ctx context.Context, items []itemData, cer
 		if len(confFromLabel.TCP.Routers) > 0 || len(confFromLabel.TCP.Services) > 0 {
 			tcpOrUDP = true
 
-			err := p.buildTCPServiceConfiguration(ctxSvc, item, confFromLabel.TCP)
-			if err != nil {
+			if err := p.buildTCPServiceConfiguration(item, confFromLabel.TCP); err != nil {
 				logger.Error(err)
 				continue
 			}
@@ -49,8 +51,7 @@ func (p *Provider) buildConfiguration(ctx context.Context, items []itemData, cer
 		if len(confFromLabel.UDP.Routers) > 0 || len(confFromLabel.UDP.Services) > 0 {
 			tcpOrUDP = true
 
-			err := p.buildUDPServiceConfiguration(ctxSvc, item, confFromLabel.UDP)
-			if err != nil {
+			if err := p.buildUDPServiceConfiguration(item, confFromLabel.UDP); err != nil {
 				logger.Error(err)
 				continue
 			}
@@ -75,8 +76,7 @@ func (p *Provider) buildConfiguration(ctx context.Context, items []itemData, cer
 			}
 		}
 
-		err = p.buildServiceConfiguration(ctxSvc, item, confFromLabel.HTTP)
-		if err != nil {
+		if err = p.buildServiceConfiguration(item, confFromLabel.HTTP); err != nil {
 			logger.Error(err)
 			continue
 		}
@@ -89,7 +89,7 @@ func (p *Provider) buildConfiguration(ctx context.Context, items []itemData, cer
 			Labels: item.Labels,
 		}
 
-		provider.BuildRouterConfiguration(ctx, confFromLabel.HTTP, provider.Normalize(item.Name), p.defaultRuleTpl, model)
+		provider.BuildRouterConfiguration(ctx, confFromLabel.HTTP, getName(item), p.defaultRuleTpl, model)
 
 		configurations[svcName] = confFromLabel
 	}
@@ -128,22 +128,20 @@ func (p *Provider) keepContainer(ctx context.Context, item itemData) bool {
 	return true
 }
 
-func (p *Provider) buildTCPServiceConfiguration(ctx context.Context, item itemData, configuration *dynamic.TCPConfiguration) error {
+func (p *Provider) buildTCPServiceConfiguration(item itemData, configuration *dynamic.TCPConfiguration) error {
 	if len(configuration.Services) == 0 {
 		configuration.Services = make(map[string]*dynamic.TCPService)
 
 		lb := &dynamic.TCPServersLoadBalancer{}
 		lb.SetDefaults()
 
-		configuration.Services[provider.Normalize(item.Name)] = &dynamic.TCPService{
+		configuration.Services[getName(item)] = &dynamic.TCPService{
 			LoadBalancer: lb,
 		}
 	}
 
-	for name, service := range configuration.Services {
-		ctxSvc := log.With(ctx, log.Str(log.ServiceName, name))
-		err := p.addServerTCP(ctxSvc, item, service.LoadBalancer)
-		if err != nil {
+	for _, service := range configuration.Services {
+		if err := p.addServerTCP(item, service.LoadBalancer); err != nil {
 			return err
 		}
 	}
@@ -151,21 +149,19 @@ func (p *Provider) buildTCPServiceConfiguration(ctx context.Context, item itemDa
 	return nil
 }
 
-func (p *Provider) buildUDPServiceConfiguration(ctx context.Context, item itemData, configuration *dynamic.UDPConfiguration) error {
+func (p *Provider) buildUDPServiceConfiguration(item itemData, configuration *dynamic.UDPConfiguration) error {
 	if len(configuration.Services) == 0 {
 		configuration.Services = make(map[string]*dynamic.UDPService)
 
 		lb := &dynamic.UDPServersLoadBalancer{}
 
-		configuration.Services[provider.Normalize(item.Name)] = &dynamic.UDPService{
+		configuration.Services[getName(item)] = &dynamic.UDPService{
 			LoadBalancer: lb,
 		}
 	}
 
-	for name, service := range configuration.Services {
-		ctxSvc := log.With(ctx, log.Str(log.ServiceName, name))
-		err := p.addServerUDP(ctxSvc, item, service.LoadBalancer)
-		if err != nil {
+	for _, service := range configuration.Services {
+		if err := p.addServerUDP(item, service.LoadBalancer); err != nil {
 			return err
 		}
 	}
@@ -173,22 +169,20 @@ func (p *Provider) buildUDPServiceConfiguration(ctx context.Context, item itemDa
 	return nil
 }
 
-func (p *Provider) buildServiceConfiguration(ctx context.Context, item itemData, configuration *dynamic.HTTPConfiguration) error {
+func (p *Provider) buildServiceConfiguration(item itemData, configuration *dynamic.HTTPConfiguration) error {
 	if len(configuration.Services) == 0 {
 		configuration.Services = make(map[string]*dynamic.Service)
 
 		lb := &dynamic.ServersLoadBalancer{}
 		lb.SetDefaults()
 
-		configuration.Services[provider.Normalize(item.Name)] = &dynamic.Service{
+		configuration.Services[getName(item)] = &dynamic.Service{
 			LoadBalancer: lb,
 		}
 	}
 
-	for name, service := range configuration.Services {
-		ctxSvc := log.With(ctx, log.Str(log.ServiceName, name))
-		err := p.addServer(ctxSvc, item, service.LoadBalancer)
-		if err != nil {
+	for _, service := range configuration.Services {
+		if err := p.addServer(item, service.LoadBalancer); err != nil {
 			return err
 		}
 	}
@@ -196,7 +190,7 @@ func (p *Provider) buildServiceConfiguration(ctx context.Context, item itemData,
 	return nil
 }
 
-func (p *Provider) addServerTCP(ctx context.Context, item itemData, loadBalancer *dynamic.TCPServersLoadBalancer) error {
+func (p *Provider) addServerTCP(item itemData, loadBalancer *dynamic.TCPServersLoadBalancer) error {
 	if loadBalancer == nil {
 		return errors.New("load-balancer is not defined")
 	}
@@ -227,7 +221,7 @@ func (p *Provider) addServerTCP(ctx context.Context, item itemData, loadBalancer
 	return nil
 }
 
-func (p *Provider) addServerUDP(ctx context.Context, item itemData, loadBalancer *dynamic.UDPServersLoadBalancer) error {
+func (p *Provider) addServerUDP(item itemData, loadBalancer *dynamic.UDPServersLoadBalancer) error {
 	if loadBalancer == nil {
 		return errors.New("load-balancer is not defined")
 	}
@@ -254,7 +248,7 @@ func (p *Provider) addServerUDP(ctx context.Context, item itemData, loadBalancer
 	return nil
 }
 
-func (p *Provider) addServer(ctx context.Context, item itemData, loadBalancer *dynamic.ServersLoadBalancer) error {
+func (p *Provider) addServer(item itemData, loadBalancer *dynamic.ServersLoadBalancer) error {
 	if loadBalancer == nil {
 		return errors.New("load-balancer is not defined")
 	}
@@ -300,3 +294,18 @@ func (p *Provider) addServer(ctx context.Context, item itemData, loadBalancer *d
 func itemServersTransportKey(item itemData) string {
 	return provider.Normalize("tls-" + item.Namespace + "-" + item.Datacenter + "-" + item.Name)
 }
+
+func getName(i itemData) string {
+	if !i.ExtraConf.ConsulCatalog.Canary {
+		return provider.Normalize(i.Name)
+	}
+
+	tags := make([]string, len(i.Tags))
+	copy(tags, i.Tags)
+
+	sort.Strings(tags)
+
+	hasher := fnv.New64()
+	hasher.Write([]byte(strings.Join(tags, "")))
+	return provider.Normalize(fmt.Sprintf("%s-%d", i.Name, hasher.Sum64()))
+}
diff --git a/pkg/provider/consulcatalog/consul_catalog.go b/pkg/provider/consulcatalog/consul_catalog.go
index 59e19ddf4d..effd67cacf 100644
--- a/pkg/provider/consulcatalog/consul_catalog.go
+++ b/pkg/provider/consulcatalog/consul_catalog.go
@@ -283,13 +283,13 @@ func (p *Provider) getConsulServicesData(ctx context.Context) ([]itemData, error
 	for name, tags := range serviceNames {
 		logger := log.FromContext(log.With(ctx, log.Str("serviceName", name)))
 
-		svcCfg, err := p.getConfiguration(tagsToNeutralLabels(tags, p.Prefix))
+		extraConf, err := p.getExtraConf(tagsToNeutralLabels(tags, p.Prefix))
 		if err != nil {
 			logger.Errorf("Skip service: %v", err)
 			continue
 		}
 
-		if !svcCfg.Enable {
+		if !extraConf.Enable {
 			logger.Debug("Filtering disabled item")
 			continue
 		}
@@ -305,12 +305,12 @@ func (p *Provider) getConsulServicesData(ctx context.Context) ([]itemData, error
 			continue
 		}
 
-		if !p.ConnectAware && svcCfg.ConsulCatalog.Connect {
+		if !p.ConnectAware && extraConf.ConsulCatalog.Connect {
 			logger.Debugf("Filtering out Connect aware item, Connect support is not enabled")
 			continue
 		}
 
-		consulServices, statuses, err := p.fetchService(ctx, name, svcCfg.ConsulCatalog.Connect)
+		consulServices, statuses, err := p.fetchService(ctx, name, extraConf.ConsulCatalog.Connect)
 		if err != nil {
 			return nil, err
 		}
@@ -344,7 +344,7 @@ func (p *Provider) getConsulServicesData(ctx context.Context) ([]itemData, error
 				Status:     status,
 			}
 
-			extraConf, err := p.getConfiguration(item.Labels)
+			extraConf, err := p.getExtraConf(item.Labels)
 			if err != nil {
 				log.FromContext(ctx).Errorf("Skip item %s: %v", item.Name, err)
 				continue
diff --git a/pkg/provider/consulcatalog/label.go b/pkg/provider/consulcatalog/label.go
index 40e67920a4..65178750c0 100644
--- a/pkg/provider/consulcatalog/label.go
+++ b/pkg/provider/consulcatalog/label.go
@@ -4,17 +4,19 @@ import (
 	"github.com/traefik/traefik/v2/pkg/config/label"
 )
 
-// configuration Contains information from the labels that are globals (not related to the dynamic configuration) or specific to the provider.
+// configuration contains information from the labels that are globals (not related to the dynamic configuration) or specific to the provider.
 type configuration struct {
 	Enable        bool
 	ConsulCatalog specificConfiguration
 }
 
 type specificConfiguration struct {
-	Connect bool
+	Connect bool // <prefix>.consulcatalog.connect is the corresponding label.
+	Canary  bool // <prefix>.consulcatalog.canary is the corresponding label.
 }
 
-func (p *Provider) getConfiguration(labels map[string]string) (configuration, error) {
+// getExtraConf returns a configuration with settings which are not part of the dynamic configuration (e.g. "<prefix>.enable").
+func (p *Provider) getExtraConf(labels map[string]string) (configuration, error) {
 	conf := configuration{
 		Enable:        p.ExposedByDefault,
 		ConsulCatalog: specificConfiguration{Connect: p.ConnectByDefault},
diff --git a/pkg/provider/nomad/config.go b/pkg/provider/nomad/config.go
index 6ae3ebdfb0..6978bf04b9 100644
--- a/pkg/provider/nomad/config.go
+++ b/pkg/provider/nomad/config.go
@@ -4,8 +4,11 @@ import (
 	"context"
 	"errors"
 	"fmt"
+	"hash/fnv"
 	"net"
+	"sort"
 	"strconv"
+	"strings"
 
 	"github.com/traefik/traefik/v2/pkg/config/dynamic"
 	"github.com/traefik/traefik/v2/pkg/config/label"
@@ -76,7 +79,7 @@ func (p *Provider) buildConfig(ctx context.Context, items []item) *dynamic.Confi
 			Labels: labels,
 		}
 
-		provider.BuildRouterConfiguration(ctx, config.HTTP, provider.Normalize(i.Name), p.defaultRuleTpl, model)
+		provider.BuildRouterConfiguration(ctx, config.HTTP, getName(i), p.defaultRuleTpl, model)
 		configurations[svcName] = config
 	}
 
@@ -90,7 +93,7 @@ func (p *Provider) buildTCPConfig(i item, configuration *dynamic.TCPConfiguratio
 		lb := new(dynamic.TCPServersLoadBalancer)
 		lb.SetDefaults()
 
-		configuration.Services[provider.Normalize(i.Name)] = &dynamic.TCPService{
+		configuration.Services[getName(i)] = &dynamic.TCPService{
 			LoadBalancer: lb,
 		}
 	}
@@ -108,7 +111,7 @@ func (p *Provider) buildUDPConfig(i item, configuration *dynamic.UDPConfiguratio
 	if len(configuration.Services) == 0 {
 		configuration.Services = make(map[string]*dynamic.UDPService)
 
-		configuration.Services[provider.Normalize(i.Name)] = &dynamic.UDPService{
+		configuration.Services[getName(i)] = &dynamic.UDPService{
 			LoadBalancer: new(dynamic.UDPServersLoadBalancer),
 		}
 	}
@@ -129,7 +132,7 @@ func (p *Provider) buildServiceConfig(i item, configuration *dynamic.HTTPConfigu
 		lb := new(dynamic.ServersLoadBalancer)
 		lb.SetDefaults()
 
-		configuration.Services[provider.Normalize(i.Name)] = &dynamic.Service{
+		configuration.Services[getName(i)] = &dynamic.Service{
 			LoadBalancer: lb,
 		}
 	}
@@ -265,3 +268,18 @@ func (p *Provider) addServer(i item, lb *dynamic.ServersLoadBalancer) error {
 
 	return nil
 }
+
+func getName(i item) string {
+	if !i.ExtraConf.Canary {
+		return provider.Normalize(i.Name)
+	}
+
+	tags := make([]string, len(i.Tags))
+	copy(tags, i.Tags)
+
+	sort.Strings(tags)
+
+	hasher := fnv.New64()
+	hasher.Write([]byte(strings.Join(tags, "")))
+	return provider.Normalize(fmt.Sprintf("%s-%d", i.Name, hasher.Sum64()))
+}
diff --git a/pkg/provider/nomad/nomad.go b/pkg/provider/nomad/nomad.go
index 1e2af4510e..34877fb915 100644
--- a/pkg/provider/nomad/nomad.go
+++ b/pkg/provider/nomad/nomad.go
@@ -185,19 +185,25 @@ func createClient(namespace string, endpoint *EndpointConfig) (*api.Client, erro
 // configuration contains information from the service's tags that are globals
 // (not specific to the dynamic configuration).
 type configuration struct {
-	Enable bool // <prefix>.enable
+	Enable bool // <prefix>.enable is the corresponding label.
+	Canary bool // <prefix>.nomad.canary is the corresponding label.
 }
 
-// globalConfig returns a configuration with settings not specific to the dynamic configuration (i.e. "<prefix>.enable").
-func (p *Provider) globalConfig(tags []string) configuration {
-	enabled := p.ExposedByDefault
+// getExtraConf returns a configuration with settings which are not part of the dynamic configuration (e.g. "<prefix>.enable").
+func (p *Provider) getExtraConf(tags []string) configuration {
 	labels := tagsToLabels(tags, p.Prefix)
 
+	enabled := p.ExposedByDefault
 	if v, exists := labels["traefik.enable"]; exists {
 		enabled = strings.EqualFold(v, "true")
 	}
 
-	return configuration{Enable: enabled}
+	var canary bool
+	if v, exists := labels["traefik.nomad.canary"]; exists {
+		canary = strings.EqualFold(v, "true")
+	}
+
+	return configuration{Enable: enabled, Canary: canary}
 }
 
 func (p *Provider) getNomadServiceData(ctx context.Context) ([]item, error) {
@@ -216,8 +222,8 @@ func (p *Provider) getNomadServiceData(ctx context.Context) ([]item, error) {
 		for _, service := range stub.Services {
 			logger := log.FromContext(log.With(ctx, log.Str("serviceName", service.ServiceName)))
 
-			globalCfg := p.globalConfig(service.Tags)
-			if !globalCfg.Enable {
+			extraConf := p.getExtraConf(service.Tags)
+			if !extraConf.Enable {
 				logger.Debug("Filter Nomad service that is not enabled")
 				continue
 			}
@@ -248,7 +254,7 @@ func (p *Provider) getNomadServiceData(ctx context.Context) ([]item, error) {
 					Address:    i.Address,
 					Port:       i.Port,
 					Tags:       i.Tags,
-					ExtraConf:  p.globalConfig(i.Tags),
+					ExtraConf:  p.getExtraConf(i.Tags),
 				})
 			}
 		}


Test patch:
diff --git a/pkg/provider/consulcatalog/config_test.go b/pkg/provider/consulcatalog/config_test.go
index a63d916d0c..f97e383f16 100644
--- a/pkg/provider/consulcatalog/config_test.go
+++ b/pkg/provider/consulcatalog/config_test.go
@@ -273,7 +273,7 @@ func TestDefaultRule(t *testing.T) {
 
 			for i := 0; i < len(test.items); i++ {
 				var err error
-				test.items[i].ExtraConf, err = p.getConfiguration(test.items[i].Labels)
+				test.items[i].ExtraConf, err = p.getExtraConf(test.items[i].Labels)
 				require.NoError(t, err)
 			}
 
@@ -2611,6 +2611,253 @@ func Test_buildConfiguration(t *testing.T) {
 				},
 			},
 		},
+		{
+			desc:         "two HTTP service instances with one canary",
+			ConnectAware: true,
+			items: []itemData{
+				{
+					ID:         "1",
+					Node:       "Node1",
+					Datacenter: "dc1",
+					Name:       "Test",
+					Namespace:  "ns",
+					Labels: map[string]string{
+						"traefik.consulcatalog.connect": "true",
+					},
+					Address: "127.0.0.1",
+					Port:    "80",
+					Status:  api.HealthPassing,
+				},
+				{
+					ID:         "2",
+					Node:       "Node1",
+					Datacenter: "dc1",
+					Name:       "Test",
+					Namespace:  "ns",
+					Labels: map[string]string{
+						"traefik.consulcatalog.connect": "true",
+						"traefik.consulcatalog.canary":  "true",
+					},
+					Address: "127.0.0.2",
+					Port:    "80",
+					Status:  api.HealthPassing,
+				},
+			},
+			expected: &dynamic.Configuration{
+				TCP: &dynamic.TCPConfiguration{
+					Routers:     map[string]*dynamic.TCPRouter{},
+					Middlewares: map[string]*dynamic.TCPMiddleware{},
+					Services:    map[string]*dynamic.TCPService{},
+				},
+				UDP: &dynamic.UDPConfiguration{
+					Routers:  map[string]*dynamic.UDPRouter{},
+					Services: map[string]*dynamic.UDPService{},
+				},
+				HTTP: &dynamic.HTTPConfiguration{
+					Routers: map[string]*dynamic.Router{
+						"Test": {
+							Service: "Test",
+							Rule:    "Host(`Test.traefik.wtf`)",
+						},
+						"Test-97077516270503695": {
+							Service: "Test-97077516270503695",
+							Rule:    "Host(`Test.traefik.wtf`)",
+						},
+					},
+					Middlewares: map[string]*dynamic.Middleware{},
+					Services: map[string]*dynamic.Service{
+						"Test": {
+							LoadBalancer: &dynamic.ServersLoadBalancer{
+								Servers: []dynamic.Server{
+									{
+										URL: "https://127.0.0.1:80",
+									},
+								},
+								PassHostHeader:   Bool(true),
+								ServersTransport: "tls-ns-dc1-Test",
+							},
+						},
+						"Test-97077516270503695": {
+							LoadBalancer: &dynamic.ServersLoadBalancer{
+								Servers: []dynamic.Server{
+									{
+										URL: "https://127.0.0.2:80",
+									},
+								},
+								PassHostHeader:   Bool(true),
+								ServersTransport: "tls-ns-dc1-Test",
+							},
+						},
+					},
+					ServersTransports: map[string]*dynamic.ServersTransport{
+						"tls-ns-dc1-Test": {
+							ServerName:         "ns-dc1-Test",
+							InsecureSkipVerify: true,
+							RootCAs: []tls.FileOrContent{
+								"root",
+							},
+							Certificates: []tls.Certificate{
+								{
+									CertFile: "cert",
+									KeyFile:  "key",
+								},
+							},
+							PeerCertURI: "spiffe:///ns/ns/dc/dc1/svc/Test",
+						},
+					},
+				},
+			},
+		},
+		{
+			desc:         "two TCP service instances with one canary",
+			ConnectAware: true,
+			items: []itemData{
+				{
+					ID:         "1",
+					Node:       "Node1",
+					Datacenter: "dc1",
+					Name:       "Test",
+					Namespace:  "ns",
+					Labels: map[string]string{
+						"traefik.tcp.routers.test.rule": "HostSNI(`foobar`)",
+					},
+					Address: "127.0.0.1",
+					Port:    "80",
+					Status:  api.HealthPassing,
+				},
+				{
+					ID:         "2",
+					Node:       "Node1",
+					Datacenter: "dc1",
+					Name:       "Test",
+					Namespace:  "ns",
+					Labels: map[string]string{
+						"traefik.consulcatalog.canary":         "true",
+						"traefik.tcp.routers.test-canary.rule": "HostSNI(`canary.foobar`)",
+					},
+					Address: "127.0.0.2",
+					Port:    "80",
+					Status:  api.HealthPassing,
+				},
+			},
+			expected: &dynamic.Configuration{
+				TCP: &dynamic.TCPConfiguration{
+					Routers: map[string]*dynamic.TCPRouter{
+						"test": {
+							Service: "Test",
+							Rule:    "HostSNI(`foobar`)",
+						},
+						"test-canary": {
+							Service: "Test-17573747155436217342",
+							Rule:    "HostSNI(`canary.foobar`)",
+						},
+					},
+					Middlewares: map[string]*dynamic.TCPMiddleware{},
+					Services: map[string]*dynamic.TCPService{
+						"Test": {
+							LoadBalancer: &dynamic.TCPServersLoadBalancer{
+								Servers: []dynamic.TCPServer{
+									{Address: "127.0.0.1:80"},
+								},
+								TerminationDelay: Int(100),
+							},
+						},
+						"Test-17573747155436217342": {
+							LoadBalancer: &dynamic.TCPServersLoadBalancer{
+								Servers: []dynamic.TCPServer{
+									{Address: "127.0.0.2:80"},
+								},
+								TerminationDelay: Int(100),
+							},
+						},
+					},
+				},
+				UDP: &dynamic.UDPConfiguration{
+					Routers:  map[string]*dynamic.UDPRouter{},
+					Services: map[string]*dynamic.UDPService{},
+				},
+				HTTP: &dynamic.HTTPConfiguration{
+					Routers:           map[string]*dynamic.Router{},
+					Middlewares:       map[string]*dynamic.Middleware{},
+					Services:          map[string]*dynamic.Service{},
+					ServersTransports: map[string]*dynamic.ServersTransport{},
+				},
+			},
+		},
+		{
+			desc:         "two UDP service instances with one canary",
+			ConnectAware: true,
+			items: []itemData{
+				{
+					ID:         "1",
+					Node:       "Node1",
+					Datacenter: "dc1",
+					Name:       "Test",
+					Namespace:  "ns",
+					Labels: map[string]string{
+						"traefik.udp.routers.test.entrypoints": "udp",
+					},
+					Address: "127.0.0.1",
+					Port:    "80",
+					Status:  api.HealthPassing,
+				},
+				{
+					ID:         "2",
+					Node:       "Node1",
+					Datacenter: "dc1",
+					Name:       "Test",
+					Namespace:  "ns",
+					Labels: map[string]string{
+						"traefik.consulcatalog.canary":                "true",
+						"traefik.udp.routers.test-canary.entrypoints": "udp",
+					},
+					Address: "127.0.0.2",
+					Port:    "80",
+					Status:  api.HealthPassing,
+				},
+			},
+			expected: &dynamic.Configuration{
+				TCP: &dynamic.TCPConfiguration{
+					Routers:     map[string]*dynamic.TCPRouter{},
+					Middlewares: map[string]*dynamic.TCPMiddleware{},
+					Services:    map[string]*dynamic.TCPService{},
+				},
+				UDP: &dynamic.UDPConfiguration{
+					Routers: map[string]*dynamic.UDPRouter{
+						"test": {
+							EntryPoints: []string{"udp"},
+							Service:     "Test",
+						},
+						"test-canary": {
+							EntryPoints: []string{"udp"},
+							Service:     "Test-12825244908842506376",
+						},
+					},
+					Services: map[string]*dynamic.UDPService{
+						"Test": {
+							LoadBalancer: &dynamic.UDPServersLoadBalancer{
+								Servers: []dynamic.UDPServer{
+									{Address: "127.0.0.1:80"},
+								},
+							},
+						},
+						"Test-12825244908842506376": {
+							LoadBalancer: &dynamic.UDPServersLoadBalancer{
+								Servers: []dynamic.UDPServer{
+									{Address: "127.0.0.2:80"},
+								},
+							},
+						},
+					},
+				},
+				HTTP: &dynamic.HTTPConfiguration{
+					Routers:           map[string]*dynamic.Router{},
+					Middlewares:       map[string]*dynamic.Middleware{},
+					Services:          map[string]*dynamic.Service{},
+					ServersTransports: map[string]*dynamic.ServersTransport{},
+				},
+			},
+		},
 	}
 
 	for _, test := range testCases {
@@ -2633,7 +2880,7 @@ func Test_buildConfiguration(t *testing.T) {
 
 			for i := 0; i < len(test.items); i++ {
 				var err error
-				test.items[i].ExtraConf, err = p.getConfiguration(test.items[i].Labels)
+				test.items[i].ExtraConf, err = p.getExtraConf(test.items[i].Labels)
 				require.NoError(t, err)
 
 				var tags []string
diff --git a/pkg/provider/nomad/config_test.go b/pkg/provider/nomad/config_test.go
index 53cfc49dde..4c72ee8548 100644
--- a/pkg/provider/nomad/config_test.go
+++ b/pkg/provider/nomad/config_test.go
@@ -2209,6 +2209,239 @@ func Test_buildConfig(t *testing.T) {
 				},
 			},
 		},
+		{
+			desc: "two HTTP service instances with one canary",
+			items: []item{
+				{
+					ID:         "1",
+					Node:       "Node1",
+					Datacenter: "dc1",
+					Name:       "Test",
+					Namespace:  "ns",
+					Tags:       []string{},
+					Address:    "127.0.0.1",
+					Port:       80,
+					ExtraConf:  configuration{Enable: true},
+				},
+				{
+					ID:         "2",
+					Node:       "Node1",
+					Datacenter: "dc1",
+					Name:       "Test",
+					Namespace:  "ns",
+					Tags: []string{
+						"traefik.nomad.canary = true",
+					},
+					Address: "127.0.0.2",
+					Port:    80,
+					ExtraConf: configuration{
+						Enable: true,
+						Canary: true,
+					},
+				},
+			},
+			expected: &dynamic.Configuration{
+				TCP: &dynamic.TCPConfiguration{
+					Routers:     map[string]*dynamic.TCPRouter{},
+					Middlewares: map[string]*dynamic.TCPMiddleware{},
+					Services:    map[string]*dynamic.TCPService{},
+				},
+				UDP: &dynamic.UDPConfiguration{
+					Routers:  map[string]*dynamic.UDPRouter{},
+					Services: map[string]*dynamic.UDPService{},
+				},
+				HTTP: &dynamic.HTTPConfiguration{
+					Routers: map[string]*dynamic.Router{
+						"Test": {
+							Service: "Test",
+							Rule:    "Host(`Test.traefik.test`)",
+						},
+						"Test-1234154071633021619": {
+							Service: "Test-1234154071633021619",
+							Rule:    "Host(`Test.traefik.test`)",
+						},
+					},
+					Middlewares: map[string]*dynamic.Middleware{},
+					Services: map[string]*dynamic.Service{
+						"Test": {
+							LoadBalancer: &dynamic.ServersLoadBalancer{
+								Servers: []dynamic.Server{
+									{
+										URL: "http://127.0.0.1:80",
+									},
+								},
+								PassHostHeader: Bool(true),
+							},
+						},
+						"Test-1234154071633021619": {
+							LoadBalancer: &dynamic.ServersLoadBalancer{
+								Servers: []dynamic.Server{
+									{
+										URL: "http://127.0.0.2:80",
+									},
+								},
+								PassHostHeader: Bool(true),
+							},
+						},
+					},
+					ServersTransports: map[string]*dynamic.ServersTransport{},
+				},
+			},
+		},
+		{
+			desc: "two TCP service instances with one canary",
+			items: []item{
+				{
+					ID:         "1",
+					Node:       "Node1",
+					Datacenter: "dc1",
+					Name:       "Test",
+					Namespace:  "ns",
+					Tags: []string{
+						"traefik.tcp.routers.test.rule = HostSNI(`foobar`)",
+					},
+					Address:   "127.0.0.1",
+					Port:      80,
+					ExtraConf: configuration{Enable: true},
+				},
+				{
+					ID:         "2",
+					Node:       "Node1",
+					Datacenter: "dc1",
+					Name:       "Test",
+					Namespace:  "ns",
+					Tags: []string{
+						"traefik.nomad.canary = true",
+						"traefik.tcp.routers.test-canary.rule = HostSNI(`canary.foobar`)",
+					},
+					Address: "127.0.0.2",
+					Port:    80,
+					ExtraConf: configuration{
+						Enable: true,
+						Canary: true,
+					},
+				},
+			},
+			expected: &dynamic.Configuration{
+				TCP: &dynamic.TCPConfiguration{
+					Routers: map[string]*dynamic.TCPRouter{
+						"test": {
+							Service: "Test",
+							Rule:    "HostSNI(`foobar`)",
+						},
+						"test-canary": {
+							Service: "Test-8769860286750522282",
+							Rule:    "HostSNI(`canary.foobar`)",
+						},
+					},
+					Middlewares: map[string]*dynamic.TCPMiddleware{},
+					Services: map[string]*dynamic.TCPService{
+						"Test": {
+							LoadBalancer: &dynamic.TCPServersLoadBalancer{
+								Servers: []dynamic.TCPServer{
+									{Address: "127.0.0.1:80"},
+								},
+								TerminationDelay: Int(100),
+							},
+						},
+						"Test-8769860286750522282": {
+							LoadBalancer: &dynamic.TCPServersLoadBalancer{
+								Servers: []dynamic.TCPServer{
+									{Address: "127.0.0.2:80"},
+								},
+								TerminationDelay: Int(100),
+							},
+						},
+					},
+				},
+				UDP: &dynamic.UDPConfiguration{
+					Routers:  map[string]*dynamic.UDPRouter{},
+					Services: map[string]*dynamic.UDPService{},
+				},
+				HTTP: &dynamic.HTTPConfiguration{
+					Routers:           map[string]*dynamic.Router{},
+					Middlewares:       map[string]*dynamic.Middleware{},
+					Services:          map[string]*dynamic.Service{},
+					ServersTransports: map[string]*dynamic.ServersTransport{},
+				},
+			},
+		},
+		{
+			desc: "two UDP service instances with one canary",
+			items: []item{
+				{
+					ID:         "1",
+					Node:       "Node1",
+					Datacenter: "dc1",
+					Name:       "Test",
+					Namespace:  "ns",
+					Tags: []string{
+						"traefik.udp.routers.test.entrypoints = udp",
+					},
+					Address:   "127.0.0.1",
+					Port:      80,
+					ExtraConf: configuration{Enable: true},
+				},
+				{
+					ID:         "2",
+					Node:       "Node1",
+					Datacenter: "dc1",
+					Name:       "Test",
+					Namespace:  "ns",
+					Tags: []string{
+						"traefik.nomad.canary = true",
+						"traefik.udp.routers.test-canary.entrypoints = udp",
+					},
+					Address: "127.0.0.2",
+					Port:    80,
+					ExtraConf: configuration{
+						Enable: true,
+						Canary: true,
+					},
+				},
+			},
+			expected: &dynamic.Configuration{
+				TCP: &dynamic.TCPConfiguration{
+					Routers:     map[string]*dynamic.TCPRouter{},
+					Middlewares: map[string]*dynamic.TCPMiddleware{},
+					Services:    map[string]*dynamic.TCPService{},
+				},
+				UDP: &dynamic.UDPConfiguration{
+					Routers: map[string]*dynamic.UDPRouter{
+						"test": {
+							EntryPoints: []string{"udp"},
+							Service:     "Test",
+						},
+						"test-canary": {
+							EntryPoints: []string{"udp"},
+							Service:     "Test-1611429260986126224",
+						},
+					},
+					Services: map[string]*dynamic.UDPService{
+						"Test": {
+							LoadBalancer: &dynamic.UDPServersLoadBalancer{
+								Servers: []dynamic.UDPServer{
+									{Address: "127.0.0.1:80"},
+								},
+							},
+						},
+						"Test-1611429260986126224": {
+							LoadBalancer: &dynamic.UDPServersLoadBalancer{
+								Servers: []dynamic.UDPServer{
+									{Address: "127.0.0.2:80"},
+								},
+							},
+						},
+					},
+				},
+				HTTP: &dynamic.HTTPConfiguration{
+					Routers:           map[string]*dynamic.Router{},
+					Middlewares:       map[string]*dynamic.Middleware{},
+					Services:          map[string]*dynamic.Service{},
+					ServersTransports: map[string]*dynamic.ServersTransport{},
+				},
+			},
+		},
 	}
 
 	for _, test := range testCases {
diff --git a/pkg/provider/nomad/nomad_test.go b/pkg/provider/nomad/nomad_test.go
index 3142eb3e96..02e1217efd 100644
--- a/pkg/provider/nomad/nomad_test.go
+++ b/pkg/provider/nomad/nomad_test.go
@@ -65,7 +65,7 @@ func Test_globalConfig(t *testing.T) {
 	for _, test := range cases {
 		t.Run(test.Name, func(t *testing.T) {
 			p := Provider{ExposedByDefault: test.ExposedByDefault, Prefix: test.Prefix}
-			result := p.globalConfig(test.Tags)
+			result := p.getExtraConf(test.Tags)
 			require.Equal(t, test.exp, result)
 		})
 	}