Repository: jcmoraisjr/haproxy-ingress

Issue:
HAProxy fails to start
**Description of the problem**

With HAProxy Ingress 0.14.0 and the external HAProxy configuration, the HAProxy container fails to start in my configuration.

```
[NOTICE]   (1) : haproxy version is 2.4.20-d59cd78
[ALERT]    (1) : parsing [/etc/haproxy/haproxy.cfg:137] : 'bind unix@/var/run/haproxy/_https_socket.sock' : 'crt-list' : cannot open the file '/var/lib/haproxy/crt/default-fake-certificate.pem'.
[ALERT]    (1) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
[ALERT]    (1) : Fatal errors found in configuration.
```

It works fine if I use the combined HAProxy + HAProxy Ingress container.

In case it's relevant, I only have two Ingress resources picked up by HAProxy Ingress and both are configured with `ingress.kubernetes.io/ssl-passthrough: "true"`.

**Expected behavior**

HAProxy should start when run separately from the HAProxy Ingress container.

**Steps to reproduce the problem**

1. Install the chart with `controller.haproxy.enabled=true`.

**Environment information**

HAProxy Ingress version: `v0.14.0`

Helm values.yaml:

```
controller:
  extraArgs:
    ignore-ingress-without-class: "true"
    wait-before-shutdown: "30"

  config:
    bind-ip-addr-http: "[::]"
    bind-ip-addr-tcp: "[::]"
    https-log-format: default
    tcp-log-format: default

  replicaCount: 2

  resources:
    requests:
      cpu: 10m
      memory: 64Mi
    limits:
      memory: 192Mi

  service:
    type: LoadBalancer
    externalTrafficPolicy: Local
    ipFamilyPolicy: RequireDualStack

  haproxy:
    enabled: true

    resources:
      requests:
        cpu: 10m
        memory: 64Mi
      limits:
        memory: 96Mi

  stats:
    enabled: true

  metrics:
    enabled: true

  serviceMonitor:
    enabled: true
```

Ingress objects:

```
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/ssl-passthrough: "true"
  name: test
spec:
  ingressClassName: haproxy
  rules:
  - host: test.example.com
    http:
      paths:
      - backend:
          service:
            name: test
            port:
              name: https
        path: /
        pathType: Prefix
```

Discussion:
@jcmoraisjr: Hi, sorry about the long delay. I couldn't reproduce the problem. Can you confirm if this issue are reproducible on older controller versions? What distro are you using? Can you confirm if both controller and haproxy containers mount `/var/lib/haproxy`, and what controller side writes can be read by the haproxy side?@joelwurtz: We got the same errors when trying to use external haproxy. Don't know about previous versions but our case, all mounted directories were empty.

Is the behavior to have shared mount directories between containers in this case ?

If that's the case that would explain it as our provider does not support sharing the same volume between containers@jcmoraisjr: Controller has only file system as the way to communicate with a haproxy instance. Running it externally means that you need to mirror fs between the containers. There are a few words in the [example page](https://haproxy-ingress.github.io/docs/examples/external-haproxy/#shared-filesystem) about that. Double check how your deployment resource is created regarding the emptyDir configuration, or even better try to deploy two containers sharing fs via emptyDir in order to reproduce the issue. Let us know if our helm chart is creating something in a wrong way.@joelwurtz: Here is some details about the error : 

Pod description:

```
Name:             main-haproxy-ingress-6b8cc8db4-j4bjq
Namespace:        router
Priority:         0
Service Account:  main-haproxy-ingress
Node:             preprod-gra-node-pool-node-b18ebe/10.0.1.159
Start Time:       Wed, 08 Feb 2023 11:04:59 +0100
Labels:           app.kubernetes.io/instance=main
                  app.kubernetes.io/name=haproxy-ingress
                  pod-template-hash=6b8cc8db4
Annotations:      cni.projectcalico.org/containerID: 1a11c22cea893a9fe5f422221f1151434e40de9ed27a20fd5cd5c43821aa0c70
                  cni.projectcalico.org/podIP: 10.2.0.225/32
                  cni.projectcalico.org/podIPs: 10.2.0.225/32
Status:           Running
IP:               10.2.0.225
IPs:
  IP:           10.2.0.225
Controlled By:  ReplicaSet/main-haproxy-ingress-6b8cc8db4
Init Containers:
  haproxy-ingress-init:
    Container ID:  containerd://7fdc74fca37d8f786b21f4e4beacd3b0183fd0d5c82c1592d8fe7c241d203953
    Image:         quay.io/jcmoraisjr/haproxy-ingress:v0.14.0
    Image ID:      quay.io/jcmoraisjr/haproxy-ingress@sha256:2afa7354e39b0952cea74727cb45350bb3c1ec2acd335062b54482677cda16ee
    Port:          <none>
    Host Port:     <none>
    Args:
      --init
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 08 Feb 2023 11:05:01 +0100
      Finished:     Wed, 08 Feb 2023 11:05:02 +0100
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     10m
      memory:  32Mi
    Requests:
      cpu:        10m
      memory:     32Mi
    Environment:  <none>
    Mounts:
      /etc/haproxy from etc (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-v7lwr (ro)
Containers:
  haproxy-ingress:
    Container ID:  containerd://f2c410baf50494cae1e29d135f620ddb1963bad39bf7cc1ae220ae0d08a04b3b
    Image:         quay.io/jcmoraisjr/haproxy-ingress:v0.14.0
    Image ID:      quay.io/jcmoraisjr/haproxy-ingress@sha256:2afa7354e39b0952cea74727cb45350bb3c1ec2acd335062b54482677cda16ee
    Port:          10254/TCP
    Host Port:     0/TCP
    Args:
      --configmap=router/main-haproxy-ingress
      --ingress-class=haproxy
      --master-socket=/var/run/haproxy/master.sock
      --sort-backends
    State:          Running
      Started:      Wed, 08 Feb 2023 11:05:03 +0100
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAME:       main-haproxy-ingress-6b8cc8db4-j4bjq (v1:metadata.name)
      POD_NAMESPACE:  router (v1:metadata.namespace)
    Mounts:
      /etc/haproxy from etc (rw)
      /var/lib/haproxy from lib (rw)
      /var/run/haproxy from run (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-v7lwr (ro)
  haproxy:
    Container ID:  containerd://d6e8bfab4c12224ab8b9efd2742e07c099bcc1404b7b1621eeb313ee0a28564b
    Image:         haproxy:2.4.20-alpine
    Image ID:      docker.io/library/haproxy@sha256:f169c8d3078c75ab08bd7836632f45042b806f3fedf87b800bf790cb38eba9f5
    Ports:         80/TCP, 443/TCP, 9101/TCP, 1936/TCP, 10253/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      -W
      -S
      /var/run/haproxy/master.sock,mode,600
      -f
      /etc/haproxy
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 08 Feb 2023 11:05:46 +0100
      Finished:     Wed, 08 Feb 2023 11:05:46 +0100
    Ready:          False
    Restart Count:  3
    Liveness:       http-get http://:10253/healthz delay=60s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:10253/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /etc/haproxy from etc (rw)
      /var/lib/haproxy from lib (rw)
      /var/run/haproxy from run (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-v7lwr (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  etc:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  lib:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  run:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-v7lwr:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  76s                default-scheduler  Successfully assigned router/main-haproxy-ingress-6b8cc8db4-j4bjq to preprod-gra-node-pool-node-b18ebe
  Normal   Pulling    75s                kubelet            Pulling image "quay.io/jcmoraisjr/haproxy-ingress:v0.14.0"
  Normal   Pulled     74s                kubelet            Successfully pulled image "quay.io/jcmoraisjr/haproxy-ingress:v0.14.0" in 498.060831ms
  Normal   Created    74s                kubelet            Created container haproxy-ingress-init
  Normal   Started    74s                kubelet            Started container haproxy-ingress-init
  Normal   Pulling    73s                kubelet            Pulling image "quay.io/jcmoraisjr/haproxy-ingress:v0.14.0"
  Normal   Pulled     73s                kubelet            Successfully pulled image "quay.io/jcmoraisjr/haproxy-ingress:v0.14.0" in 511.60126ms
  Normal   Created    73s                kubelet            Created container haproxy-ingress
  Normal   Started    72s                kubelet            Started container haproxy-ingress
  Normal   Pulled     71s                kubelet            Successfully pulled image "haproxy:2.4.20-alpine" in 924.066129ms
  Normal   Pulled     70s                kubelet            Successfully pulled image "haproxy:2.4.20-alpine" in 1.037779427s
  Normal   Pulling    54s (x3 over 72s)  kubelet            Pulling image "haproxy:2.4.20-alpine"
  Normal   Created    53s (x3 over 71s)  kubelet            Created container haproxy
  Normal   Started    53s (x3 over 71s)  kubelet            Started container haproxy
  Normal   Pulled     53s                kubelet            Successfully pulled image "haproxy:2.4.20-alpine" in 651.671222ms
  Warning  BackOff    52s (x5 over 69s)  kubelet            Back-off restarting failed container
```

Logs of the failing pod : 

````
➜  kube git:(master) ✗ kubectl logs main-haproxy-ingress-6b8cc8db4-j4bjq -c haproxy -n router
[NOTICE]   (1) : haproxy version is 2.4.20-d59cd78
[ALERT]    (1) : parsing [/etc/haproxy/haproxy.cfg:325] : 'bind unix@/var/run/haproxy/_https_socket.sock' : 'crt-list' : cannot open the file '/var/lib/haproxy/crt/default-fake-certificate.pem'.
[ALERT]    (1) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
[ALERT]    (1) : Fatal errors found in configuration.
````

The volumes seems to be correctly mapped, when i inspect the haproxy-ingress container 

```
➜  kube git:(master) ✗ kubectl exec main-haproxy-ingress-6b8cc8db4-j4bjq  -i -t -n router -- sh
Defaulted container "haproxy-ingress" out of: haproxy-ingress, haproxy, haproxy-ingress-init (init)
/ # ls -l /var/lib/haproxy/c
cacerts/  crl/      crt/
/ # ls -l /var/lib/haproxy/crt
total 20
-rw-------    1 root     root          7360 Feb  8 10:05 agent-proxy_router-domain-tls.pem
-rw-------    1 root     root          2933 Feb  8 10:05 default-fake-certificate.pem
-rw-------    1 root     root          7360 Feb  8 10:05 demo_router-domain-tls.pem
```

It's really strange, does `lib` mountpoint should also be present in the init container ?@jcmoraisjr: What about the content of `/var/lib/haproxy/crt/` on both haproxy-ingress and haproxy containers?

> It's really strange, does lib mountpoint should also be present in the init container ?

init container just need /etc/haproxy/ mounted.@joelwurtz: Okay it's a permission problem, haproxy error does not give this information, after updating the container to keep it alive and running into it : 

```
~/crt $ ls -la
total 28
drwxr-xr-x    2 root     root          4096 Feb  8 17:07 .
drwxrwxrwx    6 root     root          4096 Feb  8 17:08 ..
-rw-------    1 root     root          7360 Feb  8 17:07 agent-proxy_router-domain-tls.pem
-rw-------    1 root     root          2933 Feb  8 17:07 default-fake-certificate.pem
-rw-------    1 root     root          7360 Feb  8 17:07 demo_router-domain-tls.pem
~/crt $ cat default-fake-certificate.pem
cat: can't open 'default-fake-certificate.pem': Permission denied
~/crt $ whoami
haproxy
```

Files are created with root user but container is run with the haproxy one@jcmoraisjr: Ah sure, that makes sense. You can use a short term solution by, either, running haproxy as root (see [security considerations](http://docs.haproxy.org/2.7/management.html#13)) or start the controller as the haproxy's uid.

Changing haproxy to root:

```
controller:
  haproxy:
    securityContext:
      runAsUser: 0
```

Changing controller to haproxy's uid:

```
controller:
   securityContext:
     runAsUser: 99
```

I'll leave this issue open until we have a good long term solution, maybe adding param to choose between syncing uids or changing permission to global read.@joelwurtz: I try running the ingress controller as haproxy user (99), but the external then fail to bind to 80 and 443 port number (normal), so running it as root is better.

```
➜  kube git:(master) ✗ kubectl logs main-haproxy-ingress-c6468758b-khdl5 -c haproxy -n router
[NOTICE]   (1) : haproxy version is 2.4.20-d59cd78
[ALERT]    (1) : Starting proxy _front__tls: cannot bind socket (Permission denied) [0.0.0.0:443]
[ALERT]    (1) : Starting frontend _front_http: cannot bind socket (Permission denied) [0.0.0.0:80]
[ALERT]    (1) : [haproxy.main()] Some protocols failed to start their listeners! Exiting.
```

AFAIK we should run the haproxy container as root but use: 

```
user haproxy
group haproxy
```

in the haproxy.cfg file so that it forks itself with this user after binding network sockets

(it didn't seem to be the case when i check the current configuration)

Running the external haproxy as root seems to works fine ATM but it's definitly not a best practice in term of security@joelwurtz: This is what i end up doing in order to have a "correct" setup : 

```
    securityContext:
        runAsUser: 99

    haproxy:
        enabled: true

        securityContext:
            runAsUser: 0
            runAsGroup: 0

    config:
        use-haproxy-user: "true"
```

So the files are correctly mapped for the haproxy user, and external haproxy is launched as root but is running as haproxy user, and it seems to work fine@jcorley-sysdig: one solution to the port 80/443 problem that we've tested is adding this to the container:
```
setcap CAP_NET_BIND_SERVICE=+eip /usr/local/sbin/haproxy
```@jcmoraisjr: Fixed in v0.14.3, v0.13.12 and v0.12.17, closing. Let us know if you still find any permission related problem.@jessequinn: hi @jcmoraisjr this is still an issue on 0.14.8. I have to use root to be able to run an external haproxy. Will there be a fix?@jcmoraisjr: Hi @jessequinn thanks for the update. I missed your comment here, sorry about the delay.  Please file another issue, or instead reach out via the Slack channel, adding how you're provisioning HAProxy Ingress along with a few logs that tells what controller is complaining. In fact there are some gaps on v0.14 and earlier to properly run as non root, making it harder to configure.

Solution patch:
diff --git a/docs/content/en/docs/configuration/keys.md b/docs/content/en/docs/configuration/keys.md
index 100752ab..fee52cb7 100644
--- a/docs/content/en/docs/configuration/keys.md
+++ b/docs/content/en/docs/configuration/keys.md
@@ -419,6 +419,7 @@ The table below describes all supported configuration keys.
 | [`modsecurity-use-coraza`](#modsecurity)             | [true\|false]                           | Global  | `false`               |
 | [`nbproc-ssl`](#nbproc)                              | number of process                       | Global  | `0`                |
 | [`nbthread`](#nbthread)                              | number of threads                       | Global  |                    |
+| [`no-redirect-locations`](#redirect)                 | comma-separated list of URIs            | Global  | `/.well-known/acme-challenge` |
 | [`no-tls-redirect-locations`](#ssl-redirect)         | comma-separated list of URIs            | Global  | `/.well-known/acme-challenge` |
 | [`oauth`](#oauth)                                    | "oauth2_proxy"                          | Path    |                    |
 | [`oauth-headers`](#oauth)                            | `<header>:<var>,...`                    | Path    |                    |
@@ -2220,13 +2221,14 @@ See also:
 
 ## Redirect
 
-| Configuration key     | Scope    | Default | Since |
-|-----------------------|----------|---------|-------|
-| `redirect-from`       | `Host`   |         | v0.13 |
-| `redirect-from-code`  | `Global` | `302`   | v0.13 |
-| `redirect-from-regex` | `Host`   |         | v0.13 |
-| `redirect-to`         | `Path`   |         | v0.13 |
-| `redirect-to-code`    | `Global` | `302`   | v0.13 |
+| Configuration key       | Scope    | Default                       | Since   |
+|-------------------------|----------|-------------------------------|---------|
+| `no-redirect-locations` | `Global` | `/.well-known/acme-challenge` | v0.14.3 |
+| `redirect-from`         | `Host`   |                               | v0.13   |
+| `redirect-from-code`    | `Global` | `302`                         | v0.13   |
+| `redirect-from-regex`   | `Host`   |                               | v0.13   |
+| `redirect-to`           | `Path`   |                               | v0.13   |
+| `redirect-to-code`      | `Global` | `302`                         | v0.13   |
 
 Configures HTTP redirect. Redirect *from* matches source hostnames that should be redirected
 to the hostname declared in the ingress spec. Redirect *to* uses the hostname declared in the
@@ -2238,6 +2240,7 @@ examples below.
 * `redirect-from-code`: Which HTTP status code should be used in the redirect from. A `302` response is used by default if not configured.
 * `redirect-to`: Defines the destination URL to redirect the incoming request. The declared hostname and path are used only to match the request, the backend will not be used and it's only needed to be declared to satisfy ingress spec validation.
 * `redirect-to-code`: Which HTTP status code should be used in the redirect to. A `302` response is used by default if not configured.
+* `no-redirect-locations`: Defines a comma-separated list of paths that should be ignored by all the redirects. Default value is `/.well-known/acme-challenge`, used by ACME protocol. Configure as an empty string to make the redirect happen on all paths, including the ACME challenge.
 
 **Using redirect-from**
 
diff --git a/pkg/common/net/ssl/ssl.go b/pkg/common/net/ssl/ssl.go
index 5797bd94..554750a9 100644
--- a/pkg/common/net/ssl/ssl.go
+++ b/pkg/common/net/ssl/ssl.go
@@ -57,6 +57,14 @@ func AddOrUpdateCertAndKey(name string, cert, key, ca []byte) (*ingress.SSLCert,
 	}
 	klog.V(3).Infof("Creating temp file %v for Keypair: %v", tempPemFile.Name(), pemName)
 
+	// Controller v0.14 and older should be running as root, so trying to change
+	// tmpfile owner to haproxy PID, in that case it has a chance to read if running
+	// as a sidecar. No problem if err, should be permission due to controller not
+	// be running as root. HAProxy sidecar should be running as 99 or 0, so no
+	// problem as well. Other HAProxy distros might be using another PID, which
+	// this change won't be neither helping nor impacting.
+	_ = os.Chown(tempPemFile.Name(), 99, 99)
+
 	_, err = tempPemFile.Write(cert)
 	if err != nil {
 		return nil, fmt.Errorf("could not write to pem file %v: %v", tempPemFile.Name(), err)
diff --git a/pkg/converters/converters.go b/pkg/converters/converters.go
index e85d78d2..8a6955e7 100644
--- a/pkg/converters/converters.go
+++ b/pkg/converters/converters.go
@@ -88,7 +88,12 @@ func (c *converters) Sync() {
 	//
 	// configmap converters
 	//
-	if changed.TCPConfigMapDataCur != nil {
+	if changed.TCPConfigMapDataCur != nil || changed.TCPConfigMapDataNew != nil {
+		// We always need to run configmap based tcp sync, when configured, because
+		// we don't have any tracking in place asking us to do it on, e.g., endpoint
+		// or secret updates. Although cur is always assigned if configmap based tcp
+		// is configured, only new is assigned in the very first run. Otoh new is
+		// only assigned when the configmap is changed. So we need to check both.
 		tcpSvcConverter := configmap.NewTCPServicesConverter(c.options, c.haproxy, changed)
 		tcpSvcConverter.Sync()
 		c.timer.Tick("parse_tcp_svc")
diff --git a/pkg/converters/ingress/annotations/updater.go b/pkg/converters/ingress/annotations/updater.go
index dbf5d738..4ad05b84 100644
--- a/pkg/converters/ingress/annotations/updater.go
+++ b/pkg/converters/ingress/annotations/updater.go
@@ -153,6 +153,7 @@ func (c *updater) UpdateGlobalConfig(haproxyConfig haproxy.Config, mapper *Mappe
 	d.global.MaxConn = mapper.Get(ingtypes.GlobalMaxConnections).Int()
 	d.global.DefaultBackendRedir = mapper.Get(ingtypes.GlobalDefaultBackendRedirect).String()
 	d.global.DefaultBackendRedirCode = mapper.Get(ingtypes.GlobalDefaultBackendRedirectCode).Int()
+	d.global.NoRedirects = utils.Split(mapper.Get(ingtypes.GlobalNoRedirectLocations).String(), ",")
 	d.global.DrainSupport.Drain = mapper.Get(ingtypes.GlobalDrainSupport).Bool()
 	d.global.DrainSupport.Redispatch = mapper.Get(ingtypes.GlobalDrainSupportRedispatch).Bool()
 	d.global.Cookie.Key = mapper.Get(ingtypes.GlobalCookieKey).Value
diff --git a/pkg/converters/ingress/defaults.go b/pkg/converters/ingress/defaults.go
index 59b3a1fc..aa68b1b7 100644
--- a/pkg/converters/ingress/defaults.go
+++ b/pkg/converters/ingress/defaults.go
@@ -98,6 +98,7 @@ func createDefaults() map[string]string {
 		types.GlobalModsecurityTimeoutProcessing: "1s",
 		types.GlobalModsecurityTimeoutServer:     "5s",
 		types.GlobalNbprocBalance:                "1",
+		types.GlobalNoRedirectLocations:          "/.well-known/acme-challenge",
 		types.GlobalNoTLSRedirectLocations:       "/.well-known/acme-challenge",
 		types.GlobalOriginalForwardedForHdr:      "X-Original-Forwarded-For",
 		types.GlobalPathTypeOrder:                "exact,prefix,begin,regex",
diff --git a/pkg/converters/ingress/types/global.go b/pkg/converters/ingress/types/global.go
index 46a226d4..a6a2e2ff 100644
--- a/pkg/converters/ingress/types/global.go
+++ b/pkg/converters/ingress/types/global.go
@@ -103,6 +103,7 @@ const (
 	GlobalNbprocBalance                = "nbproc-balance"
 	GlobalNbprocSSL                    = "nbproc-ssl"
 	GlobalNbthread                     = "nbthread"
+	GlobalNoRedirectLocations          = "no-redirect-locations"
 	GlobalNoTLSRedirectLocations       = "no-tls-redirect-locations"
 	GlobalOriginalForwardedForHdr      = "original-forwarded-for-hdr"
 	GlobalPathTypeOrder                = "path-type-order"
diff --git a/pkg/haproxy/types/tcpbackend.go b/pkg/haproxy/types/tcpbackend.go
index 5ee7eb2a..1e4670b6 100644
--- a/pkg/haproxy/types/tcpbackend.go
+++ b/pkg/haproxy/types/tcpbackend.go
@@ -96,5 +96,9 @@ func (b *TCPBackend) AddEndpoint(ip string, port int) *TCPEndpoint {
 		Target: fmt.Sprintf("%s:%d", ip, port),
 	}
 	b.Endpoints = append(b.Endpoints, ep)
+	// ensures predictable result, so Changed() works properly
+	sort.Slice(b.Endpoints, func(i, j int) bool {
+		return b.Endpoints[i].Target < b.Endpoints[j].Target
+	})
 	return ep
 }
diff --git a/pkg/haproxy/types/types.go b/pkg/haproxy/types/types.go
index 2b761c13..df565b16 100644
--- a/pkg/haproxy/types/types.go
+++ b/pkg/haproxy/types/types.go
@@ -82,6 +82,7 @@ type Global struct {
 	UseHTX                  bool
 	DefaultBackendRedir     string
 	DefaultBackendRedirCode int
+	NoRedirects             []string
 	CustomConfig            []string
 	CustomDefaults          []string
 	CustomFrontendEarly     []string
diff --git a/rootfs/etc/templates/haproxy/haproxy.tmpl b/rootfs/etc/templates/haproxy/haproxy.tmpl
index 06c42216..3e9ee244 100644
--- a/rootfs/etc/templates/haproxy/haproxy.tmpl
+++ b/rootfs/etc/templates/haproxy/haproxy.tmpl
@@ -1184,11 +1184,7 @@ frontend {{ $proxy__front_http }}
 {{- end }}
 
 {{- /*------------------------------------*/}}
-{{- if $global.Acme.Enabled }}
-    {{- template "redirectTo" map $frontend $fmaps "!acme-challenge" }}
-{{- else }}
-    {{- template "redirectTo" map $frontend $fmaps "" }}
-{{- end }}
+{{- template "redirectTo" map $global $frontend $fmaps }}
 
 {{- /*------------------------------------*/}}
 {{- template "authExternalFrontend" map $hosts }}
@@ -1209,11 +1205,7 @@ frontend {{ $proxy__front_http }}
 {{- end }}
 
 {{- /*------------------------------------*/}}
-{{- if $global.Acme.Enabled }}
-    {{- template "redirectFrom" map $frontend $fmaps "req.backend" "!acme-challenge" }}
-{{- else }}
-    {{- template "redirectFrom" map $frontend $fmaps "req.backend" "" }}
-{{- end }}
+{{- template "redirectFrom" map $global $frontend $fmaps "req.backend" }}
 
 {{- /*------------------------------------*/}}
 {{- template "sourceIP" map $global }}
@@ -1279,7 +1271,7 @@ frontend {{ $frontend.Name }}
     http-request set-var(req.base) var(req.host),concat(\#,req.path)
 
 {{- /*------------------------------------*/}}
-{{- template "redirectTo" map $frontend $fmaps "" }}
+{{- template "redirectTo" map $global $frontend $fmaps }}
 
 {{- /*------------------------------------*/}}
 {{- template "authExternalFrontend" map $hosts }}
@@ -1300,7 +1292,7 @@ frontend {{ $frontend.Name }}
 {{- end }}
 
 {{- /*------------------------------------*/}}
-{{- template "redirectFrom" map $frontend $fmaps "req.hostbackend" "" }}
+{{- template "redirectFrom" map $global $frontend $fmaps "req.hostbackend" }}
 
 {{- /*------------------------------------*/}}
 {{- if $fmaps.RedirFromRootMap.HasHost }}
@@ -1448,24 +1440,24 @@ frontend {{ $frontend.Name }}
 {{- /*------------------------------------*/}}
 {{- /*------------------------------------*/}}
 {{- define "redirectFrom" }}
-{{- $frontend := .p1 }}
-{{- $fmaps := .p2 }}
-{{- $varbe := .p3 }}
-{{- $acls := .p4 }}
+{{- $global := .p1 }}
+{{- $frontend := .p2 }}
+{{- $fmaps := .p3 }}
+{{- $varbe := .p4 }}
 {{- if $fmaps.RedirFromMap.MatchFiles }}
 {{- range $match := $fmaps.RedirFromMap.MatchFiles }}
     http-request set-var(req.redirdest) var(req.host)
         {{- if $match.Lower }},lower{{ end }}
         {{- "" }},map_{{ $match.Method }}({{ $match.Filename }})
         {{- "" }} if
-        {{- if $acls }} {{ $acls }}{{ end }}
+        {{- if $global.NoRedirects }} !{ path_beg{{ range $global.NoRedirects }} "{{ . }}"{{ end }} }{{ end }}
         {{- "" }} !{ var({{ $varbe }}) -m found }
         {{- if not $match.First }} !{ var(req.redirdest) -m found }{{ end }}
 {{- end }}
     http-request redirect prefix //%[var(req.redirdest)]
         {{- "" }} code {{ $frontend.RedirectFromCode }}
         {{- "" }} if
-        {{- if $acls }} {{ $acls }}{{ end }}
+        {{- if $global.NoRedirects }} !{ path_beg{{ range $global.NoRedirects }} "{{ . }}"{{ end }} }{{ end }}
         {{- "" }} { var(req.redirdest) -m found }
 {{- end }}
 {{- end }}
@@ -1493,21 +1485,21 @@ frontend {{ $frontend.Name }}
 {{- /*------------------------------------*/}}
 {{- /*------------------------------------*/}}
 {{- define "redirectTo" }}
-{{- $frontend := .p1 }}
-{{- $fmaps := .p2 }}
-{{- $acls := .p3 }}
+{{- $global := .p1 }}
+{{- $frontend := .p2 }}
+{{- $fmaps := .p3 }}
 {{- if $fmaps.RedirToMap.MatchFiles }}
 {{- range $match := $fmaps.RedirToMap.MatchFiles }}
     http-request set-var(req.redirto) var(req.base)
         {{- if $match.Lower }},lower{{ end }}
         {{- "" }},map_{{ $match.Method }}({{ $match.Filename }})
-        {{- if $acls }} if {{ $acls }}{{ end }}
-        {{- template "httpFilters" map $match "req.redirto" (not $acls) }}
+        {{- if $global.NoRedirects }} if !{ path_beg{{ range $global.NoRedirects }} "{{ . }}"{{ end }} }{{ end }}
+        {{- template "httpFilters" map $match "req.redirto" (not $global.NoRedirects) }}
 {{- end }}
     http-request redirect location %[var(req.redirto)]
         {{- "" }} code {{ $frontend.RedirectToCode }}
         {{- "" }} if
-        {{- if $acls }} {{ $acls }}{{ end }}
+        {{- if $global.NoRedirects }} !{ path_beg{{ range $global.NoRedirects }} "{{ . }}"{{ end }} }{{ end }}
         {{- "" }} { var(req.redirto) -m found }
 {{- end }}
 {{- end }}



Test patch:
diff --git a/pkg/haproxy/instance_test.go b/pkg/haproxy/instance_test.go
index c77b7dbd..f9b43167 100644
--- a/pkg/haproxy/instance_test.go
+++ b/pkg/haproxy/instance_test.go
@@ -4194,7 +4194,7 @@ func TestInstanceRedirectFrom(t *testing.T) {
 	testCases := []struct {
 		data     [3]hatypes.HostRedirectConfig
 		code     int
-		acme     bool
+		noredirs []string
 		expHTTP  string
 		expHTTPS string
 		expMaps  map[string]string
@@ -4248,14 +4248,14 @@ sub.d2.local d2.local
 			data: [3]hatypes.HostRedirectConfig{
 				{RedirectHost: "*.d1.local"},
 			},
-			code: 301,
-			acme: true,
+			code:     301,
+			noredirs: []string{"/.well-known/acme-challenge"},
 			expHTTP: `
-    http-request set-var(req.redirdest) var(req.host),map_reg(/etc/haproxy/maps/_front_redir_from__regex.map) if !acme-challenge !{ var(req.backend) -m found }
-    http-request redirect prefix //%[var(req.redirdest)] code 301 if !acme-challenge { var(req.redirdest) -m found }`,
+    http-request set-var(req.redirdest) var(req.host),map_reg(/etc/haproxy/maps/_front_redir_from__regex.map) if !{ path_beg "/.well-known/acme-challenge" } !{ var(req.backend) -m found }
+    http-request redirect prefix //%[var(req.redirdest)] code 301 if !{ path_beg "/.well-known/acme-challenge" } { var(req.redirdest) -m found }`,
 			expHTTPS: `
-    http-request set-var(req.redirdest) var(req.host),map_reg(/etc/haproxy/maps/_front_redir_from__regex.map) if !{ var(req.hostbackend) -m found }
-    http-request redirect prefix //%[var(req.redirdest)] code 301 if { var(req.redirdest) -m found }`,
+    http-request set-var(req.redirdest) var(req.host),map_reg(/etc/haproxy/maps/_front_redir_from__regex.map) if !{ path_beg "/.well-known/acme-challenge" } !{ var(req.hostbackend) -m found }
+    http-request redirect prefix //%[var(req.redirdest)] code 301 if !{ path_beg "/.well-known/acme-challenge" } { var(req.redirdest) -m found }`,
 			expMaps: map[string]string{
 				"_front_redir_from__regex.map": `
 ^[^.]+\.d1\.local$ d1.local
@@ -4268,6 +4268,8 @@ sub.d2.local d2.local
 		c := setup(t)
 		defer c.teardown()
 
+		c.config.global.NoRedirects = test.noredirs
+
 		var h *hatypes.Host
 		var b = c.config.Backends().AcquireBackend("d1", "app", "8080")
 		b.Endpoints = []*hatypes.Endpoint{endpointS1}
@@ -4293,20 +4295,6 @@ sub.d2.local d2.local
 			c.config.frontend.RedirectFromCode = 302
 		}
 
-		var acmeBackend, acmeACL, acmeUseBackend string
-		if test.acme {
-			acmeBackend = `
-backend _acme_challenge
-    mode http
-    server _acme_server unix@local`
-			acmeACL = `
-    acl acme-challenge path_beg `
-			acmeUseBackend = `
-    use_backend _acme_challenge if acme-challenge`
-			c.config.global.Acme.Socket = "local"
-			c.config.global.Acme.Enabled = true
-		}
-
 		c.Update()
 		c.checkConfig(`
 <<global>>
@@ -4319,11 +4307,11 @@ backend d2_app_8080
     server s21 172.17.0.121:8080 weight 100
 backend d3_app_8080
     mode http
-    server s31 172.17.0.131:8080 weight 100` + acmeBackend + `
+    server s31 172.17.0.131:8080 weight 100
 <<backends-default>>
 frontend _front_http
     mode http
-    bind :80` + acmeACL + `
+    bind :80
     http-request set-var(req.path) path
     http-request set-var(req.host) hdr(host),field(1,:),lower
     http-request set-var(req.base) var(req.host),concat(\#,req.path)
@@ -4333,7 +4321,7 @@ frontend _front_http
     http-request del-header X-SSL-Client-SHA1
     http-request del-header X-SSL-Client-SHA2
     http-request del-header X-SSL-Client-Cert
-    http-request set-var(req.backend) var(req.base),lower,map_beg(/etc/haproxy/maps/_front_http_host__begin.map)` + test.expHTTP + acmeUseBackend + `
+    http-request set-var(req.backend) var(req.base),lower,map_beg(/etc/haproxy/maps/_front_http_host__begin.map)` + test.expHTTP + `
     use_backend %[var(req.backend)] if { var(req.backend) -m found }
     default_backend _error404
 frontend _front_https
@@ -4364,9 +4352,8 @@ func TestInstanceRedirectTo(t *testing.T) {
 	testCases := []struct {
 		to       [3]string
 		code     int
-		acme     bool
+		noredirs []string
 		expected string
-		expHTTPS string
 		expMaps  map[string]string
 	}{
 		// 0
@@ -4424,13 +4411,10 @@ d1.local#/app2 https://app.local/app2
 			to: [3]string{
 				"https://app.local",
 			},
-			acme: true,
+			noredirs: []string{"/.well-known/acme-challenge"},
 			expected: `
-    http-request set-var(req.redirto) var(req.base),lower,map_beg(/etc/haproxy/maps/_front_redir_to__begin.map) if !acme-challenge
-    http-request redirect location %[var(req.redirto)] code 302 if !acme-challenge { var(req.redirto) -m found }`,
-			expHTTPS: `
-    http-request set-var(req.redirto) var(req.base),lower,map_beg(/etc/haproxy/maps/_front_redir_to__begin.map)
-    http-request redirect location %[var(req.redirto)] code 302 if { var(req.redirto) -m found }`,
+    http-request set-var(req.redirto) var(req.base),lower,map_beg(/etc/haproxy/maps/_front_redir_to__begin.map) if !{ path_beg "/.well-known/acme-challenge" }
+    http-request redirect location %[var(req.redirto)] code 302 if !{ path_beg "/.well-known/acme-challenge" } { var(req.redirto) -m found }`,
 			expMaps: map[string]string{
 				"_front_redir_to__begin.map": `
 d1.local#/ https://app.local
@@ -4443,6 +4427,8 @@ d1.local#/ https://app.local
 		c := setup(t)
 		defer c.teardown()
 
+		c.config.global.NoRedirects = test.noredirs
+
 		var h = c.config.Hosts().AcquireHost("d1.local")
 		var b = c.config.Backends().AcquireBackend("d1", "app", "8080")
 		b.Endpoints = []*hatypes.Endpoint{endpointS1}
@@ -4468,30 +4454,12 @@ d1.local#/ https://app.local
 			h.AddPath(b, "/app3", hatypes.MatchBegin)
 		}
 
-		if test.expHTTPS == "" {
-			test.expHTTPS = test.expected
-		}
-
 		if test.code != 0 {
 			c.config.frontend.RedirectToCode = test.code
 		} else {
 			c.config.frontend.RedirectToCode = 302
 		}
 
-		var acmeBackend, acmeACL, acmeUseBackend string
-		if test.acme {
-			acmeBackend = `
-backend _acme_challenge
-    mode http
-    server _acme_server unix@local`
-			acmeACL = `
-    acl acme-challenge path_beg `
-			acmeUseBackend = `
-    use_backend _acme_challenge if acme-challenge`
-			c.config.global.Acme.Socket = "local"
-			c.config.global.Acme.Enabled = true
-		}
-
 		c.Update()
 		c.checkConfig(`
 <<global>>
@@ -4504,11 +4472,11 @@ backend d2_app_8080
     server s21 172.17.0.121:8080 weight 100
 backend d3_app_8080
     mode http
-    server s31 172.17.0.131:8080 weight 100` + acmeBackend + `
+    server s31 172.17.0.131:8080 weight 100
 <<backends-default>>
 frontend _front_http
     mode http
-    bind :80` + acmeACL + `
+    bind :80
     http-request set-var(req.path) path
     http-request set-var(req.host) hdr(host),field(1,:),lower
     http-request set-var(req.base) var(req.host),concat(\#,req.path)
@@ -4518,7 +4486,7 @@ frontend _front_http
     http-request del-header X-SSL-Client-SHA1
     http-request del-header X-SSL-Client-SHA2
     http-request del-header X-SSL-Client-Cert` + test.expected + `
-    http-request set-var(req.backend) var(req.base),lower,map_beg(/etc/haproxy/maps/_front_http_host__begin.map)` + acmeUseBackend + `
+    http-request set-var(req.backend) var(req.base),lower,map_beg(/etc/haproxy/maps/_front_http_host__begin.map)
     use_backend %[var(req.backend)] if { var(req.backend) -m found }
     default_backend _error404
 frontend _front_https
@@ -4526,7 +4494,7 @@ frontend _front_https
     bind :443 ssl alpn h2,http/1.1 crt-list /etc/haproxy/maps/_front_bind_crt.list ca-ignore-err all crt-ignore-err all
     http-request set-var(req.path) path
     http-request set-var(req.host) hdr(host),field(1,:),lower
-    http-request set-var(req.base) var(req.host),concat(\#,req.path)` + test.expHTTPS + `
+    http-request set-var(req.base) var(req.host),concat(\#,req.path)` + test.expected + `
     http-request set-var(req.hostbackend) var(req.base),lower,map_beg(/etc/haproxy/maps/_front_https_host__begin.map)
     http-request set-header X-Forwarded-Proto https
     http-request del-header X-SSL-Client-CN