Repository: traefik/traefik (language: go)

Issue:
Kubernetes ingress, no load balancer IPv6 address being populated
### Welcome!

- [X] Yes, I've searched similar issues on [GitHub](https://github.com/traefik/traefik/issues) and didn't find any.
- [X] Yes, I've searched similar issues on the [Traefik community forum](https://community.containo.us) and didn't find any.

### What did you do?

1. Install k3s, with Calico and IPv6 cluster-cidr and service-cidr, disable included Traefik
2. Deploy Traefik using Helm. Values provided below.
3. Deploy dashboard Ingress
4. Notice that even if the svc has IPv6 externalIP and IPv6 PodIP, but the dashboard ingress doesn't have IPv6 IPs at all in `status.loadBalancer.ip`.

### What did you see instead?

Rather than seeing IPv6 address and IPv4 address in the ingress status.loadbalancer.ip, there was only v4.

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"kustomize.toolkit.fluxcd.io/checksum":"10bec159e9fa75646e669270fca6bb43820aef49","traefik.ingress.kubernetes.io/router.entrypoints":"websecure"},"labels":{"kustomize.toolkit.fluxcd.io/name":"apps","kustomize.toolkit.fluxcd.io/namespace":"flux-system"},"name":"traefik-ipv6-dashboard","namespace":"networking"},"spec":{"ingressClassName":"traefik-v6","rules":[{"host":"traefik-v6.k8s.<snip>","http":{"paths":[{"backend":{"service":{"name":"traefik","port":{"number":9000}}},"path":"/","pathType":"Prefix"}]}}],"tls":[{"secretName":"k8s.<snip>-tls"}]}}
    kustomize.toolkit.fluxcd.io/checksum: 10bec159e9fa75646e669270fca6bb43820aef49
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
  creationTimestamp: "2021-08-27T00:30:27Z"
  generation: 2
  labels:
    kustomize.toolkit.fluxcd.io/name: apps
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: traefik-ipv6-dashboard
  namespace: networking
  resourceVersion: "3683269"
  uid: 3c6692c8-1013-4744-b06c-49d406d246fd
spec:
  ingressClassName: traefik-v6
  rules:
  - host: traefik-v6.k8s.<snip>
    http:
      paths:
      - backend:
          service:
            name: traefik
            port:
              number: 9000
        path: /
        pathType: Prefix
  tls:
  - secretName: k8s.<snip>-tls
status:
  loadBalancer:
    ingress:
    - ip: 185.xxx.xx.250
```

### What version of Traefik are you using?

Using Chart version 10.3.2.

### What is your environment & configuration?

```yaml
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: traefik-v6
  namespace: networking
spec:
  interval: 5m
  chart:
    spec:
      # renovate: registryUrl=https://helm.traefik.io/traefik
      chart: traefik
      version: 10.3.2
      sourceRef:
        kind: HelmRepository
        name: traefik-charts
        namespace: flux-system
      interval: 5m
  values:
    deployment:
      kind: Deployment
      replicas: 2
    service:
      enabled: true
      type: LoadBalancer
      spec:
        ipFamilyPolicy: PreferDualStack
        externalIPs:
        - "${SVC_TRAEFIK_ADDRESS}"
        externalTrafficPolicy: Local
    logs:
      general:
        format: json
        level: INFO
      access:
        enabled: true
        format: json
    ingressClass:
      enabled: true
      isDefaultClass: true
      fallbackApiVersion: v1
    ingressRoute:
      dashboard:
        enabled: false
    globalArguments:
    - "--api.insecure=true"
    - "--serverstransport.insecureskipverify=true"
    - "--providers.kubernetesingress.ingressclass=traefik-v6"
    - "--metrics.prometheus=true"
    - "--metrics.prometheus.entryPoint=metrics"
    - "--entryPoints.websecure.forwardedHeaders.trustedIPs=10.0.0.0/8,192.168.0.0/16,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,104.16.0.0/13,104.24.0.0/14,108.162.192.0/18,131.0.72.0/22,141.101.64.0/18,162.158.0.0/15,172.64.0.0/13,173.245.48.0/20,188.114.96.0/20,190.93.240.0/20,197.234.240.0/22,198.41.128.0/17,2400:cb00::/32,2606:4700::/32,2803:f800::/32,2405:b500::/32,2405:8100::/32,2a06:98c0::/29,2c0f:f248::/32"
    additionalArguments:
    - "--providers.kubernetesingress.ingressendpoint.ip=${SVC_TRAEFIK_ADDRESS}"
    ports:
      traefik:
        expose: true
      web:
        redirectTo: websecure
      websecure:
        tls:
          enabled: true
          options: "default"
      metrics:
        port: 8082
        expose: false
        exposedPort: 8082
    tlsOptions:
      default:
        minVersion: VersionTLS12
        maxVersion: VersionTLS13
#        sniStrict: true
    pilot:
      enabled: false
      token: ""
    experimental:
      plugins:
        enabled: false
    nodeSelector:
      node-role.kubernetes.io/worker: "true"
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app.kubernetes.io/name
              operator: In
              values:
              - traefik
          topologyKey: kubernetes.io/hostname
    resources:
      requests:
        memory: 100Mi
        cpu: 500m
      limits:
        memory: 300Mi
```

Dashboard ingress:
```yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: traefik-ipv6-dashboard
  namespace: networking
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
spec:
  ingressClassName: traefik-v6
  tls:
  - secretName: "${K8S_DOMAIN}-tls"
  rules:
  - host: traefik-v6.${K8S_DOMAIN}
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: traefik
            port:
              number: 9000
```


```yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: traefik-v6
    meta.helm.sh/release-namespace: networking
  creationTimestamp: "2021-08-27T00:30:39Z"
  labels:
    app.kubernetes.io/instance: traefik-v6
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: traefik
    helm.sh/chart: traefik-10.3.2
    helm.toolkit.fluxcd.io/name: traefik-v6
    helm.toolkit.fluxcd.io/namespace: networking
  name: traefik-v6
  namespace: networking
  resourceVersion: "3684971"
  uid: 0669efa2-2d0a-4012-aa7b-4c8fa9440e8f
spec:
  clusterIP: 10.96.192.215
  clusterIPs:
  - 10.96.192.215
  - 2001:xxx:xxx:fdb::7ec
  externalIPs:
  - 2001:xxx:xxxx:fc00::443
  externalTrafficPolicy: Local
  healthCheckNodePort: 32016
  ipFamilies:
  - IPv4
  - IPv6
  ipFamilyPolicy: PreferDualStack
  ports:
  - name: traefik
    nodePort: 32539
    port: 9000
    protocol: TCP
    targetPort: traefik
  - name: web
    nodePort: 31290
    port: 80
    protocol: TCP
    targetPort: web
  - name: websecure
    nodePort: 30766
    port: 443
    protocol: TCP
    targetPort: websecure
  selector:
    app.kubernetes.io/instance: traefik-v6
    app.kubernetes.io/name: traefik
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}
```

### If applicable, please paste the log output in DEBUG level

_No response_
Kubernetes ingress, no load balancer IPv6 address being populated
### Welcome!

- [X] Yes, I've searched similar issues on [GitHub](https://github.com/traefik/traefik/issues) and didn't find any.
- [X] Yes, I've searched similar issues on the [Traefik community forum](https://community.containo.us) and didn't find any.

### What did you do?

1. Install k3s, with Calico and IPv6 cluster-cidr and service-cidr, disable included Traefik
2. Deploy Traefik using Helm. Values provided below.
3. Deploy dashboard Ingress
4. Notice that even if the svc has IPv6 externalIP and IPv6 PodIP, but the dashboard ingress doesn't have IPv6 IPs at all in `status.loadBalancer.ip`.

### What did you see instead?

Rather than seeing IPv6 address and IPv4 address in the ingress status.loadbalancer.ip, there was only v4.

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"kustomize.toolkit.fluxcd.io/checksum":"10bec159e9fa75646e669270fca6bb43820aef49","traefik.ingress.kubernetes.io/router.entrypoints":"websecure"},"labels":{"kustomize.toolkit.fluxcd.io/name":"apps","kustomize.toolkit.fluxcd.io/namespace":"flux-system"},"name":"traefik-ipv6-dashboard","namespace":"networking"},"spec":{"ingressClassName":"traefik-v6","rules":[{"host":"traefik-v6.k8s.<snip>","http":{"paths":[{"backend":{"service":{"name":"traefik","port":{"number":9000}}},"path":"/","pathType":"Prefix"}]}}],"tls":[{"secretName":"k8s.<snip>-tls"}]}}
    kustomize.toolkit.fluxcd.io/checksum: 10bec159e9fa75646e669270fca6bb43820aef49
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
  creationTimestamp: "2021-08-27T00:30:27Z"
  generation: 2
  labels:
    kustomize.toolkit.fluxcd.io/name: apps
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: traefik-ipv6-dashboard
  namespace: networking
  resourceVersion: "3683269"
  uid: 3c6692c8-1013-4744-b06c-49d406d246fd
spec:
  ingressClassName: traefik-v6
  rules:
  - host: traefik-v6.k8s.<snip>
    http:
      paths:
      - backend:
          service:
            name: traefik
            port:
              number: 9000
        path: /
        pathType: Prefix
  tls:
  - secretName: k8s.<snip>-tls
status:
  loadBalancer:
    ingress:
    - ip: 185.xxx.xx.250
```

### What version of Traefik are you using?

Using Chart version 10.3.2.

### What is your environment & configuration?

```yaml
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: traefik-v6
  namespace: networking
spec:
  interval: 5m
  chart:
    spec:
      # renovate: registryUrl=https://helm.traefik.io/traefik
      chart: traefik
      version: 10.3.2
      sourceRef:
        kind: HelmRepository
        name: traefik-charts
        namespace: flux-system
      interval: 5m
  values:
    deployment:
      kind: Deployment
      replicas: 2
    service:
      enabled: true
      type: LoadBalancer
      spec:
        ipFamilyPolicy: PreferDualStack
        externalIPs:
        - "${SVC_TRAEFIK_ADDRESS}"
        externalTrafficPolicy: Local
    logs:
      general:
        format: json
        level: INFO
      access:
        enabled: true
        format: json
    ingressClass:
      enabled: true
      isDefaultClass: true
      fallbackApiVersion: v1
    ingressRoute:
      dashboard:
        enabled: false
    globalArguments:
    - "--api.insecure=true"
    - "--serverstransport.insecureskipverify=true"
    - "--providers.kubernetesingress.ingressclass=traefik-v6"
    - "--metrics.prometheus=true"
    - "--metrics.prometheus.entryPoint=metrics"
    - "--entryPoints.websecure.forwardedHeaders.trustedIPs=10.0.0.0/8,192.168.0.0/16,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,104.16.0.0/13,104.24.0.0/14,108.162.192.0/18,131.0.72.0/22,141.101.64.0/18,162.158.0.0/15,172.64.0.0/13,173.245.48.0/20,188.114.96.0/20,190.93.240.0/20,197.234.240.0/22,198.41.128.0/17,2400:cb00::/32,2606:4700::/32,2803:f800::/32,2405:b500::/32,2405:8100::/32,2a06:98c0::/29,2c0f:f248::/32"
    additionalArguments:
    - "--providers.kubernetesingress.ingressendpoint.ip=${SVC_TRAEFIK_ADDRESS}"
    ports:
      traefik:
        expose: true
      web:
        redirectTo: websecure
      websecure:
        tls:
          enabled: true
          options: "default"
      metrics:
        port: 8082
        expose: false
        exposedPort: 8082
    tlsOptions:
      default:
        minVersion: VersionTLS12
        maxVersion: VersionTLS13
#        sniStrict: true
    pilot:
      enabled: false
      token: ""
    experimental:
      plugins:
        enabled: false
    nodeSelector:
      node-role.kubernetes.io/worker: "true"
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app.kubernetes.io/name
              operator: In
              values:
              - traefik
          topologyKey: kubernetes.io/hostname
    resources:
      requests:
        memory: 100Mi
        cpu: 500m
      limits:
        memory: 300Mi
```

Dashboard ingress:
```yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: traefik-ipv6-dashboard
  namespace: networking
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
spec:
  ingressClassName: traefik-v6
  tls:
  - secretName: "${K8S_DOMAIN}-tls"
  rules:
  - host: traefik-v6.${K8S_DOMAIN}
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: traefik
            port:
              number: 9000
```


```yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: traefik-v6
    meta.helm.sh/release-namespace: networking
  creationTimestamp: "2021-08-27T00:30:39Z"
  labels:
    app.kubernetes.io/instance: traefik-v6
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: traefik
    helm.sh/chart: traefik-10.3.2
    helm.toolkit.fluxcd.io/name: traefik-v6
    helm.toolkit.fluxcd.io/namespace: networking
  name: traefik-v6
  namespace: networking
  resourceVersion: "3684971"
  uid: 0669efa2-2d0a-4012-aa7b-4c8fa9440e8f
spec:
  clusterIP: 10.96.192.215
  clusterIPs:
  - 10.96.192.215
  - 2001:xxx:xxx:fdb::7ec
  externalIPs:
  - 2001:xxx:xxxx:fc00::443
  externalTrafficPolicy: Local
  healthCheckNodePort: 32016
  ipFamilies:
  - IPv4
  - IPv6
  ipFamilyPolicy: PreferDualStack
  ports:
  - name: traefik
    nodePort: 32539
    port: 9000
    protocol: TCP
    targetPort: traefik
  - name: web
    nodePort: 31290
    port: 80
    protocol: TCP
    targetPort: web
  - name: websecure
    nodePort: 30766
    port: 443
    protocol: TCP
    targetPort: websecure
  selector:
    app.kubernetes.io/instance: traefik-v6
    app.kubernetes.io/name: traefik
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}
```

### If applicable, please paste the log output in DEBUG level

_No response_


Discussion / hints:
It seems traefik code doesn't get the backend addresses from the EndpointSlices

https://github.com/traefik/traefik/blob/36ffdf548dc197e10ad53f87797dbc5e11792e72/pkg/provider/kubernetes/ingress/kubernetes.go#L554

Endpoints only carry IP addresses of the primary IP family, you have to use EndpointSlices if you want to get the endpoints for each IP family

The Service.Status.LoadBalancer can be misleading, is the same definition as the Ingress.Status.Loadbalancer but both represent different things

Could you explain what you mean with EndpointSlices? @aojea 
> Could you explain what you mean with EndpointSlices? @aojea

https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/
Any updates regarding this?
This would be really needed for proper IPv6 support, as it's heading stable. Could someone please look into this?
Any idea or minestones for when this can get implemented? 
We are now in 2022, any ideas maintainers?
As of right now, [K3s](https://github.com/k3s-io/k3s) has dual-stack support, except for this bug in Traefik. As such, we must recommend that users disable Traefik on K3s if they want dual-stack and instead use ingress-nginx. We would love for this to get fixed!
Right now as Traefik maintainers seems to be thinking of going to 3.0, this really should be addressed before then.
I'm planning on working on this in the Traefik 3.0 Hackaethon, and would appreciate help with doing so.
https://traefik.io/blog/announcing-traefik-proxy-3-0-hackaethon/
After a talk with the maintainers on the Hackaetron Discord, I have a better understanding of how this is supposed to work.

One should be using the `--providers.kubernetesingress.ingressendpoint.publishedService` rather than IP, so it would copy the IPs from the service, and they would show up in status.loadbalancer.ingress.ip.

https://github.com/samip5/k8s-cluster/blob/2fe121e52258c2f63ab403daccc7ad8f0e6cd97a/cluster/apps/networking/traefik-test/helm-release.yaml#L54 this works now, and ingress does get the IPs

@dereknola The default manifest for Traefik in k3s needs to be changed to use that.

I had forgotten about this issue, and I don't want users to find this and be confused now.
Today, K3s + Traefik fully support dual-stack, it is how we suggest users implement dual-stack support. We resolved an internal issue back on https://github.com/k3s-io/k3s/commit/9ac2648f8d8c71024ea182da8579f70e363dbe97 about a month after I commented.
> I had forgotten about this issue, and I don't want users to find this and be confused now. Today, K3s + Traefik fully support dual-stack, it is how we suggest users implement dual-stack support. We resolved an internal issue back on [k3s-io/k3s@9ac2648](https://github.com/k3s-io/k3s/commit/9ac2648f8d8c71024ea182da8579f70e363dbe97) about a month after I commented.

~~Has this been tested as that alone will not bring dual-stack support with Traefik? If you try to curl the traefik svc's IPv6 address, it will more than likely fail if the cluster is IPv4, IPv6 aka IPv4 being the default IP family.~~

~~EDIT: This is originally why I was so confused with this as ipfamilypolicy wasn't enough.~~

It seems that it has indeed been enabled by default:
https://github.com/k3s-io/k3s/blob/a3b35d21e92b10cd4e032dbd7494c806847cd3a8/manifests/traefik.yaml#L29L32
It seems traefik code doesn't get the backend addresses from the EndpointSlices

https://github.com/traefik/traefik/blob/36ffdf548dc197e10ad53f87797dbc5e11792e72/pkg/provider/kubernetes/ingress/kubernetes.go#L554

Endpoints only carry IP addresses of the primary IP family, you have to use EndpointSlices if you want to get the endpoints for each IP family

The Service.Status.LoadBalancer can be misleading, is the same definition as the Ingress.Status.Loadbalancer but both represent different things

Could you explain what you mean with EndpointSlices? @aojea 
> Could you explain what you mean with EndpointSlices? @aojea

https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/
Any updates regarding this?
This would be really needed for proper IPv6 support, as it's heading stable. Could someone please look into this?
Any idea or minestones for when this can get implemented? 
We are now in 2022, any ideas maintainers?
As of right now, [K3s](https://github.com/k3s-io/k3s) has dual-stack support, except for this bug in Traefik. As such, we must recommend that users disable Traefik on K3s if they want dual-stack and instead use ingress-nginx. We would love for this to get fixed!
Right now as Traefik maintainers seems to be thinking of going to 3.0, this really should be addressed before then.
I'm planning on working on this in the Traefik 3.0 Hackaethon, and would appreciate help with doing so.
https://traefik.io/blog/announcing-traefik-proxy-3-0-hackaethon/
After a talk with the maintainers on the Hackaetron Discord, I have a better understanding of how this is supposed to work.

One should be using the `--providers.kubernetesingress.ingressendpoint.publishedService` rather than IP, so it would copy the IPs from the service, and they would show up in status.loadbalancer.ingress.ip.

https://github.com/samip5/k8s-cluster/blob/2fe121e52258c2f63ab403daccc7ad8f0e6cd97a/cluster/apps/networking/traefik-test/helm-release.yaml#L54 this works now, and ingress does get the IPs

@dereknola The default manifest for Traefik in k3s needs to be changed to use that.

I had forgotten about this issue, and I don't want users to find this and be confused now.
Today, K3s + Traefik fully support dual-stack, it is how we suggest users implement dual-stack support. We resolved an internal issue back on https://github.com/k3s-io/k3s/commit/9ac2648f8d8c71024ea182da8579f70e363dbe97 about a month after I commented.
> I had forgotten about this issue, and I don't want users to find this and be confused now. Today, K3s + Traefik fully support dual-stack, it is how we suggest users implement dual-stack support. We resolved an internal issue back on [k3s-io/k3s@9ac2648](https://github.com/k3s-io/k3s/commit/9ac2648f8d8c71024ea182da8579f70e363dbe97) about a month after I commented.

~~Has this been tested as that alone will not bring dual-stack support with Traefik? If you try to curl the traefik svc's IPv6 address, it will more than likely fail if the cluster is IPv4, IPv6 aka IPv4 being the default IP family.~~

~~EDIT: This is originally why I was so confused with this as ipfamilypolicy wasn't enough.~~

It seems that it has indeed been enabled by default:
https://github.com/k3s-io/k3s/blob/a3b35d21e92b10cd4e032dbd7494c806847cd3a8/manifests/traefik.yaml#L29L32

Fix patch:
diff --git a/docs/content/providers/kubernetes-ingress.md b/docs/content/providers/kubernetes-ingress.md
index 801820ee21..bafa082b65 100644
--- a/docs/content/providers/kubernetes-ingress.md
+++ b/docs/content/providers/kubernetes-ingress.md
@@ -374,7 +374,7 @@ providers:
 
 _Optional, Default: ""_
 
-IP used for Kubernetes Ingress endpoints.
+This IP will get copied to Ingress `status.loadbalancer.ip`, and currently only supports one IP value (IPv4 or IPv6).
 
 ```yaml tab="File (YAML)"
 providers:
@@ -398,7 +398,9 @@ providers:
 
 _Optional, Default: ""_
 
-Published Kubernetes Service to copy status from.
+The Kubernetes service to copy status from.
+When using third parties tools like External-DNS, this option can be used to copy the service `loadbalancer.status` (containing the service's endpoints IPs) to the ingresses.
+
 Format: `namespace/servicename`.
 
 ```yaml tab="File (YAML)"


Test patch: