Chapter 8. DNS Operator in OpenShift Container Platform The DNS Operator deploys and manages CoreDNS to provide a name resolution service to pods, enabling DNS-based Kubernetes Service discovery in OpenShift Container Platform. 8.1. DNS Operator The DNS Operator implements the dns API from the operator.openshift.io API group. The Operator deploys CoreDNS using a daemon set, creates a service for the daemon set, and configures the kubelet to instruct pods to use the CoreDNS service IP address for name resolution. Procedure The DNS Operator is deployed during installation with a Deployment object. Use the oc get command to view the deployment status: USD oc get -n openshift-dns-operator deployment/dns-operator Example output NAME READY UP-TO-DATE AVAILABLE AGE dns-operator 1/1 1 1 23h Use the oc get command to view the state of the DNS Operator: USD oc get clusteroperator/dns Example output NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE dns 4.1.0-0.11 True False False 92m AVAILABLE , PROGRESSING and DEGRADED provide information about the status of the operator. AVAILABLE is True when at least 1 pod from the CoreDNS daemon set reports an Available status condition. 8.2. Changing the DNS Operator managementState DNS manages the CoreDNS component to provide a name resolution service for pods and services in the cluster. The managementState of the DNS Operator is set to Managed by default, which means that the DNS Operator is actively managing its resources. You can change it to Unmanaged , which means the DNS Operator is not managing its resources. The following are use cases for changing the DNS Operator managementState : You are a developer and want to test a configuration change to see if it fixes an issue in CoreDNS. You can stop the DNS Operator from overwriting the fix by setting the managementState to Unmanaged . You are a cluster administrator and have reported an issue with CoreDNS, but need to apply a workaround until the issue is fixed. You can set the managementState field of the DNS Operator to Unmanaged to apply the workaround. Procedure Change managementState DNS Operator: oc patch dns.operator.openshift.io default --type merge --patch '{"spec":{"managementState":"Unmanaged"}}' 8.3. Controlling DNS pod placement The DNS Operator has two daemon sets: one for CoreDNS and one for managing the /etc/hosts file. The daemon set for /etc/hosts must run on every node host to add an entry for the cluster image registry to support pulling images. Security policies can prohibit communication between pairs of nodes, which prevents the daemon set for CoreDNS from running on every node. As a cluster administrator, you can use a custom node selector to configure the daemon set for CoreDNS to run or not run on certain nodes. Prerequisites You installed the oc CLI. You are logged in to the cluster with a user with cluster-admin privileges. Procedure To prevent communication between certain nodes, configure the spec.nodePlacement.nodeSelector API field: Modify the DNS Operator object named default : USD oc edit dns.operator/default Specify a node selector that includes only control plane nodes in the spec.nodePlacement.nodeSelector API field: spec: nodePlacement: nodeSelector: node-role.kubernetes.io/worker: "" To allow the daemon set for CoreDNS to run on nodes, configure a taint and toleration: Modify the DNS Operator object named default : USD oc edit dns.operator/default Specify a taint key and a toleration for the taint: spec: nodePlacement: tolerations: - effect: NoExecute key: "dns-only" operators: Equal value: abc tolerationSeconds: 3600 1 1 If the taint is dns-only , it can be tolerated indefinitely. You can omit tolerationSeconds . 8.4. View the default DNS Every new OpenShift Container Platform installation has a dns.operator named default . Procedure Use the oc describe command to view the default dns : USD oc describe dns.operator/default Example output Name: default Namespace: Labels: <none> Annotations: <none> API Version: operator.openshift.io/v1 Kind: DNS ... Status: Cluster Domain: cluster.local 1 Cluster IP: 172.30.0.10 2 ... 1 The Cluster Domain field is the base DNS domain used to construct fully qualified pod and service domain names. 2 The Cluster IP is the address pods query for name resolution. The IP is defined as the 10th address in the service CIDR range. To find the service CIDR of your cluster, use the oc get command: USD oc get networks.config/cluster -o jsonpath='{USD.status.serviceNetwork}' Example output [172.30.0.0/16] 8.5. Using DNS forwarding You can use DNS forwarding to override the default forwarding configuration in the /etc/resolv.conf file in the following ways: Specify name servers for every zone. If the forwarded zone is the Ingress domain managed by OpenShift Container Platform, then the upstream name server must be authorized for the domain. Provide a list of upstream DNS servers. Change the default forwarding policy. Note A DNS forwarding configuration for the default domain can have both the default servers specified in the /etc/resolv.conf file and the upstream DNS servers. Procedure Modify the DNS Operator object named default : USD oc edit dns.operator/default After you issue the command, the Operator creates and updates the config map named dns-default with additional server configuration blocks based on Server . If none of the servers have a zone that matches the query, then name resolution falls back to the upstream DNS servers. Configuring DNS forwarding apiVersion: operator.openshift.io/v1 kind: DNS metadata: name: default spec: servers: - name: example-server 1 zones: 2 - example.com forwardPlugin: policy: Random 3 upstreams: 4 - 1.1.1.1 - 2.2.2.2:5353 upstreamResolvers: 5 policy: Random 6 upstreams: 7 - type: SystemResolvConf 8 - type: Network address: 1.2.3.4 9 port: 53 10 1 Must comply with the rfc6335 service name syntax. 2 Must conform to the definition of a subdomain in the rfc1123 service name syntax. The cluster domain, cluster.local , is an invalid subdomain for the zones field. 3 Defines the policy to select upstream resolvers. Default value is Random . You can also use the values RoundRobin , and Sequential . 4 A maximum of 15 upstreams is allowed per forwardPlugin . 5 Optional. You can use it to override the default policy and forward DNS resolution to the specified DNS resolvers (upstream resolvers) for the default domain. If you do not provide any upstream resolvers, the DNS name queries go to the servers in /etc/resolv.conf . 6 Determines the order in which upstream servers are selected for querying. You can specify one of these values: Random , RoundRobin , or Sequential . The default value is Sequential . 7 Optional. You can use it to provide upstream resolvers. 8 You can specify two types of upstreams - SystemResolvConf and Network . SystemResolvConf configures the upstream to use /etc/resolv.conf and Network defines a Networkresolver . You can specify one or both. 9 If the specified type is Network , you must provide an IP address. The address field must be a valid IPv4 or IPv6 address. 10 If the specified type is Network , you can optionally provide a port. The port field must have a value between 1 and 65535 . If you do not specify a port for the upstream, by default port 853 is tried. Optional: When working in a highly regulated environment, you might need the ability to secure DNS traffic when forwarding requests to upstream resolvers so that you can ensure additional DNS traffic and data privacy. Cluster administrators can configure transport layer security (TLS) for forwarded DNS queries. Configuring DNS forwarding with TLS apiVersion: operator.openshift.io/v1 kind: DNS metadata: name: default spec: servers: - name: example-server 1 zones: 2 - example.com forwardPlugin: transportConfig: transport: TLS 3 tls: caBundle: name: mycacert serverName: dnstls.example.com 4 policy: Random 5 upstreams: 6 - 1.1.1.1 - 2.2.2.2:5353 upstreamResolvers: 7 transportConfig: transport: TLS tls: caBundle: name: mycacert serverName: dnstls.example.com upstreams: - type: Network 8 address: 1.2.3.4 9 port: 53 10 1 Must comply with the rfc6335 service name syntax. 2 Must conform to the definition of a subdomain in the rfc1123 service name syntax. The cluster domain, cluster.local , is an invalid subdomain for the zones field. The cluster domain, cluster.local , is an invalid subdomain for zones . 3 When configuring TLS for forwarded DNS queries, set the transport field to have the value TLS . By default, CoreDNS caches forwarded connections for 10 seconds. CoreDNS will hold a TCP connection open for those 10 seconds if no request is issued. With large clusters, ensure that your DNS server is aware that it might get many new connections to hold open because you can initiate a connection per node. Set up your DNS hierarchy accordingly to avoid performance issues. 4 When configuring TLS for forwarded DNS queries, this is a mandatory server name used as part of the server name indication (SNI) to validate the upstream TLS server certificate. 5 Defines the policy to select upstream resolvers. Default value is Random . You can also use the values RoundRobin , and Sequential . 6 Required. You can use it to provide upstream resolvers. A maximum of 15 upstreams entries are allowed per forwardPlugin entry. 7 Optional. You can use it to override the default policy and forward DNS resolution to the specified DNS resolvers (upstream resolvers) for the default domain. If you do not provide any upstream resolvers, the DNS name queries go to the servers in /etc/resolv.conf . 8 Network type indicates that this upstream resolver should handle forwarded requests separately from the upstream resolvers listed in /etc/resolv.conf . Only the Network type is allowed when using TLS and you must provide an IP address. 9 The address field must be a valid IPv4 or IPv6 address. 10 You can optionally provide a port. The port must have a value between 1 and 65535 . If you do not specify a port for the upstream, by default port 853 is tried. Note If servers is undefined or invalid, the config map only contains the default server. Verification View the config map: USD oc get configmap/dns-default -n openshift-dns -o yaml Sample DNS ConfigMap based on sample DNS apiVersion: v1 data: Corefile: | example.com:5353 { forward . 1.1.1.1 2.2.2.2:5353 } bar.com:5353 example.com:5353 { forward . 3.3.3.3 4.4.4.4:5454 1 } .:5353 { errors health kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf 1.2.3.4:53 { policy Random } cache 30 reload } kind: ConfigMap metadata: labels: dns.operator.openshift.io/owning-dns: default name: dns-default namespace: openshift-dns 1 Changes to the forwardPlugin triggers a rolling update of the CoreDNS daemon set. Additional resources For more information on DNS forwarding, see the CoreDNS forward documentation . 8.6. DNS Operator status You can inspect the status and view the details of the DNS Operator using the oc describe command. Procedure View the status of the DNS Operator: USD oc describe clusteroperators/dns 8.7. DNS Operator logs You can view DNS Operator logs by using the oc logs command. Procedure View the logs of the DNS Operator: USD oc logs -n openshift-dns-operator deployment/dns-operator -c dns-operator 8.8. Setting the CoreDNS log level You can configure the CoreDNS log level to determine the amount of detail in logged error messages. The valid values for CoreDNS log level are Normal , Debug , and Trace . The default logLevel is Normal . Note The errors plugin is always enabled. The following logLevel settings report different error responses: logLevel : Normal enables the "errors" class: log . { class error } . logLevel : Debug enables the "denial" class: log . { class denial error } . logLevel : Trace enables the "all" class: log . { class all } . Procedure To set logLevel to Debug , enter the following command: USD oc patch dnses.operator.openshift.io/default -p '{"spec":{"logLevel":"Debug"}}' --type=merge To set logLevel to Trace , enter the following command: USD oc patch dnses.operator.openshift.io/default -p '{"spec":{"logLevel":"Trace"}}' --type=merge Verification To ensure the desired log level was set, check the config map: USD oc get configmap/dns-default -n openshift-dns -o yaml 8.9. Setting the CoreDNS Operator log level Cluster administrators can configure the Operator log level to more quickly track down OpenShift DNS issues. The valid values for operatorLogLevel are Normal , Debug , and Trace . Trace has the most detailed information. The default operatorlogLevel is Normal . There are seven logging levels for issues: Trace, Debug, Info, Warning, Error, Fatal and Panic. After the logging level is set, log entries with that severity or anything above it will be logged. operatorLogLevel: "Normal" sets logrus.SetLogLevel("Info") . operatorLogLevel: "Debug" sets logrus.SetLogLevel("Debug") . operatorLogLevel: "Trace" sets logrus.SetLogLevel("Trace") . Procedure To set operatorLogLevel to Debug , enter the following command: USD oc patch dnses.operator.openshift.io/default -p '{"spec":{"operatorLogLevel":"Debug"}}' --type=merge To set operatorLogLevel to Trace , enter the following command: USD oc patch dnses.operator.openshift.io/default -p '{"spec":{"operatorLogLevel":"Trace"}}' --type=merge 8.10. Tuning the CoreDNS cache You can configure the maximum duration of both successful or unsuccessful caching, also known as positive or negative caching respectively, done by CoreDNS. Tuning the duration of caching of DNS query responses can reduce the load for any upstream DNS resolvers. Procedure Edit the DNS Operator object named default by running the following command: USD oc edit dns.operator.openshift.io/default Modify the time-to-live (TTL) caching values: Configuring DNS caching apiVersion: operator.openshift.io/v1 kind: DNS metadata: name: default spec: cache: positiveTTL: 1h 1 negativeTTL: 0.5h10m 2 1 The string value 1h is converted to its respective number of seconds by CoreDNS. If this field is omitted, the value is assumed to be 0s and the cluster uses the internal default value of 900s as a fallback. 2 The string value can be a combination of units such as 0.5h10m and is converted to its respective number of seconds by CoreDNS. If this field is omitted, the value is assumed to be 0s and the cluster uses the internal default value of 30s as a fallback. Warning Setting TTL fields to low values could lead to an increased load on the cluster, any upstream resolvers, or both.

Commands:
oc get -n openshift-dns-operator deployment/dns-operator
NAME READY UP-TO-DATE AVAILABLE AGE dns-operator 1/1 1 1 23h
oc get clusteroperator/dns
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE dns 4.1.0-0.11 True False False 92m
patch dns.operator.openshift.io default --type merge --patch '{"spec":{"managementState":"Unmanaged"}}'
oc edit dns.operator/default
spec: nodePlacement: nodeSelector: node-role.kubernetes.io/worker: ""
oc edit dns.operator/default
spec: nodePlacement: tolerations: - effect: NoExecute key: "dns-only" operators: Equal value: abc tolerationSeconds: 3600 1
oc describe dns.operator/default
Name: default Namespace: Labels: <none> Annotations: <none> API Version: operator.openshift.io/v1 Kind: DNS Status: Cluster Domain: cluster.local 1 Cluster IP: 172.30.0.10 2
oc get networks.config/cluster -o jsonpath='{USD.status.serviceNetwork}'
[172.30.0.0/16]
oc edit dns.operator/default
apiVersion: operator.openshift.io/v1 kind: DNS metadata: name: default spec: servers: - name: example-server 1 zones: 2 - example.com forwardPlugin: policy: Random 3 upstreams: 4 - 1.1.1.1 - 2.2.2.2:5353 upstreamResolvers: 5 policy: Random 6 upstreams: 7 - type: SystemResolvConf 8 - type: Network address: 1.2.3.4 9 port: 53 10
apiVersion: operator.openshift.io/v1 kind: DNS metadata: name: default spec: servers: - name: example-server 1 zones: 2 - example.com forwardPlugin: transportConfig: transport: TLS 3 tls: caBundle: name: mycacert serverName: dnstls.example.com 4 policy: Random 5 upstreams: 6 - 1.1.1.1 - 2.2.2.2:5353 upstreamResolvers: 7 transportConfig: transport: TLS tls: caBundle: name: mycacert serverName: dnstls.example.com upstreams: - type: Network 8 address: 1.2.3.4 9 port: 53 10
oc get configmap/dns-default -n openshift-dns -o yaml
apiVersion: v1 data: Corefile: | example.com:5353 { forward . 1.1.1.1 2.2.2.2:5353 } bar.com:5353 example.com:5353 { forward . 3.3.3.3 4.4.4.4:5454 1 } .:5353 { errors health kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf 1.2.3.4:53 { policy Random } cache 30 reload } kind: ConfigMap metadata: labels: dns.operator.openshift.io/owning-dns: default name: dns-default namespace: openshift-dns
oc describe clusteroperators/dns
oc logs -n openshift-dns-operator deployment/dns-operator -c dns-operator
oc patch dnses.operator.openshift.io/default -p '{"spec":{"logLevel":"Debug"}}' --type=merge
oc patch dnses.operator.openshift.io/default -p '{"spec":{"logLevel":"Trace"}}' --type=merge
oc get configmap/dns-default -n openshift-dns -o yaml
oc patch dnses.operator.openshift.io/default -p '{"spec":{"operatorLogLevel":"Debug"}}' --type=merge
oc patch dnses.operator.openshift.io/default -p '{"spec":{"operatorLogLevel":"Trace"}}' --type=merge
oc edit dns.operator.openshift.io/default
apiVersion: operator.openshift.io/v1 kind: DNS metadata: name: default spec: cache: positiveTTL: 1h 1 negativeTTL: 0.5h10m 2