Chapter 1. Using the MicroShift configuration file A YAML file customizes MicroShift instances with your preferences, settings, and parameters. Note If you want to make configuration changes or deploy applications through the MicroShift API with tools other than kustomize manifests, you must wait until the greenboot health checks have finished. This ensures that your changes are not lost if greenboot rolls your rpm-ostree system back to an earlier state. 1.1. The MicroShift YAML configuration file At start up, MicroShift checks the system-wide /etc/microshift/ directory for a configuration file named config.yaml . If the configuration file does not exist in the directory, built-in default values are used to start the service. The MicroShift configuration file must be used in combination with host and, sometimes, application and service settings. Ensure that each item is configured in tandem when you customize your MicroShift cluster. 1.1.1. Default settings If you do not create a config.yaml file or use a configuration snippet YAML file, default values are used. The following example shows the default configuration settings. To see the default values, run the following command: USD microshift show-config Default values example output in YAML form apiServer: advertiseAddress: 10.44.0.0/32 1 auditLog: maxFileAge: 0 maxFileSize: 200 maxFiles: 10 profile: Default namedCertificates: - certPath: "" keyPath: "" names: - "" subjectAltNames: [] debugging: logLevel: "Normal" dns: baseDomain: microshift.example.com etcd: memoryLimitMB: 0 ingress: defaultHTTPVersion: 1 forwardedHeaderPolicy: "" httpCompression: mimeTypes: - "" httpEmptyRequestsPolicy: Respond listenAddress: - "" logEmptyRequests: Log ports: http: 80 https: 443 routeAdmissionPolicy: namespaceOwnership: InterNamespaceAllowed status: Managed tuningOptions: clientFinTimeout: "" clientTimeout: "" headerBufferBytes: 0 headerBufferMaxRewriteBytes: 0 healthCheckInterval: "" maxConnections: 0 serverFinTimeout: "" serverTimeout: "" threadCount: 0 tlsInspectDelay: "" tunnelTimeout: "" kubelet: manifests: kustomizePaths: - /usr/lib/microshift/manifests - /usr/lib/microshift/manifests.d/* - /etc/microshift/manifests - /etc/microshift/manifests.d/* network: clusterNetwork: - 10.42.0.0/16 serviceNetwork: - 10.43.0.0/16 serviceNodePortRange: 30000-32767 node: hostnameOverride: "" nodeIP: "" 2 nodeIPv6: "" storage: driver: "" 3 optionalCsiComponents: 4 - "" 1 Calculated based on the address of the service network. 2 The IP address of the default route. 3 Default null value deploys Logical Volume Managed Storage (LVMS). 4 Default null value deploys snapshot-controller . 1.2. Using custom settings To create custom configurations, make a copy of the config.yaml.default file that is provided in the /etc/microshift/ directory, renaming it config.yaml . Keep this file in the /etc/microshift/ directory, and then you can change supported settings that are expected to override the defaults before starting or restarting MicroShift. Important Restart MicroShift after changing any configuration settings to have them take effect. The config.yaml file is read only when MicroShift starts. 1.2.1. Separate restarts Applications and other optional services used with your MicroShift cluster might also need to be restarted separately to apply configuration changes throughout the cluster. For example, when making changes to certain networking settings, you must stop and restart service and application pods to apply those changes. See each procedure for the task you are completing for more information. Tip If you add all of the configurations you need at the same time, you can minimize system restarts. 1.2.2. Parameters and values for the MicroShift config.yaml file The following table explains MicroShift configuration YAML parameters and valid values for each: Table 1.1. MicroShift config.yaml parameters Field Type Description advertiseAddress string A string that specifies the IP address from which the API server is advertised to members of the cluster. The default value is calculated based on the address of the service network. auditLog.maxFileAge number How long log files are kept before automatic deletion. The default value of 0 in the maxFileAge parameter means a log file is never deleted based on age. This value can be configured. auditLog.maxFileSize number By default, when the audit.log file reaches the maxFileSize limit, the audit.log file is rotated and MicroShift begins writing to a new audit.log file. This value can be configured. auditLog.maxFiles number The total number of log files kept. By default, MicroShift retains 10 log files. The oldest is deleted when an excess file is created. This value can be configured. auditLog.profile Default , WriteRequestBodies , AllRequestBodies , or None Logs only metadata for read and write requests; does not log request bodies except for OAuth access token requests. If you do not specify this field, the Default profile is used. namedCertificates list Defines externally generated certificates and domain names by using custom certificate authorities. namedCertificates.certPath path The full path to the certificate. namedCertificates.keyPath path The full path to the certificate key. namedCertificates.names list Optional. Add a list of explicit DNS names. Leading wildcards are allowed. If no names are provided, the implicit names are extracted from the certificates. subjectAltNames Fully qualified domain names (FQDNs), wildcards such as *.domain.com , or IP addresses. Subject Alternative Names for API server certificates. SANs indicate all of the domain names and IP addresses that are secured by a certificate. debugging.logLevel Normal , Debug , Trace , or TraceAll Log verbosity. Default is Normal . dns.baseDomain valid domain Base domain of the cluster. All managed DNS records are subdomains of this base. etcd.memoryLimitMB number By default, etcd uses as much memory as needed to handle the load on the system. However, in memory constrained systems, it might be preferred or necessary to limit the amount of memory etcd can to use at a given time. ingress.defaultHTTPVersion number Determines the default HTTP version to be used for ingress. Default value is 1 , which is the HTTP/1.1 protocol. ingress.forwardedHeaderPolicy Append , Replace , IfNone , Never Specifies when and how the ingress router sets the Forwarded , X-Forwarded-For , X-Forwarded-Host , X-Forwarded-Port , X-Forwarded-Proto , and X-Forwarded-Proto-Version HTTP headers. Append specifies that the ingress router appends existing headers. Append is the default value. Replace specifies that the ingress router sets the headers and replaces any existing Forwarded or X-Forwarded-* headers. IfNone specifies that the ingress router sets headers if they are not already set. Never specifies that ingress router never sets the headers, preserving any existing headers. ingress.httpCompression object httpCompression defines a policy for HTTP traffic compression. There is no HTTP compression by default. ingress.httpCompression.mimeTypes array or null mimeTypes is a list of MIME types to compress. When the list is empty, the ingress controller does not apply any compression. To define a list, use the format of the Content-Type definition in RFC 1341 that specifies the type and subtype of data in the body of a message and the native encoding of the data. For example, Content-Type := type \"/\" subtype *[\";\" parameter] . The value of Content-Type can be one of the following types: application, audio, image, message, multipart, text, video, or a custom type preceded by \"X-\" and followed by a token. The token must be defined in one of the following ways: The token is a string of at least one character, and does not contain white spaces, control characters, or any of the characters in the tspecials set. The tspecials set contains the characters ()\u003c\u003e@,;:\\\"/[]?.= . The subtype in Content-Type is also a token. The optional parameters following the subtype are defined as token \"=\" (token / quoted-string) . The quoted-string , as defined in RFC 822, is surrounded by double quotes and can contain white spaces plus any character except \\ , \" , and CR . The quoted-string can also contain any single ASCII character if it is escaped by the following characters: \\.", . Not all MIME types benefit from compression, but HAProxy uses resources to try to compress files when compression is configured. Generally speaking, text formats such as html , ccs , and js benefit from compression. Spending CPU resources to compress file types that are already compressed, such as images, audio, and video, is probably not worth the limited benefit. ingress.httpEmptyRequestsPolicy Respond or Ignore The default value is Respond . Describes how HTTP connections should be handled if the connection times out before a request is received. These connections typically come from the health probes of a load balancer service health or a web browser's speculative connections, such as a preconnect . If the field is set to Respond , the ingress controller sends an "HTTP 400" or "408" response, logs the connection if access logging is enabled, and counts the connection in the appropriate metrics. If the field is set to Ignore , the ingress controller closes the connection without sending a response, logging the connection, or incrementing metrics. Setting this field to Ignore might impede detection and diagnosis of problems or intrusions, especially when timed-out connections are caused by network errors or port scans. In both cases, logging empty requests can be useful for diagnosing errors and detecting intrusion attempts. ingress.listenAddress IP address, NIC name, or multiple Value defaults to the entire network of the host. The valid configurable value is a list that can be either a single IP address or NIC name or multiple IP addresses and NIC names. ingress.logEmptyRequests Log or Ignore Default value is Log . Specifies how connections on which empty requests are received are logged. These connections typically come from the health probes of a load balancer service health or a web browser's speculative connections, such as a preconnect . Logging typical requests might be undesirable, but requests can also be caused by network errors or port scans, in which case logging can be useful for diagnosing errors and detecting intrusion attempts. ingress.ports.http 80 Default port shown. Configurable. Valid value is a single, unique port in the 1-65535 range. The values of the ports.http and ports.https fields cannot be the same. ingress.ports.https 443 Default port shown. Configurable. Valid value is a single, unique port in the 1-65535 range. The values of the ports.http and ports.https fields cannot be the same. ingress.routeAdmissionPolicy.namespaceOwnership Strict or InterNamespaceAllowed Describes how hostname claims across namespaces are handled. By default, allows routes to claim different paths of the same hostname across namespaces. Specifying Strict prevents routes in different namespaces from claiming the same hostname. If the value is deleted in a customized MicroShift config.yaml , the InterNamespaceAllowed value is automatically set. ingress.status Managed or Removed Router status. Default is Managed . ingress.tuningOptions Objects Specifies options for tuning the performance of ingress controller pods. ingress.tuningOptions.clientFinTimeout string with format duration Defines how long a connection is held open while waiting for a client response to the server/backend before closing the connection. The default timeout is 1s , which is 1 second. ingress.tuningOptions.clientTimeout string with format duration Defines how long a connection is held open while waiting for a client response. The default timeout is 30s , which is 30 seconds. ingress.tuningOptions.headerBufferBytes An integer with the format of int32 ; 16384 is the minimum value when HTTP/2 is enabled. Describes how much memory in bytes must be reserved for IngressController connection sessions. Default value is 32768 in bytes. Setting this field is generally not recommended because headerBufferBytes values that are too small can break the IngressController and headerBufferBytes values that are too large can cause the IngressController to use significantly more memory than necessary. ingress.tuningOptions.headerBufferMaxRewriteBytes integer , formatted int32 ; 4096 is the minimum value Describes how much memory in bytes must be reserved from headerBufferBytes for HTTP header rewriting and appending for IngressController connection sessions. Default value is 8192 bytes. Incoming HTTP requests are limited to the headerBufferBytes bytes minus the headerBufferMaxRewriteBytes bytes, meaning that the value of headerBufferBytes must be greater than the value of headerBufferMaxRewriteBytes . Setting this field is generally not recommended because headerBufferMaxRewriteBytes values that are too small can break the IngressController and headerBufferMaxRewriteBytes values that are too large can cause the IngressController to use significantly more memory than necessary. ingress.tuningOptions.healthCheckInterval: "" string with pattern: ^(0|( (\\.[0-9] )?(ns|us|ms|ms|ms|s|m|h))+)USD The default healthCheckInterval value is 5s , which is 5 seconds. This parameter value defines how long the router waits between two consecutive health checks on the router's configured backends. Currently the minimum allowed value is 1s and the maximum allowed value is 2147483647ms , which is 24.85 days. The range might change in future releases. This value is applied globally as a default for all routes, but can be overridden per-route by the route annotation router.openshift.io/haproxy.health.check.interval . Requires an unsigned duration string of decimal numbers, each with an optional fraction and unit suffix, such as 300ms , 1.5h or 2h45m . Valid time units are ns , us (or ms U+00B5 or ms U+03BC), ms , s , m , h . Setting this parameter value to less than 5s can cause excess traffic due to too frequent TCP health checks and accompanying SYN packet storms. Setting this parameter value too high can result in increased latency because of backend servers that are no longer available, but have not yet been detected as such. An empty or 0 value means "no opinion" and the ingress controller chooses a default. Note that the default value might change in future releases. ingress.tuningOptions.maxConnections integer , valid values are: empty , 0 , -1 , and the range 2000-2000000 Default value is 0 . defines the maximum number of simultaneous connections that can be established per HAProxy process. Increasing this value allows each ingress controller pod to handle more connections at the cost of additional system resources being consumed. If this field is empty or 0 , the IngressController uses the default value of 50000 , but the default is subject to change in future releases. If the value is -1 , then HAProxy dynamically computes a maximum value based on the available resources set with ulimit values in the running container. Selecting -1 , which means auto , results in a large value being computed, and therefore each HAProxy process incurs significant memory usage compared with the current default of 50000 . Setting a value that is greater than the current operating system limit prevents the HAProxy process from starting. You can monitor memory usage for router containers with the following metric: container_memory_working_set_bytes{container=`router`,namespace=`openshift-ingress`}` You can monitor memory usage of individual `HAProxy`processes in router containers with the following metric: container_memory_working_set_bytes{container=`router`,namespace=`openshift-ingress`}/container_processes{container=`router`,namespace=`openshift-ingress`} ingress.tuningOptions.serverFinTimeout string in the format duration Defines how long a connection is held open while waiting for a server or backend response to the client before closing the connection. The default timeout is 1s . ingress.tuningOptions.serverTimeout string in the format duration Defines how long a connection is held open while waiting for a server or backend response. The default timeout is 30s . ingress.tuningOptions.threadCount integer in the form int32 ; minimum value is 1 , maximum is 64 Defines the number of threads created per HAProxy process. The default value is 4 . If this field is empty, the default value is used. Setting this field is generally not recommended. Creating more threads allows each ingress controller pod to handle more connections at the cost of more system resources being used. Increasing the number of HAProxy threads allows the ingress controller pods to use more CPU time under load, potentially starving other pods if set too high. Conversely, reducing the number of threads may cause the ingress controller to perform poorly. ingress.tuningOptions.tlsInspectDelay string in the format duration Defines how long the router can hold data to find a matching route. Setting this interval with too short a value can cause the router to revert to the default certificate for edge-terminated clients or re-encrypt routes, even when a better-matching certificate could be used. The default inspect delay is 5s which is 5 seconds, which is expected to be sufficient for most cases. Increasing the value of this configuration specifically for high-latency networks can cause a delay in finishing the SSL handshake. Any configured value must be transparent to your application. ingress.tuningOptions.tunnelTimeout string in the format duration Defines how long a tunnel connection, including websockets, are held open while the tunnel is idle. The default timeout is 1h , which is 1 hour. kubelet See the MicroShift low-latency instructions Parameter for passthrough configuration of the kubelet node agent. Used for low-latency configuration. Default value is null. manifests list of paths The locations on the file system to scan for kustomization files to use to load manifests. Set to a list of paths to scan only those paths. Set to an empty list to disable loading manifests. The entries in the list can be glob patterns to match multiple subdirectories. Default values are /usr/lib/microshift/manifests , /usr/lib/microshift/manifests.d/ , /etc/microshift/manifests , and /etc/microshift/manifests.d/ . network.clusterNetwork IP address block A block of IP addresses from which pod IP addresses are allocated. IPv4 is the default network. Dual-stack entries are supported. The first entry in this field is immutable after MicroShift starts. Default range is 10.42.0.0/16 . network.serviceNetwork IP address block A block of virtual IP addresses for Kubernetes services. IP address pool for services. IPv4 is the default. Dual-stack entries are supported. The first entry in this field is immutable after MicroShift starts. Default range is 10.43.0.0/16 . network.serviceNodePortRange range The port range allowed for Kubernetes services of type NodePort . If not specified, the default range of 30000-32767 is used. Services without a NodePort specified are automatically allocated one from this range. This parameter can be updated after MicroShift starts. node.hostnameOverride string The name of the node. The default value is the hostname. If non-empty, this string is used to identify the node instead of the hostname. This value is immutable after MicroShift starts. node.nodeIP IPv4 address The IPv4 address of the node. The default value is the IP address of the default route. nodeIPv6 IPv6 address The IPv6 address for the node for dual-stack configurations. Cannot be configured in single stack for either IPv4 or IPv6. Default is an empty value or null. storage.driver none or lvms Default value is empty. An empty value or null field defaults to LVMS deployment. storage.optionalCsiComponents array Default value is null or an empty array. A null or empty array defaults to deploying snapshot-controller . Expected values are csi-snapshot-controller or none . A value of none is mutually exclusive with all other values. 1.2.3. Using configuration snippets If you want to configure one or two settings, such as adding subject alternative names (SANs), you can use the /etc/microshift/config.d/ configuration directory to drop in configuration snippet YAML files. You must restart MicroShift for new configurations to apply. To return to values, you can delete a configuration snippet and restart MicroShift. 1.2.3.1. How configuration snippets work At runtime, the YAML files inside /etc/microshift/config.d are merged into the existing MicroShift configuration, whether that configuration is a result of default values or a user-created config.yaml file. You do not need to create a config.yaml file to use a configuration snippet. Files in the snippet directory are sorted in lexicographical order and run sequentially. You can use numerical prefixes for snippets so that each is read in the order you want. The last-read file takes precedence when there is more than one YAML for the same parameter. Important Configuration snippets take precedence over both default values and a customized config.yaml configuration file. 1.2.3.2. Example list configuration snippets Lists, or arrays, are not merged, they are overwritten. For example, you can replace a SAN or list of SANs by creating an additional snippet for the same field that is read after the first: MicroShift configuration directory contents /etc/microshift/config.yaml.default or /etc/microshift/config.yaml Example MicroShift configuration snippet directory contents /etc/microshift/config.d/10-san.yaml /etc/microshift/config.d/20-san.yaml Example 10-san.yaml snippet apiServer: subjectAltNames: - host1 - host2 Example 20-san.yaml snippet apiServer: subjectAltNames: - hostZ Example configuration result apiServer: subjectAltNames: - hostZ If you want to add a value to an existing list, you can add it to an existing snippet. For example, to add hostZ to an existing list of SANs, edit the snippet you have instead of creating a new one: Example 10-san.yaml snippet apiServer: subjectAltNames: - host1 - host2 - hostZ Example configuration result apiServer: subjectAltNames: - host1 - host2 - hostZ 1.2.3.3. Example object configuration snippets Objects are merged together. Example 10-advertiseAddress.yaml snippet apiServer: advertiseAddress: "microshift-example" Example 20-audit-log.yaml snippet apiServer: auditLog: maxFileAge: 12 Example configuration result apiServer: advertiseAddress: "microshift-example" auditLog: maxFileAge: 12 1.2.3.4. Example mixed configuration snippets In this example, the values of both advertiseAddress and auditLog.maxFileAge fields are merged into the configuration, but only the c.com and d.com subjectAltNames values are retained because the numbering in the filename indicates that they are higher priority. Example 10-advertiseAddress.yaml snippet apiServer: advertiseAddress: "microshift-example" Example 20-audit-log.yaml snippet apiServer: auditLog: maxFileAge: 12 Example 30-SAN.yaml snippet apiServer: subjectAltNames: - a.com - b.com Example 40-SAN.yaml snippet apiServer: subjectAltNames: - c.com - d.com Example configuration result apiServer: advertiseAddress: "microshift-example" auditLog: maxFileAge: 12 subjectAltNames: - c.com - d.com 1.2.4. Configuring the advertise address network flag The apiserver.advertiseAddress flag specifies the IP address on which to advertise the API server to members of the cluster. This address must be reachable by the cluster. You can set a custom IP address here, but you must also add the IP address to a host interface. Customizing this parameter preempts MicroShift from adding a default IP address to the br-ex network interface. Important If you customize the advertiseAddress IP address, make sure it is reachable by the cluster when MicroShift starts by adding the IP address to a host interface. If unset, the default value is set to the immediate subnet after the service network. For example, when the service network is 10.43.0.0/16 , the advertiseAddress is set to 10.44.0.0/32 . 1.2.5. Extending the port range for NodePort services The serviceNodePortRange setting extends the port range available to NodePort services. This option is useful when specific standard ports under the 30000-32767 range need to be exposed. For example, if your device needs to expose the 1883/tcp MQ Telemetry Transport (MQTT) port on the network because client devices cannot use a different port. Important NodePorts can overlap with system ports, causing a malfunction of the system or MicroShift. Consider the following when configuring the NodePort service ranges: Do not create any NodePort service without an explicit nodePort selection. When an explicit nodePort is not specified, the port is assigned randomly by the kube-apiserver and cannot be predicted. Do not create any NodePort service for any system service port, MicroShift port, or other services you expose on your device HostNetwork . Table one specifies ports to avoid when extending the port range: Table 1.2. Ports to avoid. Port Description 22/tcp SSH port 80/tcp OpenShift Router HTTP endpoint 443/tcp OpenShift Router HTTPS endpoint 1936/tcp Metrics service for the openshift-router, not exposed today 2379/tcp etcd port 2380/tcp etcd port 6443 kubernetes API 8445/tcp openshift-route-controller-manager 9537/tcp cri-o metrics 10250/tcp kubelet 10248/tcp kubelet healthz port 10259/tcp kube scheduler 1.3. Additional resources Checking Greenboot status

Commands:
microshift show-config
apiServer: advertiseAddress: 10.44.0.0/32 1 auditLog: maxFileAge: 0 maxFileSize: 200 maxFiles: 10 profile: Default namedCertificates: - certPath: "" keyPath: "" names: - "" subjectAltNames: [] debugging: logLevel: "Normal" dns: baseDomain: microshift.example.com etcd: memoryLimitMB: 0 ingress: defaultHTTPVersion: 1 forwardedHeaderPolicy: "" httpCompression: mimeTypes: - "" httpEmptyRequestsPolicy: Respond listenAddress: - "" logEmptyRequests: Log ports: http: 80 https: 443 routeAdmissionPolicy: namespaceOwnership: InterNamespaceAllowed status: Managed tuningOptions: clientFinTimeout: "" clientTimeout: "" headerBufferBytes: 0 headerBufferMaxRewriteBytes: 0 healthCheckInterval: "" maxConnections: 0 serverFinTimeout: "" serverTimeout: "" threadCount: 0 tlsInspectDelay: "" tunnelTimeout: "" kubelet: manifests: kustomizePaths: - /usr/lib/microshift/manifests - /usr/lib/microshift/manifests.d/* - /etc/microshift/manifests - /etc/microshift/manifests.d/* network: clusterNetwork: - 10.42.0.0/16 serviceNetwork: - 10.43.0.0/16 serviceNodePortRange: 30000-32767 node: hostnameOverride: "" nodeIP: "" 2 nodeIPv6: "" storage: driver: "" 3 optionalCsiComponents: 4 - ""
container_memory_working_set_bytes{container=`router`,namespace=`openshift-ingress`}`
container_memory_working_set_bytes{container=`router`,namespace=`openshift-ingress`}/container_processes{container=`router`,namespace=`openshift-ingress`}
apiServer: subjectAltNames: - host1 - host2
apiServer: subjectAltNames: - hostZ
apiServer: subjectAltNames: - hostZ
apiServer: subjectAltNames: - host1 - host2 - hostZ
apiServer: subjectAltNames: - host1 - host2 - hostZ
apiServer: advertiseAddress: "microshift-example"
apiServer: auditLog: maxFileAge: 12
apiServer: advertiseAddress: "microshift-example" auditLog: maxFileAge: 12
apiServer: advertiseAddress: "microshift-example"
apiServer: auditLog: maxFileAge: 12
apiServer: subjectAltNames: - a.com - b.com
apiServer: subjectAltNames: - c.com - d.com
apiServer: advertiseAddress: "microshift-example" auditLog: maxFileAge: 12 subjectAltNames: - c.com - d.com