Repository: kubernetes-sigs/node-feature-discovery
Pull request #464 (2021-03-04 20:55:35)

More extensive and expressive custom rules
## Overview

The main goal of this PR is to greatly expand the capabilities of the custom source, and, to make easier to extend in the future. After this change basically all features from the `cpu`, `kernel`, `system` `pci`, `usb` and `local` source are available for custom rule processing. This is achieved in three steps

1. Decouple "raw" feature discovery and label generation in the feature sources
1. Introduce a generalized and more expressive rule processing infrastructure, inspired by the MatchExpressions pattern in K8s nodeSelector (also, convert all existing rules to utilize this)
1. Implement a generic per domain/feature custom rule for matching all features from sources that support "raw" feature discovery (i.e. `cpu`, `kernel`, `system` `pci`, `usb` and `local` in this PR)

This PR introduces a new format for custom rules where the new capabilities are available under new `matchFeatures` directive. The old rule format (with `matchOn`) is still supported to provice backwards compatibility.

## Rule syntax

A new format for custom rule specifications is introduced, dropping the 'value' field and introducing new 'labels' field which makes it possible to specify multiple labels per rule. Also, in the new format the 'name' field is just for reference and no matching label is created.

```
  - name: <rule-name>
    labels:
      <key>: <value>
      ...
    matchFeatures:
      - feature: <domain>.<feature>
        matchExpressions:
          <attribute>:
            op: <operator>
            value:
              - <list-of-values>
      - feature: <domain>.<feature>
        ...
```

Currently supported `op`s are `In`, `NotIn`, `InRegexp`, `Exists`, `DoesNotExist`, `Gt` `Lt`, `GtLt`, `IsTrue`, `IsFalse`.

The list of available `<domain>.<feature>`'s is:

- cpu.cpuid:
    &lt;cpuid-flag&gt;: &lt;exists/does-not-exist&gt;
- cpu.cstate:
    enabled: &lt;bool&gt;
- cpu.pstate:
    status: &lt;string&gt;
    turbo: &lt;bool&gt;
    scaling_governor: &lt;string&gt;
- cpu.rdt:
    &lt;rdt-feature&gt;: &lt;exists/does-not-exist&gt;
- cpu.sst:
    bf.enabled: &lt;bool&gt;
- cpu.topology:
    hardware_multithreading: &lt;bool&gt;
- kernel.config:
    &lt;flag-name&gt;: &lt;string&gt;
- kernel.loadedmodule:
    &lt;module-name&gt;: &lt;exists/does-not-exist&gt;
- kernel.selinux:
    enabled: &lt;bool&gt;
- kernel.version:
    major: &lt;int&gt;
    minor: &lt;int&gt;
    revision: &lt;int&gt;
    full: &lt;string&gt;
- system.osrelease:
    &lt;key-name&gt;: &lt;string&gt;
    VERSION_ID.major: &lt;int&gt;
    VERSION_ID.minor: &lt;int&gt;
- system.name:
    nodename: &lt;string&gt;
- pci.device:
    &lt;device-instance&gt;:
      class: &lt;string&gt;
      vendor: &lt;string&gt;
      device: &lt;string&gt;
      subsystem_vendor: &lt;string&gt;
      susbystem_device: &lt;string&gt;
      sriov_totalvfs: &lt;int&gt;
- usb.device:
    &lt;device-instance&gt;:
      class: &lt;string&gt;
      vendor: &lt;string&gt;
      device: &lt;string&gt;
      serial: &lt;string&gt;
- local.label:
    &lt;label-name&gt;: &lt;string&gt;

## Examples

The following example below demonstrates the new capabilities:

```yaml
sources:
  custom:
  - name: "my cpu test rule"
    labels:
      cpu-test-feature: "true"
    matchFeatures:
    - feature: cpu.cpuid
      matchExpressions: ["AVX", "AESNI"]
    - feature: cpu.pstate
      matchExpressions: ["turbo=false"]
    - feature: cpu.rdt
      matchExpressions:
        RDTL3CA: {op: Exists}
    - feature: cpu.power
      matchExpressions:
        sst_bf.enabled: true
    - feature: cpu.topology
      matchExpressions:
        hardwareMultithreading: {op: IsTrue}

  - name: "my kernel test rule"
    labels:
      kernel-test-feature: "true"
    matchFeatures:
    - feature: kernel.loadedmodule
      matchExpressions: ["sbs"]
    - feature: kernel.config
      matchExpressions:
        X86:
        LOG_BUF_SHIFT: {op: Gt, value: ["17"]}
    - feature: kernel.version
      matchExpressions:
        major: {op: Gt, value: ["3"]}
        minor: {op: Gt, value: ["18"]}
    - feature: kernel.selinux
      matchExpressions:
        enabled: true

  - name: "my system test rule"
    labels:
      system-test-feature: "true"
    matchFeatures:
    - feature: system.osrelease
      matchExpressions:
        NAME: {op: InRegexp, value: ["^openSuse.*"]}
        VERSION_ID.major: {op: GtLt, value: ["13", "15"]}
    - feature: system.name
      matchExpressions:
        nodename: {op: InRegexp, value: ["xyz-.*-prod", "rack[0-9]-"]}

  - name: "my local test rule"
    labels:
      local-test-feature: "true"
    matchFeatures:
    - feature: local.label
      matchExpressions:
        example.com/local-feature: {op: Lt, value: ["200"]}
```

The example above also demonstrates some "shortcuts / shortforms" that are wired in the rule parsing

Example 1:

```yaml
cpu.cpuid: ["AVX", "AESNI"]
```
is the same as 

```yaml
cpu.cpuid:
  "AVX": {op: Exists}
  "AESNI": {op: Exists}
```

Example 2:
```yaml
kernel.config: ["X86", "INIT_ENV_ARG_LIMIT=32"]
```

is the same as
```yaml
kernel.config:
  X86:
  INIT_ENV_ARG_LIMIT: {op: In, value: ["32"]}
```

is the same as
```yaml
kernel.config:
  X86: {op: Exists}
  INIT_ENV_ARG_LIMIT: {op: In, value: ["32"]}
```

Example 3:
```yaml
cpu.power
  sst_bf.enabled: true
```

is the same as
```yaml
cpu.power
  sst_bf.enabled: {op: In, value["true"]}
```

## matchAny

In addition to `matchFeatures`, the PR also implements a new 'matchAny' directive. MatchAny applies a logical OR over multiple matchFeatures matchers. That is, it allows specifying multiple alternative matchers (at least one of which must match) in a single label rule.

The configuration format for the new matcher is

```yaml
  matchAny:
    - matchFeatures:
        - feature: <domain>.<feature>
          matchExpressions:
            <attribute>:
              op: <operation>
              value:
                - <list-of-values>
    - matchFeatures:
      ...
```

A configuration example. In order to require a cpu feature, kernel
module and one of two specific PCI devices (taking use of the shortform
notation):

```yaml
  - name: multi-device-test
    labels: {multi-device-feature: "true"}
    matchFeatures:
      - feature: kernel.loadedmodule
        matchExpressions: [driver-module]
      - feature: cpu.cpuid
        matchExpressions: [AVX512F]
    matchAny:
      - matchFeatures:
          - feature; pci.device
            matchExpressions:
              vendor: "8086"
              device: "1234"
      - matchFeatures:
          - feature: pci.device
            matchExpressions:
              vendor: "8086"
              device: "abcd"
```

## Notes

Documentation is out of the scope of this PR. Documentation will require major changes and will be addressed in a separate PR, also covering #550 and #553.

k8s-ci-robot: Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with `/test all`
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/node-feature-discovery/pull/464#" title="Author self-approved">marquiz</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fnode-feature-discovery).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/node-feature-discovery/blob/master/OWNERS)~~ [marquiz]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
marquiz: Even if in the proto stage, any feedback on the concept, data model, configuration model etc would be more than welcome 😉 
marquiz: Fixed unit tests and added `IsTrue` and `IsFalse` ops
k8s-ci-robot: @marquiz: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
k8s-ci-robot: @marquiz: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
ArangoGutierrez: /assign
k8s-ci-robot: @marquiz: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
k8s-ci-robot: @marquiz: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
marquiz: Major overhaul including a rebase. Now based on top of #541 and #543
k8s-ci-robot: @marquiz: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
k8s-ci-robot: @marquiz: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
k8s-ci-robot: @marquiz: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
k8s-ci-robot: @marquiz: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
k8s-ci-robot: @marquiz: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
k8s-ci-robot: @marquiz: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-node-feature-discovery-verify | e4c1894c1f61d1b62d498131d480b6b7a1635400 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_node-feature-discovery/464/pull-node-feature-discovery-verify/1432949138181328896) | `/test pull-node-feature-discovery-verify`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=node-feature-discovery&pr=464). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Amarquiz). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/node-feature-discovery/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
ArangoGutierrez: I think this PR is ready
My concern is opened PR's from #333 would need a major rebase after merge.

Thoughts@marquiz 
k8s-ci-robot: @marquiz: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-node-feature-discovery-verify | d571adf870c001837a9db9acd7e44f6b41110abd | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_node-feature-discovery/464/pull-node-feature-discovery-verify/1438819519249256448) | `/test pull-node-feature-discovery-verify`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=node-feature-discovery&pr=464). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Amarquiz). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/node-feature-discovery/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @marquiz: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
marquiz: > Thoughts @marquiz
> maybe unhold?

Looks like doing this (and the other related PRs, too) in smaller pieces is a better approach. Easier to review and also the CI seems to spot some mistakes in the individual commits. Next up is #604 ...
marquiz: Started to think the new rule format once more, trying to make it more comprehensible and better comply with k8s api design guidelines. Changes:
1. Rename 'matchAll' to 'matchFeatures' and make it a single "matcher" instead of a list of matchers (a list of matchers is a bit confusing here and sort of redundant as all of the sub-matchers need to match anyway)
2. Change the matcher to be a list of objects (named fields `feature` and `matchExpressions`) instead of a map. The goal would be to improve readability and follow k8s api conventions.
3. Change `matchAny` to be a list of objects (with only one field, `matchFeatures`, currently) instead of a list of maps. This better complies with k8s api guidelines and also makes matchAny extensible in the future (if other directives than `matchFeatures` would be added). This also makes `matchAny` an extension/improvement that can be left out from the first implementation (in the sake of simplicity)

An example below: instead of this (the OLD format):

```
sources:
  custom:
    - name: "my-feature-1"
      matchAll:
        - cpu.cpuid:
            AVX512F: {op: Exists}
          kernel.loadedModule:
            "sbs": {op: Exists}
          kernel.version:
            major: {op: In, value: ["5"]}
            minor: {op: Gt, value: ["10"]}
    - name: "my-feature-2"
      matchAny:
        - kernel.loadedmodule:
            "my-driver": {op: Exists}
          pci.device:
            vendor: {op: In, value: ["8086"]}
            device: {op: In, value: ["1234", "5678"]}
        - kernel.loadedmodule:
            "my-driver": {op: Exists}
          pci.device:
            vendor: {op: In, value: ["abcd"]}
            device: {op: In, value: ["9876"]}
```

We'd have this (NEW format):

```
sources:
  custom:
    - name: "my-feature-1"
      matchFeatures:
        - feature: cpu.cpuid
          matchExpressions:
            AVX512F: {op: Exists}
        - feature: kernel.loadedmodule
          matchExpressions:
            "sbs": {op: Exists}
        - feature: kernel.version
          matchExpressions:
            major: {op: In, value: ["5"]}
            minor: {op: Gt, value: ["10"]}
    - name: "my-feature-2"
      matchAny:
        - matchFeatures:
            - feature: kernel.loadedmodule
              matchExpressions:
                "my-driver": {op: Exists}
            - feature: pci.device
              matchExpressions:
                vendor: {op: In, value: ["8086"]}
                device: {op: In, value: ["1234", "5678"]}
        - matchFeatures:
            - feature: kernel.loadedmodule
              matchExpressions:
                "my-driver": {op: Exists}
            - feature: pci.device
              matchExpressions:
                vendor: {op: In, value: ["abcd"]}
                device: {op: In, value: ["9876"]}
```

Any thoughts/comments? If not, then I'll probably just make the change (I have the code ready)😸 
k8s-ci-robot: @marquiz: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-node-feature-discovery-verify | 1f7056c80da5664ae4457fcd210a8fb8a4780cd0 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_node-feature-discovery/464/pull-node-feature-discovery-verify/1458072698268684288) | true | `/test pull-node-feature-discovery-verify`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=node-feature-discovery&pr=464). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Amarquiz). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/node-feature-discovery/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @marquiz: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-node-feature-discovery-verify | f1c248a82cd5c4a15f95d7b733e84d3fe024efa5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_node-feature-discovery/464/pull-node-feature-discovery-verify/1458074295367372800) | true | `/test pull-node-feature-discovery-verify`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=node-feature-discovery&pr=464). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Amarquiz). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/node-feature-discovery/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
ArangoGutierrez: /test pull-node-feature-discovery-verify
k8s-ci-robot: @marquiz: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-node-feature-discovery-verify | f1c248a82cd5c4a15f95d7b733e84d3fe024efa5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_node-feature-discovery/464/pull-node-feature-discovery-verify/1458443249164226560) | true | `/test pull-node-feature-discovery-verify`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=node-feature-discovery&pr=464). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Amarquiz). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/node-feature-discovery/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
ArangoGutierrez: > @marquiz: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:
> Test name 	Commit 	Details 	Required 	Rerun command
> pull-node-feature-discovery-verify 	[f1c248a](https://github.com/kubernetes-sigs/node-feature-discovery/commit/f1c248a82cd5c4a15f95d7b733e84d3fe024efa5) 	[link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_node-feature-discovery/464/pull-node-feature-discovery-verify/1458443249164226560) 	true 	`/test pull-node-feature-discovery-verify`
> 
> [Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=node-feature-discovery&pr=464). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Amarquiz). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/node-feature-discovery/issues?q=is:issue+is:open) when you hit one in your PR.

```bash
 golangci-lint run --timeout 7m0s
source/custom/custom_test.go:45:2: ineffectual assignment to m (ineffassign)
	m, err = r2.execute(f)
	^
source/custom/custom_test.go:56:2: ineffectual assignment to m (ineffassign)
	m, err = r2.execute(f)
	^
make: *** [Makefile:117: ci-lint] Error 1 
```
marquiz: @ArangoGutierrez I fixed that but forgot to push. Now everything's green 😄 
ArangoGutierrez: /test all
ArangoGutierrez: @marquiz could you rebase this PR on top of yesterday test fixes for Minikube? 
ArangoGutierrez: After a manual test found 
```bash
I1111 16:11:09.590850       1 nfd-worker.go:149] Node Feature Discovery Worker 464
I1111 16:11:09.590898       1 nfd-worker.go:150] NodeName: 'minikube'
I1111 16:11:09.591035       1 nfd-worker.go:360] configuration file "/etc/kubernetes/node-feature-discovery/nfd-worker.conf" parsed
I1111 16:11:09.591075       1 nfd-worker.go:395] worker (re-)configuration successfully completed
I1111 16:11:09.591109       1 base.go:126] connecting to nfd-master at nfd-master:8080 ...
I1111 16:11:09.591130       1 component.go:36] [core]parsed scheme: ""
I1111 16:11:09.591134       1 component.go:36] [core]scheme "" not registered, fallback to default scheme
I1111 16:11:09.591151       1 component.go:36] [core]ccResolverWrapper: sending update to cc: {[{nfd-master:8080  <nil> 0 <nil>}] <nil> <nil>}
I1111 16:11:09.591156       1 component.go:36] [core]ClientConn switching balancer to "pick_first"
I1111 16:11:09.591159       1 component.go:36] [core]Channel switches to new LB policy "pick_first"
I1111 16:11:09.591173       1 component.go:36] [core]Subchannel Connectivity change to CONNECTING
I1111 16:11:09.591217       1 component.go:36] [core]Subchannel picks a new address "nfd-master:8080" to connect
I1111 16:11:09.591332       1 component.go:36] [core]Channel Connectivity change to CONNECTING
I1111 16:11:09.592040       1 component.go:36] [core]Subchannel Connectivity change to READY
I1111 16:11:09.592051       1 component.go:36] [core]Channel Connectivity change to READY
W1111 16:11:09.595222       1 kernel.go:140] failed to detect the status of selinux: open /host-sys/fs/selinux/enforce: no such file or directory
I1111 16:11:09.596929       1 nfd-worker.go:406] starting feature discovery...
I1111 16:11:09.597002       1 base.go:139] closing connection to nfd-master ...
I1111 16:11:09.597019       1 component.go:36] [core]Channel Connectivity change to SHUTDOWN
I1111 16:11:09.597030       1 component.go:36] [core]Subchannel Connectivity change to SHUTDOWN
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xc402e9]

goroutine 1 [running]:
sigs.k8s.io/node-feature-discovery/source/kernel.(*kernelSource).GetLabels(0x1584280)
        /go/node-feature-discovery/source/kernel/kernel.go:108 +0x269
sigs.k8s.io/node-feature-discovery/pkg/nfd-client/worker.getFeatureLabels({0x7f3e359b9b50, 0x1584280}, {{0x0, 0x0}, 0xc00024cc90, 0x0, 0x0, 0x15555, {0xc0001e2fa0, 0x1, ...}, ...})
        /go/node-feature-discovery/pkg/nfd-client/worker/nfd-worker.go:427 +0xb0
sigs.k8s.io/node-feature-discovery/pkg/nfd-client/worker.createFeatureLabels({0xc0000db4a0, 0xb, 0x203000}, {{0x0, 0x0}, 0xc00024cc90, 0x0, 0x0, 0x15555, {0xc0001e2fa0, ...}, ...})
        /go/node-feature-discovery/pkg/nfd-client/worker/nfd-worker.go:408 +0x1d9
sigs.k8s.io/node-feature-discovery/pkg/nfd-client/worker.(*nfdWorker).Run(0xc000260160)
        /go/node-feature-discovery/pkg/nfd-client/worker/nfd-worker.go:187 +0xbb8
main.main()
        /go/node-feature-discovery/cmd/nfd-worker/main.go:63 +0x371
```

the `nfd-worker` enters a `crashbackloop` state after this
zvonkok: @marquiz The same error from PR #604 
marquiz: Rebased, one comment (copyright year) addressed

> @marquiz The same error from PR #604

Yeah, should be now fixed in #604 (and this one, too)
marquiz: So still #604 and #639 to go before this one.
marquiz: #604 was merged so we're one step away from this

Patch:
diff --git a/deployment/components/worker-config/nfd-worker.conf.example b/deployment/components/worker-config/nfd-worker.conf.example
index 37466c40d..c37860736 100644
--- a/deployment/components/worker-config/nfd-worker.conf.example
+++ b/deployment/components/worker-config/nfd-worker.conf.example
@@ -107,3 +107,80 @@
 #      value: customValue
 #      matchOn:
 #        - nodename: ["worker-0", "my-.*-node"]
+#
+#    # The following feature demonstrates the capabilities of the matchFeatures
+#    - name: "my.ng.feature"
+#      labels:
+#        my-ng-feature: "true"
+#      # matchFeatures implements a logical AND over all matcher terms in the
+#      # list (i.e. all of the terms, or per-feature matchers, must match)
+#      matchFeatures:
+#        - feature: cpu.cpuid
+#          matchExpressions:
+#            AVX512F: {op: Exists}
+#        - feature: cpu.cstate
+#          matchExpressions:
+#            enabled: {op: IsTrue}
+#        - feature: cpu.pstate
+#          matchExpressions:
+#            no_turbo: {op: IsFalse}
+#            scaling_governor: {op: In, value: ["performance"]}
+#        - feature: cpu.rdt
+#          matchExpressions:
+#            RDTL3CA: {op: Exists}
+#        - feature: cpu.sst
+#          matchExpressions:
+#            bf.enabled: {op: IsTrue}
+#        - feature: cpu.topology
+#          matchExpressions:
+#            hardware_multithreading: {op: IsFalse}
+#
+#        - feature: kernel.config
+#          matchExpressions:
+#            X86: {op: Exists}
+#            LSM: {op: InRegexp, value: ["apparmor"]}
+#        - feature: kernel.loadedmodule
+#          matchExpressions:
+#            e1000e: {op: Exists}
+#        - feature: kernel.selinux
+#          matchExpressions:
+#            enabled: {op: IsFalse}
+#        - feature: kernel.version
+#          matchExpressions:
+#            major: {op: In, value: ["5"]}
+#            minor: {op: Gt, value: ["10"]}
+#
+#        - feature: system.osrelease
+#          matchExpressions:
+#            ID: {op: In, value: ["fedora", "centos"]}
+#        - feature: system.name
+#          matchExpressions:
+#            nodename: {op: InRegexp, value: ["^worker-X"]}
+#
+#        - feature: local.label
+#          matchExpressions:
+#            custom-feature-knob: {op: Gt, value: ["100"]}
+#
+#    # The following feature demonstrates the capabilities of the matchAny
+#    - name: "my.ng.feature.2"
+#      labels:
+#        my-ng-feature-2: "my-value"
+#      # matchAny implements a logical IF over all elements (sub-matchers) in
+#      # the list (i.e. at least one feature matcher must match)
+#      matchAny:
+#        - matchFeatures:
+#            - feature: kernel.loadedmodule
+#              matchExpressions:
+#                driver-module-X: {op: Exists}
+#            - feature: pci.device
+#              matchExpressions:
+#                vendor: {op: In, value: ["8086"]}
+#                class: {op: In, value: ["0200"]}
+#        - matchFeatures:
+#            - feature: kernel.loadedmodule
+#              matchExpressions:
+#                driver-module-Y: {op: Exists}
+#            - feature: usb.device
+#              matchExpressions:
+#                vendor: {op: In, value: ["8086"]}
+#                class: {op: In, value: ["02"]}
diff --git a/deployment/helm/node-feature-discovery/values.yaml b/deployment/helm/node-feature-discovery/values.yaml
index 349afdcb2..5bb5b2150 100644
--- a/deployment/helm/node-feature-discovery/values.yaml
+++ b/deployment/helm/node-feature-discovery/values.yaml
@@ -193,6 +193,83 @@ worker:
     #      value: customValue
     #      matchOn:
     #        - nodename: ["worker-0", "my-.*-node"]
+    #
+    #    # The following feature demonstrates the capabilities of the matchFeatures
+    #    - name: "my.ng.feature"
+    #      labels:
+    #        my-ng-feature: "true"
+    #      # matchFeatures implements a logical AND over all matcher terms in the
+    #      # list (i.e. all of the terms, or per-feature matchers, must match)
+    #      matchFeatures:
+    #        - feature: cpu.cpuid
+    #          matchExpressions:
+    #            AVX512F: {op: Exists}
+    #        - feature: cpu.cstate
+    #          matchExpressions:
+    #            enabled: {op: IsTrue}
+    #        - feature: cpu.pstate
+    #          matchExpressions:
+    #            no_turbo: {op: IsFalse}
+    #            scaling_governor: {op: In, value: ["performance"]}
+    #        - feature: cpu.rdt
+    #          matchExpressions:
+    #            RDTL3CA: {op: Exists}
+    #        - feature: cpu.sst
+    #          matchExpressions:
+    #            bf.enabled: {op: IsTrue}
+    #        - feature: cpu.topology
+    #          matchExpressions:
+    #            hardware_multithreading: {op: IsFalse}
+    #
+    #        - feature: kernel.config
+    #          matchExpressions:
+    #            X86: {op: Exists}
+    #            LSM: {op: InRegexp, value: ["apparmor"]}
+    #        - feature: kernel.loadedmodule
+    #          matchExpressions:
+    #            e1000e: {op: Exists}
+    #        - feature: kernel.selinux
+    #          matchExpressions:
+    #            enabled: {op: IsFalse}
+    #        - feature: kernel.version
+    #          matchExpressions:
+    #            major: {op: In, value: ["5"]}
+    #            minor: {op: Gt, value: ["10"]}
+    #
+    #        - feature: system.osrelease
+    #          matchExpressions:
+    #            ID: {op: In, value: ["fedora", "centos"]}
+    #        - feature: system.name
+    #          matchExpressions:
+    #            nodename: {op: InRegexp, value: ["^worker-X"]}
+    #
+    #        - feature: local.label
+    #          matchExpressions:
+    #            custom-feature-knob: {op: Gt, value: ["100"]}
+    #
+    #    # The following feature demonstrates the capabilities of the matchAny
+    #    - name: "my.ng.feature.2"
+    #      labels:
+    #        my-ng-feature-2: "my-value"
+    #      # matchAny implements a logical IF over all elements (sub-matchers) in
+    #      # the list (i.e. at least one feature matcher must match)
+    #      matchAny:
+    #        - matchFeatures:
+    #            - feature: kernel.loadedmodule
+    #              matchExpressions:
+    #                driver-module-X: {op: Exists}
+    #            - feature: pci.device
+    #              matchExpressions:
+    #                vendor: {op: In, value: ["8086"]}
+    #                class: {op: In, value: ["0200"]}
+    #        - matchFeatures:
+    #            - feature: kernel.loadedmodule
+    #              matchExpressions:
+    #                driver-module-Y: {op: Exists}
+    #            - feature: usb.device
+    #              matchExpressions:
+    #                vendor: {op: In, value: ["8086"]}
+    #                class: {op: In, value: ["02"]}
 ### <NFD-WORKER-CONF-END-DO-NOT-REMOVE>
 
   podSecurityContext: {}
diff --git a/source/custom/custom.go b/source/custom/custom.go
index 427f4a5cb..4ccc5f85a 100644
--- a/source/custom/custom.go
+++ b/source/custom/custom.go
@@ -17,19 +17,25 @@ limitations under the License.
 package custom
 
 import (
+	"encoding/json"
+	"fmt"
 	"reflect"
+	"strings"
 
 	"k8s.io/klog/v2"
+	"sigs.k8s.io/yaml"
 
+	"sigs.k8s.io/node-feature-discovery/pkg/api/feature"
 	"sigs.k8s.io/node-feature-discovery/pkg/utils"
 	"sigs.k8s.io/node-feature-discovery/source"
+	"sigs.k8s.io/node-feature-discovery/source/custom/expression"
 	"sigs.k8s.io/node-feature-discovery/source/custom/rules"
 )
 
 const Name = "custom"
 
-// Custom Features Configurations
-type MatchRule struct {
+// LegacyMatcher contains the legacy custom rules.
+type LegacyMatcher struct {
 	PciID      *rules.PciIDRule      `json:"pciId,omitempty"`
 	UsbID      *rules.UsbIDRule      `json:"usbId,omitempty"`
 	LoadedKMod *rules.LoadedKModRule `json:"loadedKMod,omitempty"`
@@ -38,13 +44,36 @@ type MatchRule struct {
 	Nodename   *rules.NodenameRule   `json:"nodename,omitempty"`
 }
 
-type FeatureSpec struct {
-	Name    string      `json:"name"`
-	Value   *string     `json:"value,omitempty"`
-	MatchOn []MatchRule `json:"matchOn"`
+type LegacyRule struct {
+	Name    string          `json:"name"`
+	Value   *string         `json:"value,omitempty"`
+	MatchOn []LegacyMatcher `json:"matchOn"`
 }
 
-type config []FeatureSpec
+type Rule struct {
+	Name          string            `json:"name"`
+	Labels        map[string]string `json:"labels"`
+	MatchFeatures FeatureMatcher    `json:"matchFeatures"`
+	MatchAny      []MatchAnyElem    `json:"matchAny"`
+}
+
+type MatchAnyElem struct {
+	MatchFeatures FeatureMatcher
+}
+
+type FeatureMatcher []FeatureMatcherTerm
+
+type FeatureMatcherTerm struct {
+	Feature          string
+	MatchExpressions expression.MatchExpressionSet
+}
+
+type config []CustomRule
+
+type CustomRule struct {
+	*LegacyRule
+	*Rule
+}
 
 // newDefaultConfig returns a new config with pre-populated defaults
 func newDefaultConfig() *config {
@@ -91,64 +120,201 @@ func (s *customSource) Priority() int { return 10 }
 
 // GetLabels method of the LabelSource interface
 func (s *customSource) GetLabels() (source.FeatureLabels, error) {
-	features := source.FeatureLabels{}
+	// Get raw features from all sources
+	domainFeatures := make(map[string]*feature.DomainFeatures)
+	for n, s := range source.GetAllFeatureSources() {
+		domainFeatures[n] = s.GetFeatures()
+	}
+
+	labels := source.FeatureLabels{}
 	allFeatureConfig := append(getStaticFeatureConfig(), *s.config...)
 	allFeatureConfig = append(allFeatureConfig, getDirectoryFeatureConfig()...)
 	utils.KlogDump(2, "custom features configuration:", "  ", allFeatureConfig)
 	// Iterate over features
-	for _, customFeature := range allFeatureConfig {
-		featureExist, err := s.discoverFeature(customFeature)
+	for _, rule := range allFeatureConfig {
+		ruleOut, err := rule.execute(domainFeatures)
 		if err != nil {
-			klog.Errorf("failed to discover feature: %q: %s", customFeature.Name, err.Error())
+			klog.Error(err)
 			continue
 		}
-		if featureExist {
-			var value interface{} = true
-			if customFeature.Value != nil {
-				value = *customFeature.Value
-			}
-			features[customFeature.Name] = value
+
+		for n, v := range ruleOut {
+			labels[n] = v
 		}
 	}
-	return features, nil
+	return labels, nil
 }
 
-// Process a single feature by Matching on the defined rules.
-// A feature is present if all defined Rules in a MatchRule return a match.
-func (s *customSource) discoverFeature(feature FeatureSpec) (bool, error) {
-	for _, matchRules := range feature.MatchOn {
+func (r *CustomRule) execute(features map[string]*feature.DomainFeatures) (map[string]string, error) {
+	if r.LegacyRule != nil {
+		ruleOut, err := r.LegacyRule.execute(features)
+		if err != nil {
+			return nil, fmt.Errorf("failed to execute legacy rule %s: %w", r.LegacyRule.Name, err)
+		}
+		return ruleOut, err
+	}
 
-		allRules := []legacyRule{
-			matchRules.PciID,
-			matchRules.UsbID,
-			matchRules.LoadedKMod,
-			matchRules.CpuID,
-			matchRules.Kconfig,
-			matchRules.Nodename,
+	if r.Rule != nil {
+		ruleOut, err := r.Rule.execute(features)
+		if err != nil {
+			return nil, fmt.Errorf("failed to execute rule %s: %w", r.Rule.Name, err)
 		}
+		return ruleOut, err
+	}
+
+	return nil, fmt.Errorf("BUG: an empty rule, this really should not happen")
+}
+
+func (r *LegacyRule) execute(features map[string]*feature.DomainFeatures) (map[string]string, error) {
+	if len(r.MatchOn) > 0 {
+		// Logical OR over the legacy rules
+		matched := false
+		for _, matcher := range r.MatchOn {
+			if m, err := matcher.match(); err != nil {
+				return nil, err
+			} else if m {
+				matched = true
+				break
+			}
+		}
+		if !matched {
+			return nil, nil
+		}
+	}
 
-		// return true, nil if all rules match
-		matchRules := func(rules []legacyRule) (bool, error) {
-			for _, rule := range rules {
-				if reflect.ValueOf(rule).IsNil() {
-					continue
-				}
-				if match, err := rule.Match(); err != nil {
-					return false, err
-				} else if !match {
-					return false, nil
-				}
+	value := "true"
+	if r.Value != nil {
+		value = *r.Value
+	}
+	return map[string]string{r.Name: value}, nil
+}
+
+func (r *Rule) execute(features map[string]*feature.DomainFeatures) (map[string]string, error) {
+	if len(r.MatchAny) > 0 {
+		// Logical OR over the matchAny matchers
+		matched := false
+		for _, matcher := range r.MatchAny {
+			if m, err := matcher.match(features); err != nil {
+				return nil, err
+			} else if m {
+				matched = true
+				break
 			}
-			return true, nil
 		}
+		if !matched {
+			return nil, nil
+		}
+	}
+
+	if len(r.MatchFeatures) > 0 {
+		if m, err := r.MatchFeatures.match(features); err != nil {
+			return nil, err
+		} else if !m {
+			return nil, nil
+		}
+	}
 
-		if match, err := matchRules(allRules); err != nil {
+	labels := make(map[string]string, len(r.Labels))
+	for k, v := range r.Labels {
+		labels[k] = v
+	}
+
+	return labels, nil
+}
+
+func (e *MatchAnyElem) match(features map[string]*feature.DomainFeatures) (bool, error) {
+	return e.MatchFeatures.match(features)
+}
+
+func (m *FeatureMatcher) match(features map[string]*feature.DomainFeatures) (bool, error) {
+	// Logical AND over the terms
+	for _, term := range *m {
+		split := strings.SplitN(term.Feature, ".", 2)
+		if len(split) != 2 {
+			return false, fmt.Errorf("invalid selector %q: must be <domain>.<feature>", term.Feature)
+		}
+		domain := split[0]
+		// Ignore case
+		featureName := strings.ToLower(split[1])
+
+		domainFeatures, ok := features[domain]
+		if !ok {
+			return false, fmt.Errorf("unknown feature source/domain %q", domain)
+		}
+
+		var m bool
+		var err error
+		if f, ok := domainFeatures.Keys[featureName]; ok {
+			m, err = term.MatchExpressions.MatchKeys(f.Elements)
+		} else if f, ok := domainFeatures.Values[featureName]; ok {
+			m, err = term.MatchExpressions.MatchValues(f.Elements)
+		} else if f, ok := domainFeatures.Instances[featureName]; ok {
+			m, err = term.MatchExpressions.MatchInstances(f.Elements)
+		} else {
+			return false, fmt.Errorf("%q feature of source/domain %q not available", featureName, domain)
+		}
+
+		if err != nil {
 			return false, err
-		} else if match {
-			return true, nil
+		} else if !m {
+			return false, nil
+		}
+	}
+	return true, nil
+}
+
+func (m *LegacyMatcher) match() (bool, error) {
+	allRules := []legacyRule{
+		m.PciID,
+		m.UsbID,
+		m.LoadedKMod,
+		m.CpuID,
+		m.Kconfig,
+		m.Nodename,
+	}
+
+	// return true, nil if all rules match
+	matchRules := func(rules []legacyRule) (bool, error) {
+		for _, rule := range rules {
+			if reflect.ValueOf(rule).IsNil() {
+				continue
+			}
+			if match, err := rule.Match(); err != nil {
+				return false, err
+			} else if !match {
+				return false, nil
+			}
+		}
+		return true, nil
+	}
+
+	return matchRules(allRules)
+}
+
+// UnmarshalJSON implements the Unmarshaler interface from "encoding/json"
+func (c *CustomRule) UnmarshalJSON(data []byte) error {
+	// Do a raw parse to determine if this is a legacy rule
+	raw := map[string]json.RawMessage{}
+	err := yaml.Unmarshal(data, &raw)
+	if err != nil {
+		return err
+	}
+
+	for k := range raw {
+		if strings.ToLower(k) == "matchon" {
+			return yaml.Unmarshal(data, &c.LegacyRule)
 		}
 	}
-	return false, nil
+
+	return yaml.Unmarshal(data, &c.Rule)
+}
+
+// MarshalJSON implements the Marshaler interface from "encoding/json"
+func (c *CustomRule) MarshalJSON() ([]byte, error) {
+	if c.LegacyRule != nil {
+		return json.Marshal(c.LegacyRule)
+	}
+	return json.Marshal(c.Rule)
 }
 
 func init() {
diff --git a/source/custom/directory_features.go b/source/custom/directory_features.go
index d2a725ac6..2df415e3a 100644
--- a/source/custom/directory_features.go
+++ b/source/custom/directory_features.go
@@ -31,14 +31,14 @@ const Directory = "/etc/kubernetes/node-feature-discovery/custom.d"
 
 // getDirectoryFeatureConfig returns features configured in the "/etc/kubernetes/node-feature-discovery/custom.d"
 // host directory and its 1st level subdirectories, which can be populated e.g. by ConfigMaps
-func getDirectoryFeatureConfig() []FeatureSpec {
+func getDirectoryFeatureConfig() []CustomRule {
 	features := readDir(Directory, true)
 	klog.V(1).Infof("all configmap based custom feature specs: %+v", features)
 	return features
 }
 
-func readDir(dirName string, recursive bool) []FeatureSpec {
-	features := make([]FeatureSpec, 0)
+func readDir(dirName string, recursive bool) []CustomRule {
+	features := make([]CustomRule, 0)
 
 	klog.V(1).Infof("getting files in %s", dirName)
 	files, err := ioutil.ReadDir(dirName)
@@ -76,7 +76,7 @@ func readDir(dirName string, recursive bool) []FeatureSpec {
 		}
 		klog.V(2).Infof("custom config rules raw: %s", string(bytes))
 
-		config := &[]FeatureSpec{}
+		config := &[]CustomRule{}
 		err = yaml.UnmarshalStrict(bytes, config)
 		if err != nil {
 			klog.Errorf("could not parse custom config file %q, %v", fileName, err)
diff --git a/source/custom/expression/expression.go b/source/custom/expression/expression.go
index 6ff961f03..f5595306b 100644
--- a/source/custom/expression/expression.go
+++ b/source/custom/expression/expression.go
@@ -20,6 +20,7 @@ import (
 	"encoding/json"
 	"fmt"
 	"regexp"
+	"sort"
 	"strconv"
 	"strings"
 
@@ -269,26 +270,66 @@ func (m *MatchExpression) Match(valid bool, value interface{}) (bool, error) {
 
 // MatchKeys evaluates the MatchExpression against a set of keys.
 func (m *MatchExpression) MatchKeys(name string, keys map[string]feature.Nil) (bool, error) {
-	klog.V(3).Infof("matching %q %q against %v", name, m.Op, keys)
+	matched := false
 
 	_, ok := keys[name]
 	switch m.Op {
 	case MatchAny:
-		return true, nil
+		matched = true
 	case MatchExists:
-		return ok, nil
+		matched = ok
 	case MatchDoesNotExist:
-		return !ok, nil
+		matched = !ok
 	default:
 		return false, fmt.Errorf("invalid Op %q when matching keys", m.Op)
 	}
+
+	if klog.V(3).Enabled() {
+		mString := map[bool]string{false: "no match", true: "match found"}[matched]
+		k := make([]string, 0, len(keys))
+		for n := range keys {
+			k = append(k, n)
+		}
+		sort.Strings(k)
+		if len(keys) < 10 || klog.V(4).Enabled() {
+			klog.Infof("%s when matching %q %q against %s", mString, name, m.Op, strings.Join(k, " "))
+		} else {
+			klog.Infof("%s when matching %q %q against %s... (list truncated)", mString, name, m.Op, strings.Join(k[0:10], ", "))
+		}
+	}
+	return matched, nil
 }
 
 // MatchValues evaluates the MatchExpression against a set of key-value pairs.
 func (m *MatchExpression) MatchValues(name string, values map[string]string) (bool, error) {
-	klog.V(3).Infof("matching %q %q %v against %v", name, m.Op, m.Value, values)
 	v, ok := values[name]
-	return m.Match(ok, v)
+	matched, err := m.Match(ok, v)
+	if err != nil {
+		return false, err
+	}
+
+	if klog.V(3).Enabled() {
+		mString := map[bool]string{false: "no match", true: "match found"}[matched]
+
+		keys := make([]string, 0, len(values))
+		for k := range values {
+			keys = append(keys, k)
+		}
+		sort.Strings(keys)
+
+		kv := make([]string, len(keys))
+		for i, k := range keys {
+			kv[i] = k + ":" + values[k]
+		}
+
+		if len(values) < 10 || klog.V(4).Enabled() {
+			klog.Infof("%s when matching %q %q %v against %s", mString, name, m.Op, m.Value, strings.Join(kv, " "))
+		} else {
+			klog.Infof("%s when matching %q %q %v against %s... (list truncated)", mString, name, m.Op, m.Value, strings.Join(kv[0:10], " "))
+		}
+	}
+
+	return matched, nil
 }
 
 // matchExpression is a helper type for unmarshalling MatchExpression
diff --git a/source/custom/static_features.go b/source/custom/static_features.go
index 3d632e479..d0c4cd96e 100644
--- a/source/custom/static_features.go
+++ b/source/custom/static_features.go
@@ -23,28 +23,32 @@ import (
 
 // getStaticFeatures returns statically configured custom features to discover
 // e.g RMDA related features. NFD configuration file may extend these custom features by adding rules.
-func getStaticFeatureConfig() []FeatureSpec {
-	return []FeatureSpec{
+func getStaticFeatureConfig() []CustomRule {
+	return []CustomRule{
 		{
-			Name: "rdma.capable",
-			MatchOn: []MatchRule{
-				{
-					PciID: &rules.PciIDRule{
-						MatchExpressionSet: expression.MatchExpressionSet{
-							"vendor": expression.MustCreateMatchExpression(expression.MatchIn, "15b3"),
+			LegacyRule: &LegacyRule{
+				Name: "rdma.capable",
+				MatchOn: []LegacyMatcher{
+					{
+						PciID: &rules.PciIDRule{
+							MatchExpressionSet: expression.MatchExpressionSet{
+								"vendor": expression.MustCreateMatchExpression(expression.MatchIn, "15b3"),
+							},
 						},
 					},
 				},
 			},
 		},
 		{
-			Name: "rdma.available",
-			MatchOn: []MatchRule{
-				{
-					LoadedKMod: &rules.LoadedKModRule{
-						MatchExpressionSet: expression.MatchExpressionSet{
-							"ib_uverbs": expression.MustCreateMatchExpression(expression.MatchExists),
-							"rdma_ucm":  expression.MustCreateMatchExpression(expression.MatchExists),
+			LegacyRule: &LegacyRule{
+				Name: "rdma.available",
+				MatchOn: []LegacyMatcher{
+					{
+						LoadedKMod: &rules.LoadedKModRule{
+							MatchExpressionSet: expression.MatchExpressionSet{
+								"ib_uverbs": expression.MustCreateMatchExpression(expression.MatchExists),
+								"rdma_ucm":  expression.MustCreateMatchExpression(expression.MatchExists),
+							},
 						},
 					},
 				},


Test patch:
diff --git a/source/custom/custom_test.go b/source/custom/custom_test.go
new file mode 100644
index 000000000..968413260
--- /dev/null
+++ b/source/custom/custom_test.go
@@ -0,0 +1,181 @@
+/*
+Copyright 2021 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package custom
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"sigs.k8s.io/node-feature-discovery/pkg/api/feature"
+	"sigs.k8s.io/node-feature-discovery/source/custom/expression"
+)
+
+func TestRule(t *testing.T) {
+	f := map[string]*feature.DomainFeatures{}
+	r1 := Rule{Labels: map[string]string{"label-1": "", "label-2": "true"}}
+	r2 := Rule{
+		Labels: map[string]string{"label-1": "label-val-1"},
+		MatchFeatures: FeatureMatcher{
+			FeatureMatcherTerm{
+				Feature:          "domain-1.kf-1",
+				MatchExpressions: expression.MatchExpressionSet{"key-1": expression.MustCreateMatchExpression(expression.MatchExists)},
+			},
+		},
+	}
+
+	// Test totally empty features
+	m, err := r1.execute(f)
+	assert.Nilf(t, err, "unexpected error: %v", err)
+	assert.Equal(t, r1.Labels, m, "empty matcher should have matched empty features")
+
+	_, err = r2.execute(f)
+	assert.Error(t, err, "matching agains a missing domain should have returned an error")
+
+	// Test empty domain
+	d := feature.NewDomainFeatures()
+	f["domain-1"] = d
+
+	m, err = r1.execute(f)
+	assert.Nilf(t, err, "unexpected error: %v", err)
+	assert.Equal(t, r1.Labels, m, "empty matcher should have matched empty features")
+
+	_, err = r2.execute(f)
+	assert.Error(t, err, "matching agains a missing feature type should have returned an error")
+
+	// Test empty feature sets
+	d.Keys["kf-1"] = feature.NewKeyFeatures()
+	d.Values["vf-1"] = feature.NewValueFeatures(nil)
+	d.Instances["if-1"] = feature.NewInstanceFeatures(nil)
+
+	m, err = r1.execute(f)
+	assert.Nilf(t, err, "unexpected error: %v", err)
+	assert.Equal(t, r1.Labels, m, "empty matcher should have matched empty features")
+
+	m, err = r2.execute(f)
+	assert.Nilf(t, err, "unexpected error: %v", err)
+	assert.Nil(t, m, "unexpected match")
+
+	// Test non-empty feature sets
+	d.Keys["kf-1"].Elements["key-x"] = feature.Nil{}
+	d.Values["vf-1"].Elements["key-1"] = "val-x"
+	d.Instances["if-1"] = feature.NewInstanceFeatures([]feature.InstanceFeature{
+		*feature.NewInstanceFeature(map[string]string{"attr-1": "val-x"})})
+
+	m, err = r1.execute(f)
+	assert.Nilf(t, err, "unexpected error: %v", err)
+	assert.Equal(t, r1.Labels, m, "empty matcher should have matched empty features")
+
+	// Match "key" features
+	m, err = r2.execute(f)
+	assert.Nilf(t, err, "unexpected error: %v", err)
+	assert.Nil(t, m, "keys should not have matched")
+
+	d.Keys["kf-1"].Elements["key-1"] = feature.Nil{}
+	m, err = r2.execute(f)
+	assert.Nilf(t, err, "unexpected error: %v", err)
+	assert.Equal(t, r2.Labels, m, "keys should have matched")
+
+	// Match "value" features
+	r3 := Rule{
+		Labels: map[string]string{"label-3": "label-val-3", "empty": ""},
+		MatchFeatures: FeatureMatcher{
+			FeatureMatcherTerm{
+				Feature:          "domain-1.vf-1",
+				MatchExpressions: expression.MatchExpressionSet{"key-1": expression.MustCreateMatchExpression(expression.MatchIn, "val-1")},
+			},
+		},
+	}
+	m, err = r3.execute(f)
+	assert.Nilf(t, err, "unexpected error: %v", err)
+	assert.Nil(t, m, "values should not have matched")
+
+	d.Values["vf-1"].Elements["key-1"] = "val-1"
+	m, err = r3.execute(f)
+	assert.Nilf(t, err, "unexpected error: %v", err)
+	assert.Equal(t, r3.Labels, m, "values should have matched")
+
+	// Match "instance" features
+	r4 := Rule{
+		Labels: map[string]string{"label-4": "label-val-4"},
+		MatchFeatures: FeatureMatcher{
+			FeatureMatcherTerm{
+				Feature:          "domain-1.if-1",
+				MatchExpressions: expression.MatchExpressionSet{"attr-1": expression.MustCreateMatchExpression(expression.MatchIn, "val-1")},
+			},
+		},
+	}
+	m, err = r4.execute(f)
+	assert.Nilf(t, err, "unexpected error: %v", err)
+	assert.Nil(t, m, "instances should not have matched")
+
+	d.Instances["if-1"].Elements[0].Attributes["attr-1"] = "val-1"
+	m, err = r4.execute(f)
+	assert.Nilf(t, err, "unexpected error: %v", err)
+	assert.Equal(t, r4.Labels, m, "instances should have matched")
+
+	// Test multiple feature matchers
+	r5 := Rule{
+		Labels: map[string]string{"label-5": "label-val-5"},
+		MatchFeatures: FeatureMatcher{
+			FeatureMatcherTerm{
+				Feature:          "domain-1.vf-1",
+				MatchExpressions: expression.MatchExpressionSet{"key-1": expression.MustCreateMatchExpression(expression.MatchIn, "val-x")},
+			},
+			FeatureMatcherTerm{
+				Feature:          "domain-1.if-1",
+				MatchExpressions: expression.MatchExpressionSet{"attr-1": expression.MustCreateMatchExpression(expression.MatchIn, "val-1")},
+			},
+		},
+	}
+	m, err = r5.execute(f)
+	assert.Nilf(t, err, "unexpected error: %v", err)
+	assert.Nil(t, m, "instances should not have matched")
+
+	r5.MatchFeatures[0].MatchExpressions["key-1"] = expression.MustCreateMatchExpression(expression.MatchIn, "val-1")
+	m, err = r5.execute(f)
+	assert.Nilf(t, err, "unexpected error: %v", err)
+	assert.Equal(t, r5.Labels, m, "instances should have matched")
+
+	// Test MatchAny
+	r5.MatchAny = []MatchAnyElem{
+		MatchAnyElem{
+			MatchFeatures: FeatureMatcher{
+				FeatureMatcherTerm{
+					Feature:          "domain-1.kf-1",
+					MatchExpressions: expression.MatchExpressionSet{"key-na": expression.MustCreateMatchExpression(expression.MatchExists)},
+				},
+			},
+		},
+	}
+	m, err = r5.execute(f)
+	assert.Nilf(t, err, "unexpected error: %v", err)
+	assert.Nil(t, m, "instances should not have matched")
+
+	r5.MatchAny = append(r5.MatchAny,
+		MatchAnyElem{
+			MatchFeatures: FeatureMatcher{
+				FeatureMatcherTerm{
+					Feature:          "domain-1.kf-1",
+					MatchExpressions: expression.MatchExpressionSet{"key-1": expression.MustCreateMatchExpression(expression.MatchExists)},
+				},
+			},
+		})
+	r5.MatchFeatures[0].MatchExpressions["key-1"] = expression.MustCreateMatchExpression(expression.MatchIn, "val-1")
+	m, err = r5.execute(f)
+	assert.Nilf(t, err, "unexpected error: %v", err)
+	assert.Equal(t, r5.Labels, m, "instances should have matched")
+}