Repository: kubernetes/kubernetes (language: go)

Issue:
DRA: Extra unexpected devices allocated when using 'allocationMode: All'
### What happened?

I was testing the ability to allocate both node-local resources (GPUs) along with a new network attached resource called an IMEX channel.

My setup is as follows:
* 8 nodes with 1 GPU each
* 2 pools of IMEX channels (with 10 available in each)
* IMEX channels from the first pool are part of "IMEX domain 1" and can be attached to nodes 0-3
* IMEX channels from the second pool are part of "IMEX domain 2" and can be attached to nodes 4-7

With this setup, I create two ResourceClaims, one ResourceClaim template, and 2 deployments. The two resource Claims are for two distinct IMEX channels, the ResourceClaimTemplate is for all GPUs on a given node, and the 2 deployments (of 4 replicas each) consume these claims / claimtemplates across each replica in order to simulate running 2 MPI jobs across two different IMEX domains.

Here are the specs:
```
---
apiVersion: v1
kind: Namespace
metadata:
  name: imex-test1
---
apiVersion: resource.k8s.io/v1alpha3
kind: ResourceClaim
metadata:
  namespace: imex-test1
  name: shared-imex-channel0
spec:
  devices:
    requests:
    - name: channel
      deviceClassName: imex.nvidia.com
---
apiVersion: resource.k8s.io/v1alpha3
kind: ResourceClaim
metadata:
  namespace: imex-test1
  name: shared-imex-channel1
spec:
  devices:
    requests:
    - name: channel
      deviceClassName: imex.nvidia.com
---
apiVersion: resource.k8s.io/v1alpha3
kind: ResourceClaimTemplate
metadata:
  namespace: imex-test1
  name: all-node-gpus
spec:
  spec:
    devices:
      requests:
      - name: all-gpus
        deviceClassName: gpu.nvidia.com
        allocationMode: All
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: imex-test1
  name: pod0
  labels:
    app: imex-test1-pod0
spec:
  replicas: 4
  selector:
    matchLabels:
      app: pod0
  template:
    metadata:
      labels:
        app: pod0
    spec:
      containers:
      - name: ctr
        image: ubuntu:22.04
        command: ["bash", "-c"]
        args: ["trap 'exit 0' TERM; sleep 9999 & wait"]
        resources:
          claims:
          - name: gpus
          - name: imex-channel
      resourceClaims:
      - name: gpus
        resourceClaimTemplateName: all-node-gpus
      - name: imex-channel
        resourceClaimName: shared-imex-channel0
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: imex-test1
  name: pod1
  labels:
    app: imex-test1-pod1
spec:
  replicas: 4
  selector:
    matchLabels:
      app: pod1
  template:
    metadata:
      labels:
        app: pod1
    spec:
      containers:
      - name: ctr
        image: ubuntu:22.04
        command: ["bash", "-c"]
        args: ["trap 'exit 0' TERM; sleep 9999 & wait"]
        resources:
          claims:
          - name: gpus
          - name: imex-channel
      resourceClaims:
      - name: gpus
        resourceClaimTemplateName: all-node-gpus
      - name: imex-channel
        resourceClaimName: shared-imex-channel1
```

When I run this, I do not get the resource allocated to each pod in each deployment as expected (instead all pods remain pending forever).

However, If I change to explicitly requesting 1 GPU from a node instead of using `allocationMode: All`, things work as expected.

One thing to note is that the code linked below doesn't consider the CEL expression selector in the `gpu.nvidia.com` and `imex.nvidia.com` device classes when calculating `requestData.numDevices`. That might be factor in this somehow:
https://github.com/kubernetes/kubernetes/blob/52095a8b7b9b75d67a3882a21a6647e4f90ade48/staging/src/k8s.io/dynamic-resource-allocation/structured/allocator.go#L176-L210

### What did you expect to happen?

My expectation was that we would see 4 pods from each deployment with the following set of resources:

**deployment 0, pod0**:
1 GPU from a node in the same IMEX domain as all other pods in deployment 0
IMEX channel 0 from the same IMEX domain as all other pods in deployment 0

**deployment 0, pod1**:
1 GPU from a node in the same IMEX domain as all other pods in deployment 0
IMEX channel 0 from the same IMEX domain as all other pods in deployment 0

**deployment 0, pod2**:
1 GPU from a node in the same IMEX domain as all other pods in deployment 0
IMEX channel 0 from the same IMEX domain as all other pods in deployment 0

**deployment 0, pod3**:
1 GPU from a node in the same IMEX domain as all other pods in deployment 0
IMEX channel 0 from the same IMEX domain as all other pods in deployment 0

**deployment 1, pod0**:
1 GPU from a node in the same IMEX domain as all other pods in deployment 1
IMEX channel 0 from the same IMEX domain as all other pods in deployment 1

**deployment 1, pod1**:
1 GPU from a node in the same IMEX domain as all other pods in deployment 1
IMEX channel 0 from the same IMEX domain as all other pods in deployment 1

**deployment 1, pod2**:
1 GPU from a node in the same IMEX domain as all other pods in deployment 1
IMEX channel 0 from the same IMEX domain as all other pods in deployment 1

**deployment 1, pod3**:
1 GPU from a node in the same IMEX domain as all other pods in deployment 1
IMEX channel 0 from the same IMEX domain as all other pods in deployment 1

### How can we reproduce it (as minimally and precisely as possible)?

Apply the specs listed above in a cluster with the following deviceClasses and DRA resources available...

Here are my (relevant) device classes:
```
---
- apiVersion: resource.k8s.io/v1alpha3
  kind: DeviceClass
  metadata:
    name: gpu.nvidia.com
  spec:
    selectors:
    - cel:
        expression: device.driver == 'gpu.nvidia.com' && device.attributes['gpu.nvidia.com'].type
          == 'gpu'
---
- apiVersion: resource.k8s.io/v1alpha3
  kind: DeviceClass
    name: imex.nvidia.com
  spec:
    selectors:
    - cel:
        expression: device.driver == 'gpu.nvidia.com' && device.attributes['gpu.nvidia.com'].type
          == 'imex-channel'
```

Here is the definition of one of my GPU nodes (the others look similar except for the node name):
```
- apiVersion: resource.k8s.io/v1alpha3
  kind: ResourceSlice
  metadata:
    name: k8s-dra-driver-cluster-worker-gpu.nvidia.com-dpwj8
  spec:
    devices:
    - basic:
        attributes:
          architecture:
            string: Ampere
          brand:
            string: Nvidia
          cudaComputeCapability:
            version: 8.0.0
          cudaDriverVersion:
            version: 12.6.0
          driverVersion:
            version: 560.35.3
          index:
            int: 0
          minor:
            int: 7
          productName:
            string: NVIDIA A100-SXM4-40GB
          type:
            string: gpu
          uuid:
            string: GPU-b1028956-cfa2-0990-bf4a-5da9abb51763
        capacity:
          memory: 40Gi
      name: gpu-0
    driver: gpu.nvidia.com
    nodeName: k8s-dra-driver-cluster-worker
    pool:
      generation: 0
      name: k8s-dra-driver-cluster-worker
      resourceSliceCount: 1
```

Here is the definition of my 2 IMEX channel resourceSlices:
```
---
apiVersion: v1
items:
- apiVersion: resource.k8s.io/v1alpha3
  kind: ResourceSlice
  metadata:
    name: imex-domain-0f884867-ba2f-4294-9155-b495ff367eea-1
  spec:
    devices:
    - basic:
        attributes:
          channel:
            int: 0
          type:
            string: imex-channel
      name: imex-channel-0
    - basic:
        attributes:
          channel:
            int: 1
          type:
            string: imex-channel
      name: imex-channel-1
    ...
    driver: gpu.nvidia.com
    nodeSelector:
      nodeSelectorTerms:
      - matchExpressions:
        - key: nvidia.com/gpu.clusteruuid
          operator: In
          values:
          - 0f884867-ba2f-4294-9155-b495ff367eea
        - key: nvidia.com/gpu.cliqueid
          operator: In
          values:
          - "1"
    pool:
      generation: 0
      name: imex-domain-0f884867-ba2f-4294-9155-b495ff367eea-1
      resourceSliceCount: 1
---
apiVersion: v1
items:
- apiVersion: resource.k8s.io/v1alpha3
  kind: ResourceSlice
  metadata:
    name: imex-domain-0f884867-ba2f-4294-9155-b495ff367eea-2
  spec:
    devices:
    - basic:
        attributes:
          channel:
            int: 0
          type:
            string: imex-channel
      name: imex-channel-0
    - basic:
        attributes:
          channel:
            int: 1
          type:
            string: imex-channel
      name: imex-channel-1
    ...
    driver: gpu.nvidia.com
    nodeSelector:
      nodeSelectorTerms:
      - matchExpressions:
        - key: nvidia.com/gpu.clusteruuid
          operator: In
          values:
          - 0f884867-ba2f-4294-9155-b495ff367eea
        - key: nvidia.com/gpu.cliqueid
          operator: In
          values:
          - "2"
    pool:
      generation: 0
      name: imex-domain-0f884867-ba2f-4294-9155-b495ff367eea-2
      resourceSliceCount: 1
```

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.31.0
```

</details>


### Cloud provider

NONE

### OS version

_No response_

### Install tools

_No response_

### Container runtime (CRI) and version (if applicable)

_No response_

### Related plugins (CNI, CSI, ...) and versions (if applicable)

_No response_


Discussion / hints:
> One thing to note is that the code linked below doesn't consider the CEL expression selector in the gpu.nvidia.com and imex.nvidia.com device classes when calculating requestData.numDevices.

In the `resourceapi.DeviceAllocationModeExactCount` case the `numDevices` is exactly what is requested. In the `resourceapi.DeviceAllocationModeAll` case, `isSelectable` checks the device class selector.

Hmm, OK, I should have said (doesn't appear to consider...). Thanks for the clarification.
Can you simplify? Does it still happen with just two nodes and two pods?

It even happens with the following:
```
---
apiVersion: v1
kind: Namespace
metadata:
  name: imex-test1
---
apiVersion: resource.k8s.io/v1alpha3
kind: ResourceClaim
metadata:
  namespace: imex-test1
  name: shared-imex-channel0
spec:
  devices:
    requests:
    - name: channel
      deviceClassName: imex.nvidia.com
---
apiVersion: resource.k8s.io/v1alpha3
kind: ResourceClaimTemplate
metadata:
  namespace: imex-test1
  name: all-node-gpus
spec:
  spec:
    devices:
      requests:
      - name: all-gpus
        deviceClassName: gpu.nvidia.com
        allocationMode: All
---
apiVersion: v1
kind: Pod
metadata:
  namespace: imex-test1
  name: pod0
spec:
  containers:
  - name: ctr
    image: ubuntu:22.04
    command: ["bash", "-c"]
    args: ["trap 'exit 0' TERM; sleep 9999 & wait"]
    resources:
      claims:
      - name: gpus
      - name: imex-channel
  resourceClaims:
  - name: gpus
    resourceClaimTemplateName: all-node-gpus
  - name: imex-channel
    resourceClaimName: shared-imex-channel0
```

The error is:
```
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  7s (x12 over 93s)  default-scheduler  running "DynamicResources" filter plugin: stop allocation
```

And (as before) removing `allocationMode: All` allows it to succeed in scheduling (with the correct set of expected resources).
One thing to note is that I am using the same driver to advertise both node-local resources and network-attached resources. Maybe something in the allocation logic never considered this case?
The allocator doesn't care about who publishes the resource slices.

With just one pod, it's simple enough to fit into https://github.com/kubernetes/kubernetes/blob/df5787a57fe391b75767028ea2a0255c82185680/staging/src/k8s.io/dynamic-resource-allocation/structured/allocator_test.go#L358-L367

I'll try...

/assign

/sig node
/label wg-devicemanagement
@kannon92: The label(s) `/label wg-devicemanagement` cannot be applied. These labels are supported: `api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor, official-cve-feed`. Is this label configured under `labels -> additional_labels` or `labels -> restricted_labels` in `plugin.yaml`?

<details>

In response to [this](https://github.com/kubernetes/kubernetes/issues/127554#issuecomment-2368065091):

>/sig node
>/label wg-devicemanagement


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
/wg device-management
@kannon92: The label(s) `wg/device-management` cannot be applied, because the repository doesn't have them.

<details>

In response to [this](https://github.com/kubernetes/kubernetes/issues/127554#issuecomment-2368069921):

>/wg device-management


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>

Fix patch:
diff --git a/staging/src/k8s.io/dynamic-resource-allocation/structured/allocator.go b/staging/src/k8s.io/dynamic-resource-allocation/structured/allocator.go
index c7bdd1f424742..0b99720707809 100644
--- a/staging/src/k8s.io/dynamic-resource-allocation/structured/allocator.go
+++ b/staging/src/k8s.io/dynamic-resource-allocation/structured/allocator.go
@@ -68,7 +68,7 @@ func NewAllocator(ctx context.Context,
 	}, nil
 }
 
-// ClaimsToAllocate returns the claims that the allocated was created for.
+// ClaimsToAllocate returns the claims that the allocator was created for.
 func (a *Allocator) ClaimsToAllocate() []*resourceapi.ResourceClaim {
 	return a.claimsToAllocate
 }
@@ -169,9 +169,13 @@ func (a *Allocator) Allocate(ctx context.Context, node *v1.Node) (finalResult []
 				return nil, fmt.Errorf("claim %s, request %s: could not retrieve device class %s: %w", klog.KObj(claim), request.Name, request.DeviceClassName, err)
 			}
 
+			// Start collecting information about the request.
+			// The class must be set and stored before calling isSelectable.
 			requestData := requestData{
 				class: class,
 			}
+			requestKey := requestIndices{claimIndex: claimIndex, requestIndex: requestIndex}
+			alloc.requestData[requestKey] = requestData
 
 			switch request.AllocationMode {
 			case resourceapi.DeviceAllocationModeExactCount:
@@ -190,7 +194,7 @@ func (a *Allocator) Allocate(ctx context.Context, node *v1.Node) (finalResult []
 
 					for _, slice := range pool.Slices {
 						for deviceIndex := range slice.Spec.Devices {
-							selectable, err := alloc.isSelectable(requestIndices{claimIndex: claimIndex, requestIndex: requestIndex}, slice, deviceIndex)
+							selectable, err := alloc.isSelectable(requestKey, slice, deviceIndex)
 							if err != nil {
 								return nil, err
 							}
@@ -205,7 +209,7 @@ func (a *Allocator) Allocate(ctx context.Context, node *v1.Node) (finalResult []
 			default:
 				return nil, fmt.Errorf("claim %s, request %s: unsupported count mode %s", klog.KObj(claim), request.Name, request.AllocationMode)
 			}
-			alloc.requestData[requestIndices{claimIndex: claimIndex, requestIndex: requestIndex}] = requestData
+			alloc.requestData[requestKey] = requestData
 			numDevices += requestData.numDevices
 		}
 		alloc.logger.V(6).Info("Checked claim", "claim", klog.KObj(claim), "numDevices", numDevices)
@@ -290,10 +294,13 @@ func (a *Allocator) Allocate(ctx context.Context, node *v1.Node) (finalResult []
 	// All errors get created such that they can be returned by Allocate
 	// without further wrapping.
 	done, err := alloc.allocateOne(deviceIndices{})
+	if errors.Is(err, errStop) {
+		return nil, nil
+	}
 	if err != nil {
 		return nil, err
 	}
-	if errors.Is(err, errStop) || !done {
+	if !done {
 		return nil, nil
 	}
 
@@ -347,7 +354,6 @@ type allocator struct {
 	constraints          [][]constraint                 // one list of constraints per claim
 	requestData          map[requestIndices]requestData // one entry per request
 	allocated            map[DeviceID]bool
-	skippedUnknownDevice bool
 	result               []*resourceapi.AllocationResult
 }
 
@@ -534,20 +540,23 @@ func (alloc *allocator) allocateOne(r deviceIndices) (bool, error) {
 		// For "all" devices we already know which ones we need. We
 		// just need to check whether we can use them.
 		deviceWithID := requestData.allDevices[r.deviceIndex]
-		_, _, err := alloc.allocateDevice(r, deviceWithID.device, deviceWithID.DeviceID, true)
+		success, _, err := alloc.allocateDevice(r, deviceWithID.device, deviceWithID.DeviceID, true)
 		if err != nil {
 			return false, err
 		}
+		if !success {
+			// The order in which we allocate "all" devices doesn't matter,
+			// so we only try with the one which was up next. If we couldn't
+			// get all of them, then there is no solution and we have to stop.
+			return false, errStop
+		}
 		done, err := alloc.allocateOne(deviceIndices{claimIndex: r.claimIndex, requestIndex: r.requestIndex, deviceIndex: r.deviceIndex + 1})
 		if err != nil {
 			return false, err
 		}
-
-		// The order in which we allocate "all" devices doesn't matter,
-		// so we only try with the one which was up next. If we couldn't
-		// get all of them, then there is no solution and we have to stop.
 		if !done {
-			return false, errStop
+			// Backtrack.
+			return false, nil
 		}
 		return done, nil
 	}
@@ -610,9 +619,6 @@ func (alloc *allocator) isSelectable(r requestIndices, slice *resourceapi.Resour
 	device := slice.Spec.Devices[deviceIndex].Basic
 	if device == nil {
 		// Must be some future, unknown device type. We cannot select it.
-		// If we don't find anything else, then this will get reported
-		// in the final result, so remember that we skipped some device.
-		alloc.skippedUnknownDevice = true
 		return false, nil
 	}
 


Test patch:
diff --git a/staging/src/k8s.io/dynamic-resource-allocation/structured/allocator_test.go b/staging/src/k8s.io/dynamic-resource-allocation/structured/allocator_test.go
index 92c37cc955267..e219890f0a6f4 100644
--- a/staging/src/k8s.io/dynamic-resource-allocation/structured/allocator_test.go
+++ b/staging/src/k8s.io/dynamic-resource-allocation/structured/allocator_test.go
@@ -62,6 +62,7 @@ const (
 	slice2  = "slice-2"
 	device1 = "device-1"
 	device2 = "device-2"
+	device3 = "device-3"
 )
 
 func init() {
@@ -582,11 +583,10 @@ func TestAllocator(t *testing.T) {
 
 			expectResults: nil,
 		},
-		"all-devices": {
+		"all-devices-single": {
 			claimsToAllocate: objects(claimWithRequests(claim0, nil, resourceapi.DeviceRequest{
 				Name:            req0,
 				AllocationMode:  resourceapi.DeviceAllocationModeAll,
-				Count:           1,
 				DeviceClassName: classA,
 			})),
 			classes: objects(class(classA, driverA)),
@@ -598,6 +598,25 @@ func TestAllocator(t *testing.T) {
 				deviceAllocationResult(req0, driverA, pool1, device1),
 			)},
 		},
+		"all-devices-many": {
+			claimsToAllocate: objects(claimWithRequests(claim0, nil, resourceapi.DeviceRequest{
+				Name:            req0,
+				AllocationMode:  resourceapi.DeviceAllocationModeAll,
+				DeviceClassName: classA,
+			})),
+			classes: objects(class(classA, driverA)),
+			slices: objects(
+				sliceWithOneDevice(slice1, node1, pool1, driverA),
+				sliceWithOneDevice(slice1, node1, pool2, driverA),
+			),
+			node: node(node1, region1),
+
+			expectResults: []any{allocationResult(
+				localNodeSelector(node1),
+				deviceAllocationResult(req0, driverA, pool1, device1),
+				deviceAllocationResult(req0, driverA, pool2, device1),
+			)},
+		},
 		"all-devices-of-the-incomplete-pool": {
 			claimsToAllocate: objects(claimWithRequests(claim0, nil, resourceapi.DeviceRequest{
 				Name:            req0,
@@ -619,6 +638,200 @@ func TestAllocator(t *testing.T) {
 			expectResults: nil,
 			expectError:   gomega.MatchError(gomega.ContainSubstring("claim claim-0, request req-0: asks for all devices, but resource pool driver-a/pool-1 is currently being updated")),
 		},
+		"all-devices-plus-another": {
+			claimsToAllocate: objects(
+				claimWithRequests(claim0, nil, resourceapi.DeviceRequest{
+					Name:            req0,
+					AllocationMode:  resourceapi.DeviceAllocationModeAll,
+					DeviceClassName: classA,
+				}),
+				claimWithRequests(claim1, nil, resourceapi.DeviceRequest{
+					Name:            req0,
+					AllocationMode:  resourceapi.DeviceAllocationModeExactCount,
+					Count:           1,
+					DeviceClassName: classB,
+				}),
+			),
+			classes: objects(
+				class(classA, driverA),
+				class(classB, driverB),
+			),
+			slices: objects(
+				sliceWithOneDevice(slice1, node1, pool1, driverA),
+				sliceWithOneDevice(slice1, node1, pool1, driverB),
+			),
+			node: node(node1, region1),
+
+			expectResults: []any{
+				allocationResult(
+					localNodeSelector(node1),
+					deviceAllocationResult(req0, driverA, pool1, device1),
+				),
+				allocationResult(
+					localNodeSelector(node1),
+					deviceAllocationResult(req0, driverB, pool1, device1),
+				),
+			},
+		},
+		"all-devices-plus-another-reversed": {
+			claimsToAllocate: objects(
+				claimWithRequests(claim1, nil, resourceapi.DeviceRequest{
+					Name:            req0,
+					AllocationMode:  resourceapi.DeviceAllocationModeExactCount,
+					Count:           1,
+					DeviceClassName: classB,
+				}),
+				claimWithRequests(claim0, nil, resourceapi.DeviceRequest{
+					Name:            req0,
+					AllocationMode:  resourceapi.DeviceAllocationModeAll,
+					DeviceClassName: classA,
+				}),
+			),
+			classes: objects(
+				class(classA, driverA),
+				class(classB, driverB),
+			),
+			slices: objects(
+				sliceWithOneDevice(slice1, node1, pool1, driverA),
+				sliceWithOneDevice(slice1, node1, pool1, driverB),
+			),
+			node: node(node1, region1),
+
+			expectResults: []any{
+				allocationResult(
+					localNodeSelector(node1),
+					deviceAllocationResult(req0, driverB, pool1, device1),
+				),
+				allocationResult(
+					localNodeSelector(node1),
+					deviceAllocationResult(req0, driverA, pool1, device1),
+				),
+			},
+		},
+		"all-devices-many-plus-another": {
+			claimsToAllocate: objects(
+				claimWithRequests(claim0, nil, resourceapi.DeviceRequest{
+					Name:            req0,
+					AllocationMode:  resourceapi.DeviceAllocationModeAll,
+					DeviceClassName: classA,
+				}),
+				claimWithRequests(claim1, nil, resourceapi.DeviceRequest{
+					Name:            req0,
+					AllocationMode:  resourceapi.DeviceAllocationModeExactCount,
+					Count:           1,
+					DeviceClassName: classB,
+				}),
+			),
+			classes: objects(
+				class(classA, driverA),
+				class(classB, driverB),
+			),
+			slices: objects(
+				slice(slice1, node1, pool1, driverA,
+					device(device1, nil, nil),
+					device(device2, nil, nil),
+				),
+				sliceWithOneDevice(slice1, node1, pool1, driverB),
+			),
+			node: node(node1, region1),
+
+			expectResults: []any{
+				allocationResult(
+					localNodeSelector(node1),
+					deviceAllocationResult(req0, driverA, pool1, device1),
+					deviceAllocationResult(req0, driverA, pool1, device2),
+				),
+				allocationResult(
+					localNodeSelector(node1),
+					deviceAllocationResult(req0, driverB, pool1, device1),
+				),
+			},
+		},
+		"all-devices-many-plus-another-reversed": {
+			claimsToAllocate: objects(
+				claimWithRequests(claim1, nil, resourceapi.DeviceRequest{
+					Name:            req0,
+					AllocationMode:  resourceapi.DeviceAllocationModeExactCount,
+					Count:           1,
+					DeviceClassName: classB,
+				}),
+				claimWithRequests(claim0, nil, resourceapi.DeviceRequest{
+					Name:            req0,
+					AllocationMode:  resourceapi.DeviceAllocationModeAll,
+					DeviceClassName: classA,
+				}),
+			),
+			classes: objects(
+				class(classA, driverA),
+				class(classB, driverB),
+			),
+			slices: objects(
+				slice(slice1, node1, pool1, driverA,
+					device(device1, nil, nil),
+					device(device2, nil, nil),
+				),
+				sliceWithOneDevice(slice1, node1, pool1, driverB),
+			),
+			node: node(node1, region1),
+
+			expectResults: []any{
+				allocationResult(
+					localNodeSelector(node1),
+					deviceAllocationResult(req0, driverB, pool1, device1),
+				),
+				allocationResult(
+					localNodeSelector(node1),
+					deviceAllocationResult(req0, driverA, pool1, device1),
+					deviceAllocationResult(req0, driverA, pool1, device2),
+				),
+			},
+		},
+		"all-devices-no-solution": {
+			// One device, two claims both trying to allocate it.
+			claimsToAllocate: objects(
+				claimWithRequests(claim1, nil, resourceapi.DeviceRequest{
+					Name:            req0,
+					AllocationMode:  resourceapi.DeviceAllocationModeExactCount,
+					Count:           1,
+					DeviceClassName: classA,
+				}),
+				claimWithRequests(claim0, nil, resourceapi.DeviceRequest{
+					Name:            req0,
+					AllocationMode:  resourceapi.DeviceAllocationModeAll,
+					DeviceClassName: classA,
+				}),
+			),
+			classes: objects(
+				class(classA, driverA),
+			),
+			slices: objects(
+				sliceWithOneDevice(slice1, node1, pool1, driverA),
+			),
+			node: node(node1, region1),
+		},
+		"all-devices-no-solution-reversed": {
+			// One device, two claims both trying to allocate it.
+			claimsToAllocate: objects(
+				claimWithRequests(claim0, nil, resourceapi.DeviceRequest{
+					Name:            req0,
+					AllocationMode:  resourceapi.DeviceAllocationModeAll,
+					DeviceClassName: classA,
+				}),
+				claimWithRequests(claim1, nil, resourceapi.DeviceRequest{
+					Name:            req0,
+					AllocationMode:  resourceapi.DeviceAllocationModeExactCount,
+					Count:           1,
+					DeviceClassName: classA,
+				}),
+			),
+			classes: objects(
+				class(classA, driverA),
+			),
+			slices: objects(
+				sliceWithOneDevice(slice1, node1, pool1, driverA),
+			),
+			node: node(node1, region1),
+		},
 		"network-attached-device": {
 			claimsToAllocate: objects(claim(claim0, req0, classA)),
 			classes:          objects(class(classA, driverA)),
@@ -766,7 +979,7 @@ func TestAllocator(t *testing.T) {
 			claimsToAllocate: objects(claimWithRequests(
 				claim0,
 				[]resourceapi.DeviceConstraint{{MatchAttribute: &intAttribute}},
-				request(req0, classA, 3)),
+				request(req0, classA, 2)),
 			),
 			classes: objects(class(classA, driverA), class(classB, driverB)),
 			slices: objects(slice(slice1, node1, pool1, driverA,
@@ -781,30 +994,36 @@ func TestAllocator(t *testing.T) {
 
 			expectResults: nil,
 		},
-		"with-constraint-not-matching-version-attribute": {
-			claimsToAllocate: objects(claimWithRequests(
-				claim0,
-				[]resourceapi.DeviceConstraint{{MatchAttribute: &versionAttribute}},
-				request(req0, classA, 3)),
+		"with-constraint-not-matching-int-attribute-all-devices": {
+			claimsToAllocate: objects(
+				func() *resourceapi.ResourceClaim {
+					claim := claimWithRequests(
+						claim0,
+						[]resourceapi.DeviceConstraint{{MatchAttribute: &intAttribute}},
+						request(req0, classA, 0),
+					)
+					claim.Spec.Devices.Requests[0].AllocationMode = resourceapi.DeviceAllocationModeAll
+					return claim
+				}(),
 			),
 			classes: objects(class(classA, driverA), class(classB, driverB)),
 			slices: objects(slice(slice1, node1, pool1, driverA,
 				device(device1, nil, map[resourceapi.QualifiedName]resourceapi.DeviceAttribute{
-					"driverVersion": {VersionValue: ptr.To("1.0.0")},
+					"numa": {IntValue: ptr.To(int64(1))},
 				}),
 				device(device2, nil, map[resourceapi.QualifiedName]resourceapi.DeviceAttribute{
-					"driverVersion": {VersionValue: ptr.To("2.0.0")},
+					"numa": {IntValue: ptr.To(int64(2))},
 				}),
 			)),
 			node: node(node1, region1),
 
-			expectResults: nil,
+			expectError: gomega.MatchError(gomega.ContainSubstring("claim claim-0, request req-0: cannot add device driver-a/pool-1/device-2 because a claim constraint would not be satisfied")),
 		},
 		"with-constraint-not-matching-string-attribute": {
 			claimsToAllocate: objects(claimWithRequests(
 				claim0,
 				[]resourceapi.DeviceConstraint{{MatchAttribute: &stringAttribute}},
-				request(req0, classA, 3)),
+				request(req0, classA, 2)),
 			),
 			classes: objects(class(classA, driverA), class(classB, driverB)),
 			slices: objects(slice(slice1, node1, pool1, driverA,
@@ -823,7 +1042,7 @@ func TestAllocator(t *testing.T) {
 			claimsToAllocate: objects(claimWithRequests(
 				claim0,
 				[]resourceapi.DeviceConstraint{{MatchAttribute: &boolAttribute}},
-				request(req0, classA, 3)),
+				request(req0, classA, 2)),
 			),
 			classes: objects(class(classA, driverA), class(classB, driverB)),
 			slices: objects(slice(slice1, node1, pool1, driverA,
@@ -838,6 +1057,95 @@ func TestAllocator(t *testing.T) {
 
 			expectResults: nil,
 		},
+		"with-constraint-not-matching-version-attribute": {
+			claimsToAllocate: objects(claimWithRequests(
+				claim0,
+				[]resourceapi.DeviceConstraint{{MatchAttribute: &versionAttribute}},
+				request(req0, classA, 2)),
+			),
+			classes: objects(class(classA, driverA), class(classB, driverB)),
+			slices: objects(slice(slice1, node1, pool1, driverA,
+				device(device1, nil, map[resourceapi.QualifiedName]resourceapi.DeviceAttribute{
+					"driverVersion": {VersionValue: ptr.To("1.0.0")},
+				}),
+				device(device2, nil, map[resourceapi.QualifiedName]resourceapi.DeviceAttribute{
+					"driverVersion": {VersionValue: ptr.To("2.0.0")},
+				}),
+			)),
+			node: node(node1, region1),
+
+			expectResults: nil,
+		},
+		"with-constraint-for-request": {
+			claimsToAllocate: objects(claimWithRequests(
+				claim0,
+				[]resourceapi.DeviceConstraint{
+					{
+						Requests:       []string{req0},
+						MatchAttribute: &versionAttribute,
+					},
+				},
+				request(req0, classA, 1),
+				request(req1, classA, 1),
+			)),
+			classes: objects(class(classA, driverA), class(classB, driverB)),
+			slices: objects(slice(slice1, node1, pool1, driverA,
+				device(device1, nil, map[resourceapi.QualifiedName]resourceapi.DeviceAttribute{
+					"driverVersion": {VersionValue: ptr.To("1.0.0")},
+				}),
+				device(device2, nil, map[resourceapi.QualifiedName]resourceapi.DeviceAttribute{
+					"driverVersion": {VersionValue: ptr.To("2.0.0")},
+				}),
+			)),
+			node: node(node1, region1),
+
+			expectResults: []any{allocationResult(
+				localNodeSelector(node1),
+				deviceAllocationResult(req0, driverA, pool1, device1),
+				deviceAllocationResult(req1, driverA, pool1, device2),
+			)},
+		},
+		"with-constraint-for-request-retry": {
+			claimsToAllocate: objects(claimWithRequests(
+				claim0,
+				[]resourceapi.DeviceConstraint{
+					{
+						Requests:       []string{req0},
+						MatchAttribute: &versionAttribute,
+					},
+					{
+						MatchAttribute: &stringAttribute,
+					},
+				},
+				request(req0, classA, 1),
+				request(req1, classA, 1),
+			)),
+			classes: objects(class(classA, driverA), class(classB, driverB)),
+			slices: objects(slice(slice1, node1, pool1, driverA,
+				// This device does not satisfy the second
+				// match attribute, so the allocator must
+				// backtrack.
+				device(device1, nil, map[resourceapi.QualifiedName]resourceapi.DeviceAttribute{
+					"driverVersion":   {VersionValue: ptr.To("1.0.0")},
+					"stringAttribute": {StringValue: ptr.To("a")},
+				}),
+				device(device2, nil, map[resourceapi.QualifiedName]resourceapi.DeviceAttribute{
+					"driverVersion":   {VersionValue: ptr.To("2.0.0")},
+					"stringAttribute": {StringValue: ptr.To("b")},
+				}),
+				device(device3, nil, map[resourceapi.QualifiedName]resourceapi.DeviceAttribute{
+					"driverVersion":   {VersionValue: ptr.To("3.0.0")},
+					"stringAttribute": {StringValue: ptr.To("b")},
+				}),
+			)),
+			node: node(node1, region1),
+
+			expectResults: []any{allocationResult(
+				localNodeSelector(node1),
+				deviceAllocationResult(req0, driverA, pool1, device2),
+				deviceAllocationResult(req1, driverA, pool1, device3),
+			)},
+		},
 		"with-class-device-config": {
 			claimsToAllocate: objects(claim(claim0, req0, classA)),
 			classes:          objects(classWithConfig(classA, driverA, "classAttribute")),
@@ -870,6 +1178,138 @@ func TestAllocator(t *testing.T) {
 				),
 			},
 		},
+		"unknown-selector": {
+			claimsToAllocate: objects(
+				func() *resourceapi.ResourceClaim {
+					claim := claim(claim0, req0, classA)
+					claim.Spec.Devices.Requests[0].Selectors = []resourceapi.DeviceSelector{
+						{ /* empty = unknown future selector */ },
+					}
+					return claim
+				}(),
+			),
+			classes: objects(class(classA, driverA)),
+			slices:  objects(sliceWithOneDevice(slice1, node1, pool1, driverA)),
+			node:    node(node1, region1),
+
+			expectError: gomega.MatchError(gomega.ContainSubstring("CEL expression empty (unsupported selector type?)")),
+		},
+		"unknown-allocation-mode": {
+			claimsToAllocate: objects(
+				func() *resourceapi.ResourceClaim {
+					claim := claim(claim0, req0, classA)
+					claim.Spec.Devices.Requests[0].AllocationMode = resourceapi.DeviceAllocationMode("future-mode")
+					return claim
+				}(),
+			),
+			classes: objects(class(classA, driverA)),
+			slices:  objects(sliceWithOneDevice(slice1, node1, pool1, driverA)),
+			node:    node(node1, region1),
+
+			expectError: gomega.MatchError(gomega.ContainSubstring("unsupported count mode future-mode")),
+		},
+		"unknown-constraint": {
+			claimsToAllocate: objects(
+				func() *resourceapi.ResourceClaim {
+					claim := claim(claim0, req0, classA)
+					claim.Spec.Devices.Constraints = []resourceapi.DeviceConstraint{
+						{ /* empty = unknown */ },
+					}
+					return claim
+				}(),
+			),
+			classes: objects(class(classA, driverA)),
+			slices:  objects(sliceWithOneDevice(slice1, node1, pool1, driverA)),
+			node:    node(node1, region1),
+
+			expectError: gomega.MatchError(gomega.ContainSubstring("empty constraint (unsupported constraint type?)")),
+		},
+		"unknown-device": {
+			claimsToAllocate: objects(claim(claim0, req0, classA)),
+			classes:          objects(class(classA, driverA)),
+			slices: objects(
+				func() *resourceapi.ResourceSlice {
+					slice := sliceWithOneDevice(slice1, node1, pool1, driverA)
+					slice.Spec.Devices[0].Basic = nil /* empty = unknown future extension */
+					return slice
+				}(),
+			),
+			node: node(node1, region1),
+		},
+		"invalid-CEL-one-device": {
+			claimsToAllocate: objects(
+				func() *resourceapi.ResourceClaim {
+					claim := claim(claim0, req0, classA)
+					claim.Spec.Devices.Requests[0].Selectors = []resourceapi.DeviceSelector{
+						{CEL: &resourceapi.CELDeviceSelector{Expression: "noSuchVar"}},
+					}
+					return claim
+				}(),
+			),
+			classes: objects(class(classA, driverA)),
+			slices:  objects(sliceWithOneDevice(slice1, node1, pool1, driverA)),
+			node:    node(node1, region1),
+
+			expectError: gomega.MatchError(gomega.ContainSubstring("undeclared reference")),
+		},
+		"invalid-CEL-one-device-class": {
+			claimsToAllocate: objects(claim(claim0, req0, classA)),
+			classes: objects(
+				func() *resourceapi.DeviceClass {
+					c := class(classA, driverA)
+					c.Spec.Selectors[0].CEL.Expression = "noSuchVar"
+					return c
+				}(),
+			),
+			slices: objects(sliceWithOneDevice(slice1, node1, pool1, driverA)),
+			node:   node(node1, region1),
+
+			expectError: gomega.MatchError(gomega.ContainSubstring("undeclared reference")),
+		},
+		"invalid-CEL-all-devices": {
+			claimsToAllocate: objects(
+				func() *resourceapi.ResourceClaim {
+					claim := claim(claim0, req0, classA)
+					claim.Spec.Devices.Requests[0].Selectors = []resourceapi.DeviceSelector{
+						{CEL: &resourceapi.CELDeviceSelector{Expression: "noSuchVar"}},
+					}
+					claim.Spec.Devices.Requests[0].AllocationMode = resourceapi.DeviceAllocationModeAll
+					return claim
+				}(),
+			),
+			classes: objects(class(classA, driverA)),
+			slices:  objects(sliceWithOneDevice(slice1, node1, pool1, driverA)),
+			node:    node(node1, region1),
+
+			expectError: gomega.MatchError(gomega.ContainSubstring("undeclared reference")),
+		},
+		"too-many-devices-single-request": {
+			claimsToAllocate: objects(claimWithRequests(claim0, nil, request(req0, classA, 500))),
+			classes:          objects(class(classA, driverA)),
+
+			expectError: gomega.MatchError(gomega.ContainSubstring("exceeds the claim limit")),
+		},
+		"many-devices-okay": {
+			claimsToAllocate: objects(claimWithRequests(claim0, nil, request(req0, classA, resourceapi.AllocationResultsMaxSize))),
+			classes:          objects(class(classA, driverA)),
+		},
+		"too-many-devices-total": {
+			claimsToAllocate: objects(
+				claimWithRequests(claim0, nil,
+					request(req0, classA, resourceapi.AllocationResultsMaxSize),
+					request(req1, classA, 1),
+				),
+			),
+			classes: objects(class(classA, driverA)),
+
+			expectError: gomega.MatchError(gomega.ContainSubstring("exceeds the claim limit")),
+		},
+		"all-devices-invalid-CEL": {
+			claimsToAllocate: objects(claimWithRequests(claim0, nil, request(req0, classA, 500))),
+			classes:          objects(class(classA, driverA)),
+
+			expectError: gomega.MatchError(gomega.ContainSubstring("exceeds the claim limit")),
+		},
 	}
 
 	for name, tc := range testcases {