Repository: kubermatic/machine-controller
Pull request #1845 (2024-08-20 13:06:07)

Anexia: extend network configuration to allow multi-interface and multi-IP
**What this PR does / why we need it**:

Can now create multiple network interfaces, each with multiple addresses (one from each configured prefix), easily allowing Machines to have e.g. an IPv4 and IPv6 address.

**What type of PR is this?**

/kind feature

**Special notes for your reviewer**:

We had an internal review already, which you see in the PR of our fork: https://github.com/anexia-it/machine-controller/pull/23

Might add some context if something looks odd.

**Does this PR introduce a user-facing change? Then add your Release Note here**:

```release-note
Anexia: extend network configuration, allowing multiple interfaces and multiple IPs per interface
```

**Documentation**:

```documentation
NONE
```


kubermatic-bot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by:
**Once this PR has been reviewed and has the lgtm label**, please assign [kron4eg](https://github.com/kron4eg) for approval. For more information see [the Kubernetes Code Review Process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process).

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubermatic%2Fmachine-controller).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubermatic/machine-controller/blob/main/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["kron4eg"]} -->
kubermatic-bot: Hi @LittleFox94. Thanks for your PR.

I'm waiting for a [kubermatic](https://github.com/orgs/kubermatic/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://github.com/orgs/kubermatic/people) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=kubermatic%2Fmachine-controller).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>

ahmedwaleedmalik: Some files are missing the boilerplate
LittleFox94: > Some files are missing the boilerplate

oops, gonna fix - sorry!
kubermatic-bot: @LittleFox94: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-verify-boilerplate | baa41af58f2946afa7ff6997434bb443e09a0b7f | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1845/pull-machine-controller-verify-boilerplate/1825882597721378816) | true | `/test pull-machine-controller-verify-boilerplate`
pull-machine-controller-lint | baa41af58f2946afa7ff6997434bb443e09a0b7f | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1845/pull-machine-controller-lint/1825882597650075648) | true | `/test pull-machine-controller-lint`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1845)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @LittleFox94: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-gce | 49780c31cb80a9d018e376102a3aaa1a52a38f11 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1845/pull-machine-controller-e2e-gce/1825885752806871040) | true | `/test pull-machine-controller-e2e-gce`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1845)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
LittleFox94: @ahmedwaleedmalik the failed GCE tests look very unrelated, can we ignore them?
kubermatic-bot: LGTM label has been added.  <details>Git tree hash: 5c5662877bc2889514d4fe1d64f83a877bafb1bd</details>
kubermatic-bot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubermatic/machine-controller/pull/1845#pullrequestreview-2249954875" title="Approved">ahmedwaleedmalik</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubermatic%2Fmachine-controller).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubermatic/machine-controller/blob/main/OWNERS)~~ [ahmedwaleedmalik]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->

Patch:
diff --git a/examples/anexia-machinedeployment.yaml b/examples/anexia-machinedeployment.yaml
index 9049c60b..0b875b4d 100644
--- a/examples/anexia-machinedeployment.yaml
+++ b/examples/anexia-machinedeployment.yaml
@@ -30,7 +30,7 @@ spec:
                 namespace: kube-system
                 name: machine-controller-anexia
                 key: token
-            vlanID: "<< ANEXIA_VLAN_ID >>"
+
             # Currently only the "Flatcar Linux Stable" template is supported.
             # Use templateBuild to specify a build. If empty => latest
             # Alternatively use templateID for a specific template.
@@ -48,6 +48,30 @@ spec:
               - size: 60
                 performanceType: ENT6
 
+            # Each entry in this array will create a network interface in each
+            # Machine, connected to the given VLAN.
+            networks:
+              - vlan: "<< ANEXIA_VLAN_ID >>"
+
+                # If prefixes are given, we reserve an IP address for each of
+                # them - if you give one IPv4 and one IPv6 prefix, your
+                # Machines will have dual-stack connectivity
+                #
+                # As an compatibility-aid for the old cloudProviderSpec.vlanID,
+                # which reserved an IP for the configured VLAN, you can also
+                # have an entry "" (empty string) to get the same behavior -
+                # but this is not recommended.
+                #
+                # Not configuring any prefix might be useful if you want to
+                # configure IP addresses on this interface via other means,
+                # e.g. a Layer2 load balancer.
+                #
+                # Each MachineDeployment needs at least one Network with at
+                # least one Prefix, because we have to know (and thus, reserve)
+                # at least one IP address for each Machine.
+                prefixes:
+                - "<< ANEXIA_PREFIX_ID >>"
+
             # You may have this old disk config attribute in your config - please migrate to the disks attribute.
             # For now it is still recognized though.
             #diskSize: 60
diff --git a/pkg/cloudprovider/provider/anexia/network_provisioning.go b/pkg/cloudprovider/provider/anexia/network_provisioning.go
new file mode 100644
index 00000000..277685d3
--- /dev/null
+++ b/pkg/cloudprovider/provider/anexia/network_provisioning.go
@@ -0,0 +1,158 @@
+/*
+Copyright 2024 The Machine Controller Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package anexia
+
+import (
+	"context"
+	"sync"
+	"time"
+
+	"github.com/kubermatic/machine-controller/pkg/apis/cluster/common"
+	anxtypes "github.com/kubermatic/machine-controller/pkg/cloudprovider/provider/anexia/types"
+	anxclient "go.anx.io/go-anxcloud/pkg/client"
+	anxaddr "go.anx.io/go-anxcloud/pkg/ipam/address"
+	anxvm "go.anx.io/go-anxcloud/pkg/vsphere/provisioning/vm"
+	"go.uber.org/zap"
+)
+
+func networkInterfacesForProvisioning(ctx context.Context, log *zap.SugaredLogger, client anxclient.Client) ([]anxvm.Network, error) {
+	reconcileContext := getReconcileContext(ctx)
+
+	config := reconcileContext.Config
+	status := reconcileContext.Status
+
+	// make sure we have the status.Networks array allocated to fill it with
+	// data, warning if we already have something but not matching the
+	// configuration.
+	if len(status.Networks) != len(config.Networks) {
+		if len(status.Networks) != 0 {
+			log.Warn("size of status.Networks != config.Networks, this should not happen in normal operation - ignoring existing status")
+		}
+
+		status.Networks = make([]anxtypes.NetworkStatus, len(config.Networks))
+	}
+
+	ret := make([]anxvm.Network, len(config.Networks))
+	for netIndex, network := range config.Networks {
+		networkStatus := &status.Networks[netIndex]
+		addresses := make([]string, len(network.Prefixes))
+
+		for prefixIndex, prefix := range network.Prefixes {
+			// make sure we have the address status array allocated to fill it
+			// with our IP reserve status, warning if we already have something
+			// there but not matching the configuration.
+			if len(networkStatus.Addresses) != len(network.Prefixes) {
+				if len(networkStatus.Addresses) != 0 {
+					log.Warnf("size of status.Networks[%[1]v].Addresses != config.Networks[%[1]v].Prefixes, this should not happen in normal operation - ignoring existing status", netIndex)
+				}
+
+				networkStatus.Addresses = make([]anxtypes.NetworkAddressStatus, len(network.Prefixes))
+			}
+
+			reservedIP, err := getIPAddress(ctx, log, &network, prefix, &networkStatus.Addresses[prefixIndex], client)
+			if err != nil {
+				return nil, newError(common.CreateMachineError, "failed to reserve IP: %v", err)
+			}
+
+			addresses[prefixIndex] = reservedIP
+		}
+
+		ret[netIndex] = anxvm.Network{
+			VLAN: network.VlanID,
+			IPs:  addresses,
+
+			// the one NIC type supported by the ADC API
+			NICType: anxtypes.VmxNet3NIC,
+		}
+	}
+
+	return ret, nil
+}
+
+// ENGSUP-3404 is about a race condition when reserving IPs - two calls for one
+// IP each, coming in at "nearly the same millisecond", can result in both
+// reserving the same IP.
+//
+// The proposed fix was to reserve n IPs in one call, but that would require
+// lots of architecture changes - we can't really do the "reserve IPs for all
+// the Machines we want to create and then create the Machines" here.
+//
+// This mutex alleviates the issue enough, that we didn't see it in a long
+// time. It's not impossible this race condition was fixed in some other change
+// and we weren't told, but I'd rather not test this and risk having problems
+// again.. it's not too expensive of a Mutex.
+var _engsup3404mutex sync.Mutex
+
+func getIPAddress(ctx context.Context, log *zap.SugaredLogger, network *resolvedNetwork, prefix string, status *anxtypes.NetworkAddressStatus, client anxclient.Client) (string, error) {
+	reconcileContext := getReconcileContext(ctx)
+
+	// only use IP if it is still unbound
+	if status.ReservedIP != "" && status.IPState == anxtypes.IPStateUnbound && (!status.IPProvisioningExpires.IsZero() && status.IPProvisioningExpires.After(time.Now())) {
+		log.Infow("Re-using already provisioned IP", "ip", status.ReservedIP)
+		return status.ReservedIP, nil
+	}
+
+	_engsup3404mutex.Lock()
+	defer _engsup3404mutex.Unlock()
+
+	log.Info("Creating a new IP for machine")
+	addrAPI := anxaddr.NewAPI(client)
+	config := reconcileContext.Config
+
+	res, err := addrAPI.ReserveRandom(ctx, anxaddr.ReserveRandom{
+		LocationID:        config.LocationID,
+		VlanID:            network.VlanID,
+		PrefixID:          prefix,
+		ReservationPeriod: uint(anxtypes.IPProvisioningExpires / time.Second),
+		Count:             1,
+	})
+	if err != nil {
+		return "", newError(common.InvalidConfigurationMachineError, "failed to reserve an ip address: %v", err)
+	}
+
+	if len(res.Data) < 1 {
+		return "", newError(common.InsufficientResourcesMachineError, "no ip address is available for this machine")
+	}
+
+	ip := res.Data[0].Address
+	status.ReservedIP = ip
+	status.IPState = anxtypes.IPStateUnbound
+	status.IPProvisioningExpires = time.Now().Add(anxtypes.IPProvisioningExpires)
+
+	return ip, nil
+}
+
+func networkReservedAddresses(status *anxtypes.ProviderStatus) []string {
+	ret := make([]string, 0)
+	for _, network := range status.Networks {
+		for _, address := range network.Addresses {
+			if address.ReservedIP != "" && address.IPState == anxtypes.IPStateBound {
+				ret = append(ret, address.ReservedIP)
+			}
+		}
+	}
+
+	return ret
+}
+
+func networkStatusMarkIPsBound(status *anxtypes.ProviderStatus) {
+	for network := range status.Networks {
+		for addr := range status.Networks[network].Addresses {
+			status.Networks[network].Addresses[addr].IPState = anxtypes.IPStateBound
+		}
+	}
+}
diff --git a/pkg/cloudprovider/provider/anexia/provider.go b/pkg/cloudprovider/provider/anexia/provider.go
index a5c28360..4581fbc2 100644
--- a/pkg/cloudprovider/provider/anexia/provider.go
+++ b/pkg/cloudprovider/provider/anexia/provider.go
@@ -24,15 +24,11 @@ import (
 	"fmt"
 	"net/http"
 	"strings"
-	"sync"
 	"time"
 
 	"go.anx.io/go-anxcloud/pkg/api"
-	corev1 "go.anx.io/go-anxcloud/pkg/apis/core/v1"
-	vspherev1 "go.anx.io/go-anxcloud/pkg/apis/vsphere/v1"
 	"go.anx.io/go-anxcloud/pkg/client"
 	anxclient "go.anx.io/go-anxcloud/pkg/client"
-	anxaddr "go.anx.io/go-anxcloud/pkg/ipam/address"
 	"go.anx.io/go-anxcloud/pkg/vsphere"
 	"go.anx.io/go-anxcloud/pkg/vsphere/provisioning/progress"
 	anxvm "go.anx.io/go-anxcloud/pkg/vsphere/provisioning/vm"
@@ -62,31 +58,15 @@ const (
 var (
 	// ErrConfigDiskSizeAndDisks is returned when the config has both DiskSize and Disks set, which is unsupported.
 	ErrConfigDiskSizeAndDisks = errors.New("both the deprecated DiskSize and new Disks attribute are set")
+
+	// ErrConfigVlanIDAndNetworks is returned when the config has both VlanID and Networks set, which is unsupported.
+	ErrConfigVlanIDAndNetworks = errors.New("both the deprecated VlanID and new Networks attribute are set")
 )
 
 type provider struct {
 	configVarResolver *providerconfig.ConfigVarResolver
 }
 
-// resolvedDisk contains the resolved values from types.RawDisk.
-type resolvedDisk struct {
-	anxtypes.RawDisk
-
-	PerformanceType string
-}
-
-// resolvedConfig contains the resolved values from types.RawConfig.
-type resolvedConfig struct {
-	anxtypes.RawConfig
-
-	Token      string
-	VlanID     string
-	LocationID string
-	TemplateID string
-
-	Disks []resolvedDisk
-}
-
 func (p *provider) Create(ctx context.Context, log *zap.SugaredLogger, machine *clusterv1alpha1.Machine, data *cloudprovidertypes.ProviderData, userdata string) (instance instance.Instance, retErr error) {
 	status := getProviderStatus(log, machine)
 	log.Debugw("Machine status", "status", status)
@@ -139,15 +119,10 @@ func provisionVM(ctx context.Context, log *zap.SugaredLogger, client anxclient.C
 		log.Info("Machine does not contain a provisioningID yet. Starting to provision")
 
 		config := reconcileContext.Config
-		reservedIP, err := getIPAddress(ctx, log, client)
+		networkInterfaces, err := networkInterfacesForProvisioning(ctx, log, client)
 		if err != nil {
-			return newError(common.CreateMachineError, "failed to reserve IP: %v", err)
+			return fmt.Errorf("error generating network config for machine: %w", err)
 		}
-		networkInterfaces := []anxvm.Network{{
-			NICType: anxtypes.VmxNet3NIC,
-			IPs:     []string{reservedIP},
-			VLAN:    config.VlanID,
-		}}
 
 		vm := vmAPI.Provisioning().VM().NewDefinition(
 			config.LocationID,
@@ -210,7 +185,7 @@ func provisionVM(ctx context.Context, log *zap.SugaredLogger, client anxclient.C
 		}
 
 		// we successfully sent a VM provisioning request to the API, we consider the IP as 'Bound' now
-		status.IPState = anxtypes.IPStateBound
+		networkStatusMarkIPsBound(status)
 
 		status.ProvisioningID = provisionResponse.Identifier
 		err = updateMachineStatus(reconcileContext.Machine, *status, reconcileContext.ProviderData.Update)
@@ -231,44 +206,6 @@ func provisionVM(ctx context.Context, log *zap.SugaredLogger, client anxclient.C
 	return updateMachineStatus(reconcileContext.Machine, *status, reconcileContext.ProviderData.Update)
 }
 
-var _engsup3404mutex sync.Mutex
-
-func getIPAddress(ctx context.Context, log *zap.SugaredLogger, client anxclient.Client) (string, error) {
-	reconcileContext := getReconcileContext(ctx)
-	status := reconcileContext.Status
-
-	// only use IP if it is still unbound
-	if status.ReservedIP != "" && status.IPState == anxtypes.IPStateUnbound && (!status.IPProvisioningExpires.IsZero() && status.IPProvisioningExpires.After(time.Now())) {
-		log.Infow("Re-using already provisioned IP", "ip", status.ReservedIP)
-		return status.ReservedIP, nil
-	}
-
-	_engsup3404mutex.Lock()
-	defer _engsup3404mutex.Unlock()
-
-	log.Info("Creating a new IP for machine")
-	addrAPI := anxaddr.NewAPI(client)
-	config := reconcileContext.Config
-	res, err := addrAPI.ReserveRandom(ctx, anxaddr.ReserveRandom{
-		LocationID: config.LocationID,
-		VlanID:     config.VlanID,
-		Count:      1,
-	})
-	if err != nil {
-		return "", newError(common.InvalidConfigurationMachineError, "failed to reserve an ip address: %v", err)
-	}
-	if len(res.Data) < 1 {
-		return "", newError(common.InsufficientResourcesMachineError, "no ip address is available for this machine")
-	}
-
-	ip := res.Data[0].Address
-	status.ReservedIP = ip
-	status.IPState = anxtypes.IPStateUnbound
-	status.IPProvisioningExpires = time.Now().Add(anxtypes.IPProvisioningExpires)
-
-	return ip, nil
-}
-
 func isAlreadyProvisioning(ctx context.Context) bool {
 	status := getReconcileContext(ctx).Status
 	condition := meta.FindStatusCondition(status.Conditions, ProvisionedType)
@@ -297,95 +234,6 @@ func ensureConditions(status *anxtypes.ProviderStatus) {
 	}
 }
 
-func resolveTemplateID(ctx context.Context, a api.API, config anxtypes.RawConfig, configVarResolver *providerconfig.ConfigVarResolver, locationID string) (string, error) {
-	templateName, err := configVarResolver.GetConfigVarStringValue(config.Template)
-	if err != nil {
-		return "", fmt.Errorf("failed to get 'template': %w", err)
-	}
-
-	templateBuild, err := configVarResolver.GetConfigVarStringValue(config.TemplateBuild)
-	if err != nil {
-		return "", fmt.Errorf("failed to get 'templateBuild': %w", err)
-	}
-
-	template, err := vspherev1.FindNamedTemplate(ctx, a, templateName, templateBuild, corev1.Location{Identifier: locationID})
-	if err != nil {
-		return "", fmt.Errorf("failed to retrieve named template: %w", err)
-	}
-
-	return template.Identifier, nil
-}
-
-func (p *provider) resolveConfig(ctx context.Context, log *zap.SugaredLogger, config anxtypes.RawConfig) (*resolvedConfig, error) {
-	var err error
-	ret := resolvedConfig{
-		RawConfig: config,
-	}
-
-	ret.Token, err = p.configVarResolver.GetConfigVarStringValueOrEnv(config.Token, anxtypes.AnxTokenEnv)
-	if err != nil {
-		return nil, fmt.Errorf("failed to get 'token': %w", err)
-	}
-
-	ret.LocationID, err = p.configVarResolver.GetConfigVarStringValue(config.LocationID)
-	if err != nil {
-		return nil, fmt.Errorf("failed to get 'locationID': %w", err)
-	}
-
-	ret.TemplateID, err = p.configVarResolver.GetConfigVarStringValue(config.TemplateID)
-	if err != nil {
-		return nil, fmt.Errorf("failed to get 'templateID': %w", err)
-	}
-
-	// when "templateID" is not set, we expect "template" to be
-	if ret.TemplateID == "" {
-		a, _, err := getClient(ret.Token, nil)
-		if err != nil {
-			return nil, fmt.Errorf("failed initializing API clients: %w", err)
-		}
-
-		templateID, err := resolveTemplateID(ctx, a, config, p.configVarResolver, ret.LocationID)
-		if err != nil {
-			return nil, fmt.Errorf("failed retrieving template id from named template: %w", err)
-		}
-
-		ret.TemplateID = templateID
-	}
-
-	ret.VlanID, err = p.configVarResolver.GetConfigVarStringValue(config.VlanID)
-	if err != nil {
-		return nil, fmt.Errorf("failed to get 'vlanID': %w", err)
-	}
-
-	if config.DiskSize != 0 {
-		if len(config.Disks) != 0 {
-			return nil, ErrConfigDiskSizeAndDisks
-		}
-
-		log.Info("Configuration uses the deprecated DiskSize attribute, please migrate to the Disks array instead.")
-
-		config.Disks = []anxtypes.RawDisk{
-			{
-				Size: config.DiskSize,
-			},
-		}
-		config.DiskSize = 0
-	}
-
-	ret.Disks = make([]resolvedDisk, len(config.Disks))
-
-	for idx, disk := range config.Disks {
-		ret.Disks[idx].RawDisk = disk
-
-		ret.Disks[idx].PerformanceType, err = p.configVarResolver.GetConfigVarStringValue(disk.PerformanceType)
-		if err != nil {
-			return nil, fmt.Errorf("failed to get 'performanceType' of disk %v: %w", idx, err)
-		}
-	}
-
-	return &ret, nil
-}
-
 func (p *provider) getConfig(ctx context.Context, log *zap.SugaredLogger, provSpec clusterv1alpha1.ProviderSpec) (*resolvedConfig, *providerconfigtypes.Config, error) {
 	pconfig, err := providerconfigtypes.GetConfig(provSpec)
 	if err != nil {
@@ -456,8 +304,19 @@ func (p *provider) Validate(ctx context.Context, log *zap.SugaredLogger, machine
 		return errors.New("no valid template configured")
 	}
 
-	if config.VlanID == "" {
-		return errors.New("vlan id is missing")
+	if len(config.Networks) == 0 {
+		return errors.New("no networks configured")
+	}
+
+	atLeastOneAddressSourceConfigured := false
+	for _, network := range config.Networks {
+		if len(network.Prefixes) > 0 {
+			atLeastOneAddressSourceConfigured = true
+			break
+		}
+	}
+	if !atLeastOneAddressSourceConfigured {
+		return errors.New("none of the configured networks define an address source, cannot create Machines without any IP")
 	}
 
 	return nil
@@ -506,10 +365,7 @@ func (p *provider) Get(ctx context.Context, log *zap.SugaredLogger, machine *clu
 	}
 
 	instance := anexiaInstance{}
-
-	if status.IPState == anxtypes.IPStateBound && status.ReservedIP != "" {
-		instance.reservedAddresses = []string{status.ReservedIP}
-	}
+	instance.reservedAddresses = networkReservedAddresses(&status)
 
 	timeoutCtx, cancel := context.WithTimeout(ctx, anxtypes.GetRequestTimeout)
 	defer cancel()
diff --git a/pkg/cloudprovider/provider/anexia/resolve_config.go b/pkg/cloudprovider/provider/anexia/resolve_config.go
new file mode 100644
index 00000000..fbe146dc
--- /dev/null
+++ b/pkg/cloudprovider/provider/anexia/resolve_config.go
@@ -0,0 +1,212 @@
+/*
+Copyright 2024 The Machine Controller Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package anexia
+
+import (
+	"context"
+	"fmt"
+
+	"go.uber.org/zap"
+
+	"go.anx.io/go-anxcloud/pkg/api"
+	corev1 "go.anx.io/go-anxcloud/pkg/apis/core/v1"
+	vspherev1 "go.anx.io/go-anxcloud/pkg/apis/vsphere/v1"
+
+	anxtypes "github.com/kubermatic/machine-controller/pkg/cloudprovider/provider/anexia/types"
+)
+
+// resolvedDisk contains the resolved values from types.RawDisk.
+type resolvedDisk struct {
+	anxtypes.RawDisk
+
+	PerformanceType string
+}
+
+// resolvedNetwork contains the resolved values from types.RawNetwork.
+type resolvedNetwork struct {
+	anxtypes.RawNetwork
+
+	VlanID string
+
+	// List of prefixes to each reserve an IP address from.
+	//
+	// Legacy compatibility: may contain an empty string as entry to reserve an IP address from the given VLAN instead of a specific prefix.
+	Prefixes []string
+}
+
+// resolvedConfig contains the resolved values from types.RawConfig.
+type resolvedConfig struct {
+	anxtypes.RawConfig
+
+	Token      string
+	LocationID string
+	TemplateID string
+
+	Disks    []resolvedDisk
+	Networks []resolvedNetwork
+}
+
+func (p *provider) resolveTemplateID(ctx context.Context, a api.API, config anxtypes.RawConfig, locationID string) (string, error) {
+	templateName, err := p.configVarResolver.GetConfigVarStringValue(config.Template)
+	if err != nil {
+		return "", fmt.Errorf("failed to get 'template': %w", err)
+	}
+
+	templateBuild, err := p.configVarResolver.GetConfigVarStringValue(config.TemplateBuild)
+	if err != nil {
+		return "", fmt.Errorf("failed to get 'templateBuild': %w", err)
+	}
+
+	template, err := vspherev1.FindNamedTemplate(ctx, a, templateName, templateBuild, corev1.Location{Identifier: locationID})
+	if err != nil {
+		return "", fmt.Errorf("failed to retrieve named template: %w", err)
+	}
+
+	return template.Identifier, nil
+}
+
+func (p *provider) resolveNetworkConfig(log *zap.SugaredLogger, config anxtypes.RawConfig) (*[]resolvedNetwork, error) {
+	legacyVlanIDConfig, _ := config.VlanID.MarshalJSON()
+	if string(legacyVlanIDConfig) != `""` {
+		if len(config.Networks) != 0 {
+			return nil, ErrConfigVlanIDAndNetworks
+		}
+
+		log.Info("Configuration uses the deprecated VlanID attribute, please migrate to the Networks array instead.")
+
+		vlanID, err := p.configVarResolver.GetConfigVarStringValue(config.VlanID)
+		if err != nil {
+			return nil, fmt.Errorf("failed to get 'vlanID': %w", err)
+		}
+
+		return &[]resolvedNetwork{
+			{
+				VlanID:   vlanID,
+				Prefixes: []string{""},
+			},
+		}, nil
+	}
+
+	ret := make([]resolvedNetwork, len(config.Networks))
+	for netIndex, net := range config.Networks {
+		vlanID, err := p.configVarResolver.GetConfigVarStringValue(net.VlanID)
+		if err != nil {
+			return nil, fmt.Errorf("failed to get 'vlanID' for network %v: %w", netIndex, err)
+		}
+
+		prefixes := make([]string, len(net.PrefixIDs))
+		for prefixIndex, prefix := range net.PrefixIDs {
+			prefixID, err := p.configVarResolver.GetConfigVarStringValue(prefix)
+			if err != nil {
+				return nil, fmt.Errorf("failed to get 'prefixID' for network %v, prefix %v: %w", netIndex, prefixIndex, err)
+			}
+
+			prefixes[prefixIndex] = prefixID
+		}
+
+		ret[netIndex] = resolvedNetwork{
+			VlanID:   vlanID,
+			Prefixes: prefixes,
+		}
+	}
+
+	return &ret, nil
+}
+
+func (p *provider) resolveDiskConfig(log *zap.SugaredLogger, config anxtypes.RawConfig) (*[]resolvedDisk, error) {
+	if config.DiskSize != 0 {
+		if len(config.Disks) != 0 {
+			return nil, ErrConfigDiskSizeAndDisks
+		}
+
+		log.Info("Configuration uses the deprecated DiskSize attribute, please migrate to the Disks array instead.")
+
+		config.Disks = []anxtypes.RawDisk{
+			{
+				Size: config.DiskSize,
+			},
+		}
+		config.DiskSize = 0
+	}
+
+	ret := make([]resolvedDisk, len(config.Disks))
+
+	for idx, disk := range config.Disks {
+		performanceType, err := p.configVarResolver.GetConfigVarStringValue(disk.PerformanceType)
+		if err != nil {
+			return nil, fmt.Errorf("failed to get 'performanceType' of disk %v: %w", idx, err)
+		}
+
+		ret[idx] = resolvedDisk{
+			RawDisk:         disk,
+			PerformanceType: performanceType,
+		}
+	}
+
+	return &ret, nil
+}
+
+func (p *provider) resolveConfig(ctx context.Context, log *zap.SugaredLogger, config anxtypes.RawConfig) (*resolvedConfig, error) {
+	var err error
+	ret := resolvedConfig{
+		RawConfig: config,
+	}
+
+	ret.Token, err = p.configVarResolver.GetConfigVarStringValueOrEnv(config.Token, anxtypes.AnxTokenEnv)
+	if err != nil {
+		return nil, fmt.Errorf("failed to get 'token': %w", err)
+	}
+
+	ret.LocationID, err = p.configVarResolver.GetConfigVarStringValue(config.LocationID)
+	if err != nil {
+		return nil, fmt.Errorf("failed to get 'locationID': %w", err)
+	}
+
+	ret.TemplateID, err = p.configVarResolver.GetConfigVarStringValue(config.TemplateID)
+	if err != nil {
+		return nil, fmt.Errorf("failed to get 'templateID': %w", err)
+	}
+
+	diskConfig, err := p.resolveDiskConfig(log, config)
+	if err != nil {
+		return nil, fmt.Errorf("failed to resolve disk config: %w", err)
+	}
+	ret.Disks = *diskConfig
+
+	networkConfig, err := p.resolveNetworkConfig(log, config)
+	if err != nil {
+		return nil, fmt.Errorf("failed to resolve network config: %w", err)
+	}
+	ret.Networks = *networkConfig
+
+	// when "templateID" is not set, we expect "template" to be
+	if ret.TemplateID == "" {
+		a, _, err := getClient(ret.Token, nil)
+		if err != nil {
+			return nil, fmt.Errorf("failed initializing API clients: %w", err)
+		}
+
+		templateID, err := p.resolveTemplateID(ctx, a, config, ret.LocationID)
+		if err != nil {
+			return nil, fmt.Errorf("failed retrieving template id from named template: %w", err)
+		}
+
+		ret.TemplateID = templateID
+	}
+
+	return &ret, nil
+}
diff --git a/pkg/cloudprovider/provider/anexia/types/types.go b/pkg/cloudprovider/provider/anexia/types/types.go
index 04cb2c99..8b7bfdff 100644
--- a/pkg/cloudprovider/provider/anexia/types/types.go
+++ b/pkg/cloudprovider/provider/anexia/types/types.go
@@ -53,10 +53,20 @@ type RawDisk struct {
 	PerformanceType providerconfigtypes.ConfigVarString `json:"performanceType"`
 }
 
+// RawNetwork specifies a single network interface.
+type RawNetwork struct {
+	// Identifier of the VLAN to attach this network interface to.
+	VlanID providerconfigtypes.ConfigVarString `json:"vlan"`
+
+	// IDs of prefixes to reserve IP addresses from for each Machine on network interface.
+	//
+	// Empty list means that no IPs will be reserved, but the interface will still be added.
+	PrefixIDs []providerconfigtypes.ConfigVarString `json:"prefixes"`
+}
+
 // RawConfig contains all the configuration values for VMs to create, with some values maybe being fetched from secrets.
 type RawConfig struct {
 	Token      providerconfigtypes.ConfigVarString `json:"token,omitempty"`
-	VlanID     providerconfigtypes.ConfigVarString `json:"vlanID"`
 	LocationID providerconfigtypes.ConfigVarString `json:"locationID"`
 
 	TemplateID    providerconfigtypes.ConfigVarString `json:"templateID"`
@@ -71,16 +81,34 @@ type RawConfig struct {
 	DiskSize int `json:"diskSize"`
 
 	Disks []RawDisk `json:"disks"`
+
+	// Deprecated, use Networks instead.
+	VlanID providerconfigtypes.ConfigVarString `json:"vlanID"`
+
+	// Configuration of the network interfaces. At least one entry with at
+	// least one Prefix is required.
+	Networks []RawNetwork `json:"networks"`
+}
+
+type NetworkAddressStatus struct {
+	ReservedIP            string    `json:"reservedIP"`
+	IPState               string    `json:"ipState"`
+	IPProvisioningExpires time.Time `json:"ipProvisioningExpires"`
+}
+
+type NetworkStatus struct {
+	// each entry belongs to a config.Networks.Prefix entry at the same index
+	Addresses []NetworkAddressStatus `json:"addresses"`
 }
 
 type ProviderStatus struct {
-	InstanceID            string         `json:"instanceID"`
-	ProvisioningID        string         `json:"provisioningID"`
-	DeprovisioningID      string         `json:"deprovisioningID"`
-	ReservedIP            string         `json:"reservedIP"`
-	IPState               string         `json:"ipState"`
-	IPProvisioningExpires time.Time      `json:"ipProvisioningExpires"`
-	Conditions            []v1.Condition `json:"conditions,omitempty"`
+	InstanceID       string         `json:"instanceID"`
+	ProvisioningID   string         `json:"provisioningID"`
+	DeprovisioningID string         `json:"deprovisioningID"`
+	Conditions       []v1.Condition `json:"conditions,omitempty"`
+
+	// each entry belongs to the config.Networks entry at the same index
+	Networks []NetworkStatus `json:"networkStatus,omitempty"`
 }
 
 func GetConfig(pconfig providerconfigtypes.Config) (*RawConfig, error) {


Test patch:
diff --git a/pkg/cloudprovider/provider/anexia/helper_test.go b/pkg/cloudprovider/provider/anexia/helper_test.go
index ff5a1f2b..eeb4000c 100644
--- a/pkg/cloudprovider/provider/anexia/helper_test.go
+++ b/pkg/cloudprovider/provider/anexia/helper_test.go
@@ -90,8 +90,11 @@ func hookableConfig(hook func(*anxtypes.RawConfig)) anxtypes.RawConfig {
 			{Size: 5, PerformanceType: newConfigVarString("ENT6")},
 		},
 
+		Networks: []anxtypes.RawNetwork{
+			{VlanID: newConfigVarString("test-vlan"), PrefixIDs: []types.ConfigVarString{newConfigVarString("test-prefix")}},
+		},
+
 		Token:      newConfigVarString("test-token"),
-		VlanID:     newConfigVarString("test-vlan"),
 		LocationID: newConfigVarString("test-location"),
 		TemplateID: newConfigVarString("test-template-id"),
 	}
@@ -112,7 +115,6 @@ func hookableReconcileContext(locationID string, templateID string, hook func(*r
 		Status:   &anxtypes.ProviderStatus{},
 		UserData: "",
 		Config: resolvedConfig{
-			VlanID:     "VLAN-ID",
 			LocationID: locationID,
 			TemplateID: templateID,
 			Disks: []resolvedDisk{
@@ -122,6 +124,14 @@ func hookableReconcileContext(locationID string, templateID string, hook func(*r
 					},
 				},
 			},
+			Networks: []resolvedNetwork{
+				{
+					VlanID: "VLAN-ID",
+					Prefixes: []string{
+						"Prefix-ID",
+					},
+				},
+			},
 			RawConfig: anxtypes.RawConfig{
 				CPUs:   5,
 				Memory: 5,
diff --git a/pkg/cloudprovider/provider/anexia/provider_test.go b/pkg/cloudprovider/provider/anexia/provider_test.go
index bae7a23c..d26546fa 100644
--- a/pkg/cloudprovider/provider/anexia/provider_test.go
+++ b/pkg/cloudprovider/provider/anexia/provider_test.go
@@ -242,7 +242,7 @@ func TestAnexiaProvider(t *testing.T) {
 
 		provider := New(nil).(*provider)
 		for _, testCase := range testCases {
-			templateID, err := resolveTemplateID(context.Background(), a, testCase.config, provider.configVarResolver, "foo")
+			templateID, err := provider.resolveTemplateID(context.Background(), a, testCase.config, "foo")
 			if testCase.expectedError != "" {
 				if err != nil {
 					testhelper.AssertErr(t, err)
@@ -292,17 +292,25 @@ func TestAnexiaProvider(t *testing.T) {
 	t.Run("Test getIPAddress", func(t *testing.T) {
 		t.Parallel()
 		providerStatus := &anxtypes.ProviderStatus{
-			ReservedIP: "",
-			IPState:    "",
+			Networks: []anxtypes.NetworkStatus{
+				{
+					Addresses: []anxtypes.NetworkAddressStatus{
+						{
+							ReservedIP: "",
+							IPState:    "",
+						},
+					},
+				},
+			},
 		}
 		ctx := createReconcileContext(context.Background(), reconcileContext{Status: providerStatus})
 
 		t.Run("with unbound reserved IP", func(t *testing.T) {
 			expectedIP := "8.8.8.8"
-			providerStatus.ReservedIP = expectedIP
-			providerStatus.IPState = anxtypes.IPStateUnbound
-			providerStatus.IPProvisioningExpires = time.Now().Add(anxtypes.IPProvisioningExpires)
-			reservedIP, err := getIPAddress(ctx, log, client)
+			providerStatus.Networks[0].Addresses[0].ReservedIP = expectedIP
+			providerStatus.Networks[0].Addresses[0].IPState = anxtypes.IPStateUnbound
+			providerStatus.Networks[0].Addresses[0].IPProvisioningExpires = time.Now().Add(anxtypes.IPProvisioningExpires)
+			reservedIP, err := getIPAddress(ctx, log, &resolvedNetwork{}, "Prefix-ID", &providerStatus.Networks[0].Addresses[0], client)
 			testhelper.AssertNoErr(t, err)
 			testhelper.AssertEquals(t, expectedIP, reservedIP)
 		})
@@ -342,9 +350,14 @@ func TestValidate(t *testing.T) {
 			Config: hookableConfig(func(c *anxtypes.RawConfig) { c.LocationID.Value = "" }),
 			Error:  errors.New("location id is missing"),
 		},
+
+		ConfigTestCase{
+			Config: hookableConfig(func(c *anxtypes.RawConfig) { c.Networks = []anxtypes.RawNetwork{} }),
+			Error:  errors.New("no networks configured"),
+		},
 		ConfigTestCase{
-			Config: hookableConfig(func(c *anxtypes.RawConfig) { c.VlanID.Value = "" }),
-			Error:  errors.New("vlan id is missing"),
+			Config: hookableConfig(func(c *anxtypes.RawConfig) { c.VlanID.Value = "legacy VLAN-ID" }),
+			Error:  ErrConfigVlanIDAndNetworks,
 		},
 		ConfigTestCase{
 			Config: hookableConfig(func(c *anxtypes.RawConfig) { c.DiskSize = 10; c.Disks = []anxtypes.RawDisk{} }),