Repository: fleetdm/fleet (Go)

Issue:
IdP vars: Trigger profile resend when IdP values change
ScimUser is connected to 1+ hosts via join table.
1. Add an indicator to the scim_user row to indicate which user changed and what changed (for PATCH/PUT Users/Groups,  when username/username_local_part/groups changes).
2. Scan all configuration profiles for the changed user's host(s) to see if they use the updated var(s). This can be done as its own batch job.
3. For profiles that contain the variables, mark them for resend. 
4. After marking profile for resend, clear the indicator on scim_user row.

## Task

- [x] Replace variables with value on send profile to device
- [x] Trigger resend for affected hosts/profiles when IdP user/group is modified 
- [x] Store variables used by profiles on create via UI or batch-set via fleetctl
- [x] Implement DB migration to collect used variables by existing profiles
- [x] Integration test that verifies delivery of profile with vars, some change that should retrigger it, and the redelivery and address Victor's comments: https://github.com/fleetdm/fleet/pull/28645


Discussion: Idea: DB table to join variable references to profiles. ++ migration for current profiles

++ is secrets apart of this table.
@georgekarrv @getvictor I did some code research to see how it works currently and figure out the best way to tackle this, here's what I have in mind:

* I will create new DB tables `fleet_variables` and `mdm_configuration_profile_variables` to store variable names (or prefixes) and the config profiles that use them:
    - `fleet_variables.name`, `fleet_variables.is_prefix`
    - `mdm_config_prof_var.apple_profile_uuid`, `...windows_profile_uuid` and `...fleet_variable_id` to map apple/windows config profile to a variable
    - Why not a single `profile_uuid` column? for the same reason we didn't use this for `mdm_configuration_profile_labels`, we can't have proper FKs to 2 distinct tables with this. I'll also add a CHECK CONSTRAINT to ensure only one is set, as we did for labels.
    - Why not also `apple_declaration_uuid`? because we don't support vars in DDM currently. We can add it later if we add support for this.
    - Why not support secrets too? secrets are handled quite differently than variables, and AFAICT currently do not trigger a resend on change, so no current benefit to be stored there. Will be relatively easy to add in the future if we need that lookup table (e.g. add a `is_secret` column, or a `variable_type` if we expect to have more types).
* I will *not* alter `scim_user` table to track what changed, as I don't think it will be necessary with the lookup table - in the same transaction that updates the user/groups, we'll be able to efficiently identify the impacted profiles with the lookup table and mark them as pending.
> [@georgekarrv](https://github.com/georgekarrv) [@getvictor](https://github.com/getvictor) I did some code research to see how it works currently and figure out the best way to tackle this, here's what I have in mind:
> 
> * I will create new DB tables `fleet_variables` and `mdm_configuration_profile_variables` to store variable names (or prefixes) and the config profiles that use them:
>   
>   * `fleet_variables.name`, `fleet_variables.is_prefix`
>   * `mdm_config_prof_var.apple_profile_uuid`, `...windows_profile_uuid` and `...fleet_variable_id` to map apple/windows config profile to a variable
>   * Why not a single `profile_uuid` column? for the same reason we didn't use this for `mdm_configuration_profile_labels`, we can't have proper FKs to 2 distinct tables with this. I'll also add a CHECK CONSTRAINT to ensure only one is set, as we did for labels.
>   * Why not also `apple_declaration_uuid`? because we don't support vars in DDM currently. We can add it later if we add support for this.
>   * Why not support secrets too? secrets are handled quite differently than variables, and AFAICT currently do not trigger a resend on change, so no current benefit to be stored there. Will be relatively easy to add in the future if we need that lookup table (e.g. add a `is_secret` column, or a `variable_type` if we expect to have more types).
> * I will _not_ alter `scim_user` table to track what changed, as I don't think it will be necessary with the lookup table - in the same transaction that updates the user/groups, we'll be able to efficiently identify the impacted profiles with the lookup table and mark them as pending.

Secret updates are only processed during gitops using timestamps. For example: `host_mdm_apple_profiles.secrets_updated_at`. So if a secret was updated during gitops, the batch profile update endpoint would save the new timestamp, causing the profile to be resent.
So, it is true that we do not trigger a resend on change of secret, but we trigger a resend on batch profile update.

Does the `fleet_variables` table contain static values, and will we need to keep it updated with migrations?

We may want to resend scripts or software installs with fleet variables in the future. Maybe we worry about that later.
 
> Does the fleet_variables table contain static values, and will we need to keep it updated with migrations?

Not sure what you mean here, the goal is to have them contain variable _names_ (e.g. `FLEET_VAR_HOST_END_USER_IDP_USERNAME`) or variable _prefixes_ for those that can have dynamic suffixes (e.g. `FLEET_VAR_DIGICERT_DATA_`) and a column that indicates if it is an exact variable name or a prefix. Prefix variables are not technically needed for this story, but I think that since we'll have a migration to parse the profiles and extract variables, we may as well collect them there immediately for future use.

We'll need to keep `fleet_variables` table updated with migrations when we add new variables, yes. And we'll need to keep `mdm_configuration_profile_variables` mapping updated when the profiles get inserted/updated/deleted.

> We may want to resend scripts or software installs with fleet variables in the future. Maybe we worry about that later.

Would we really want to rerun a script/reinstall software because a variable's value changed? I don't know, maybe for software installers I guess. In any case, at least some of the mechanism will be in place (the `fleet_variables` table). What would be needed to support those in the future (off the top of my head) would be a DB migration to scan existing scripts for variables, and store them in a new table(s?) that maps script/installer IDs to the variables it uses.



Fix (patch):
diff --git a/ee/server/service/mdm.go b/ee/server/service/mdm.go
index 32088f2dfc6f..705648c8a7dd 100644
--- a/ee/server/service/mdm.go
+++ b/ee/server/service/mdm.go
@@ -163,7 +163,7 @@ func (svc *Service) MDMAppleEnableFileVaultAndEscrow(ctx context.Context, teamID
 		return ctxerr.Wrap(ctx, err, "enabling FileVault")
 	}
 
-	// filevault profile is a fleet-controlled profile that doesn't use any variable
+	// filevault profile is a fleet-controlled profile that doesn't use any Fleet variables
 	_, err = svc.ds.NewMDMAppleConfigProfile(ctx, *cp, nil)
 	return ctxerr.Wrap(ctx, err, "enabling FileVault")
 }
diff --git a/server/datastore/mysql/apple_mdm.go b/server/datastore/mysql/apple_mdm.go
index 08193ae471c4..12847b14da6b 100644
--- a/server/datastore/mysql/apple_mdm.go
+++ b/server/datastore/mysql/apple_mdm.go
@@ -36,7 +36,7 @@ import (
 // addHostMDMCommandsBatchSize is the number of host MDM commands to add in a single batch. This is a var so that it can be modified in tests.
 var addHostMDMCommandsBatchSize = 10000
 
-func (ds *Datastore) NewMDMAppleConfigProfile(ctx context.Context, cp fleet.MDMAppleConfigProfile, usesFleetVars map[string]struct{}) (*fleet.MDMAppleConfigProfile, error) {
+func (ds *Datastore) NewMDMAppleConfigProfile(ctx context.Context, cp fleet.MDMAppleConfigProfile, usesFleetVars []string) (*fleet.MDMAppleConfigProfile, error) {
 	profUUID := "a" + uuid.New().String()
 	stmt := `
 INSERT INTO
@@ -107,7 +107,9 @@ INSERT INTO
 		if _, err := batchSetProfileLabelAssociationsDB(ctx, tx, labels, profWithoutLabels, "darwin"); err != nil {
 			return ctxerr.Wrap(ctx, err, "inserting darwin profile label associations")
 		}
-		if err := batchSetProfileVariableAssociationsDB(ctx, tx, map[string]map[string]struct{}{profUUID: usesFleetVars}, "darwin"); err != nil {
+		if err := batchSetProfileVariableAssociationsDB(ctx, tx, []fleet.MDMProfileUUIDFleetVariables{
+			{ProfileUUID: profUUID, FleetVariables: usesFleetVars},
+		}, "darwin"); err != nil {
 			return ctxerr.Wrap(ctx, err, "inserting darwin profile variable associations")
 		}
 
@@ -1857,7 +1859,7 @@ func (ds *Datastore) batchSetMDMAppleProfilesDB(
 	tx sqlx.ExtContext,
 	tmID *uint,
 	profiles []*fleet.MDMAppleConfigProfile,
-	profilesVariablesByIdentifier map[string]map[string]struct{},
+	profilesVariablesByIdentifier []fleet.MDMProfileIdentifierFleetVariables,
 ) (updatedDB bool, err error) {
 	const loadExistingProfiles = `
 SELECT
@@ -2003,11 +2005,19 @@ ON DUPLICATE KEY UPDATE
 			return false, ctxerr.Wrap(ctx, err, "load inserted/updated profiles")
 		}
 
+		lookupVariablesByIdentifier := make(map[string][]string, len(profilesVariablesByIdentifier))
+		for _, p := range profilesVariablesByIdentifier {
+			lookupVariablesByIdentifier[p.Identifier] = p.FleetVariables
+		}
+
 		// collect the profiles+variables that need to be upserted, keyed by
 		// profile UUID as needed by the batch-set profile variables function.
-		profilesVarsToUpsert := make(map[string]map[string]struct{}, len(insertedUpdatedProfs))
+		profilesVarsToUpsert := make([]fleet.MDMProfileUUIDFleetVariables, 0, len(insertedUpdatedProfs))
 		for _, p := range insertedUpdatedProfs {
-			profilesVarsToUpsert[p.ProfileUUID] = profilesVariablesByIdentifier[p.Identifier]
+			profilesVarsToUpsert = append(profilesVarsToUpsert, fleet.MDMProfileUUIDFleetVariables{
+				ProfileUUID:    p.ProfileUUID,
+				FleetVariables: lookupVariablesByIdentifier[p.Identifier],
+			})
 		}
 
 		if err := batchSetProfileVariableAssociationsDB(ctx, tx, profilesVarsToUpsert, "darwin"); err != nil {
diff --git a/server/datastore/mysql/mdm.go b/server/datastore/mysql/mdm.go
index bdc7aac9c2c6..54db2b32c598 100644
--- a/server/datastore/mysql/mdm.go
+++ b/server/datastore/mysql/mdm.go
@@ -123,7 +123,7 @@ func (ds *Datastore) getMDMCommand(ctx context.Context, q sqlx.QueryerContext, c
 }
 
 func (ds *Datastore) BatchSetMDMProfiles(ctx context.Context, tmID *uint, macProfiles []*fleet.MDMAppleConfigProfile,
-	winProfiles []*fleet.MDMWindowsConfigProfile, macDeclarations []*fleet.MDMAppleDeclaration, profilesVariablesByIdentifier map[string]map[string]struct{},
+	winProfiles []*fleet.MDMWindowsConfigProfile, macDeclarations []*fleet.MDMAppleDeclaration, profilesVariablesByIdentifier []fleet.MDMProfileIdentifierFleetVariables,
 ) (updates fleet.MDMProfilesUpdates, err error) {
 	err = ds.withRetryTxx(ctx, func(tx sqlx.ExtContext) error {
 		var err error
@@ -131,7 +131,7 @@ func (ds *Datastore) BatchSetMDMProfiles(ctx context.Context, tmID *uint, macPro
 			return ctxerr.Wrap(ctx, err, "batch set windows profiles")
 		}
 
-		// for now, only apple profiles support variables
+		// for now, only apple profiles support Fleet variables
 		if updates.AppleConfigProfile, err = ds.batchSetMDMAppleProfilesDB(ctx, tx, tmID, macProfiles, profilesVariablesByIdentifier); err != nil {
 			return ctxerr.Wrap(ctx, err, "batch set apple profiles")
 		}
@@ -1506,7 +1506,7 @@ func (ds *Datastore) IsHostConnectedToFleetMDM(ctx context.Context, host *fleet.
 func batchSetProfileVariableAssociationsDB(
 	ctx context.Context,
 	tx sqlx.ExtContext,
-	profileVariablesByUUID map[string]map[string]struct{},
+	profileVariablesByUUID []fleet.MDMProfileUUIDFleetVariables,
 	platform string,
 ) error {
 	if len(profileVariablesByUUID) == 0 {
@@ -1525,8 +1525,13 @@ func batchSetProfileVariableAssociationsDB(
 
 	// collect the profile uuids to clear
 	profileUUIDsToDelete := make([]string, 0, len(profileVariablesByUUID))
-	for profileUUID := range profileVariablesByUUID {
-		profileUUIDsToDelete = append(profileUUIDsToDelete, profileUUID)
+	// small optimization - if there are no variables to insert, we can stop here
+	var varsToSet bool
+	for _, profVars := range profileVariablesByUUID {
+		profileUUIDsToDelete = append(profileUUIDsToDelete, profVars.ProfileUUID)
+		if len(profVars.FleetVariables) > 0 {
+			varsToSet = true
+		}
 	}
 
 	// delete variables associated with those profiles
@@ -1539,14 +1544,6 @@ func batchSetProfileVariableAssociationsDB(
 		return ctxerr.Wrap(ctx, err, "deleting variables for profiles")
 	}
 
-	// small optimization - if there are no variables to insert, we can stop here
-	var varsToSet bool
-	for _, vars := range profileVariablesByUUID {
-		if len(vars) > 0 {
-			varsToSet = true
-			break
-		}
-	}
 	if !varsToSet {
 		return nil
 	}
@@ -1572,18 +1569,18 @@ func batchSetProfileVariableAssociationsDB(
 		VarID       uint
 	}
 	profVars := make([]profVarTuple, 0, len(profileVariablesByUUID))
-	for profUUID, vars := range profileVariablesByUUID {
-		for v := range vars {
+	for _, pv := range profileVariablesByUUID {
+		for _, v := range pv.FleetVariables {
 			// variables received here do not have the FLEET_VAR_ prefix, but variables
 			// in the fleet_variables table do.
 			v = "FLEET_VAR_" + v
 			for _, def := range varDefs {
 				if !def.IsPrefix && def.Name == v {
-					profVars = append(profVars, profVarTuple{profUUID, def.ID})
+					profVars = append(profVars, profVarTuple{pv.ProfileUUID, def.ID})
 					break
 				}
 				if def.IsPrefix && strings.HasPrefix(v, def.Name) {
-					profVars = append(profVars, profVarTuple{profUUID, def.ID})
+					profVars = append(profVars, profVarTuple{pv.ProfileUUID, def.ID})
 					break
 				}
 			}
diff --git a/server/fleet/datastore.go b/server/fleet/datastore.go
index f57ddc229678..cf7c50630e5a 100644
--- a/server/fleet/datastore.go
+++ b/server/fleet/datastore.go
@@ -1082,7 +1082,7 @@ type Datastore interface {
 	// Apple MDM
 
 	// NewMDMAppleConfigProfile creates and returns a new configuration profile.
-	NewMDMAppleConfigProfile(ctx context.Context, p MDMAppleConfigProfile, usesFleetVars map[string]struct{}) (*MDMAppleConfigProfile, error)
+	NewMDMAppleConfigProfile(ctx context.Context, p MDMAppleConfigProfile, usesFleetVars []string) (*MDMAppleConfigProfile, error)
 
 	// BulkUpsertMDMAppleConfigProfiles inserts or updates a configuration
 	// profiles in bulk with the current payload.
@@ -1658,7 +1658,7 @@ type Datastore interface {
 	// BatchSetMDMProfiles sets the MDM Apple or Windows profiles for the given team or
 	// no team in a single transaction.
 	BatchSetMDMProfiles(ctx context.Context, tmID *uint, macProfiles []*MDMAppleConfigProfile, winProfiles []*MDMWindowsConfigProfile,
-		macDeclarations []*MDMAppleDeclaration, profilesVariables map[string]map[string]struct{}) (updates MDMProfilesUpdates, err error)
+		macDeclarations []*MDMAppleDeclaration, profilesVariables []MDMProfileIdentifierFleetVariables) (updates MDMProfilesUpdates, err error)
 
 	// NewMDMAppleDeclaration creates and returns a new MDM Apple declaration.
 	NewMDMAppleDeclaration(ctx context.Context, declaration *MDMAppleDeclaration) (*MDMAppleDeclaration, error)
diff --git a/server/fleet/mdm.go b/server/fleet/mdm.go
index f4509eb48beb..e7102b693cff 100644
--- a/server/fleet/mdm.go
+++ b/server/fleet/mdm.go
@@ -965,3 +965,26 @@ type HostMDMCommand struct {
 	HostID      uint   `db:"host_id"`
 	CommandType string `db:"command_type"`
 }
+
+// MDMProfileUUIDFleetVariables represents the Fleet variables used by a
+// profile identified by its UUID.
+type MDMProfileUUIDFleetVariables struct {
+	// ProfileUUID is the UUID of the profile.
+	ProfileUUID string
+	// FleetVariables is the (deduplicated) list of Fleet variables used by the
+	// profile, without the "FLEET_VAR_" prefix (as returned by
+	// findFleetVariables).
+	FleetVariables []string
+}
+
+// MDMProfileIdentifierFleetVariables represents the Fleet variables used by a
+// profile identified by its identifier.
+type MDMProfileIdentifierFleetVariables struct {
+	// Identifier is the identifier of the profile (which is unique by team for
+	// Apple profiles).
+	Identifier string
+	// FleetVariables is the (deduplicated) list of Fleet variables used by the
+	// profile, without the "FLEET_VAR_" prefix (as returned by
+	// findFleetVariables).
+	FleetVariables []string
+}
diff --git a/server/mock/datastore_mock.go b/server/mock/datastore_mock.go
index faab657d81f0..c14fe9b7ff5c 100644
--- a/server/mock/datastore_mock.go
+++ b/server/mock/datastore_mock.go
@@ -768,7 +768,7 @@ type UpdateVulnerabilityHostCountsFunc func(ctx context.Context, maxRoutines int
 
 type IsCVEKnownToFleetFunc func(ctx context.Context, cve string) (bool, error)
 
-type NewMDMAppleConfigProfileFunc func(ctx context.Context, p fleet.MDMAppleConfigProfile, usesFleetVars map[string]struct{}) (*fleet.MDMAppleConfigProfile, error)
+type NewMDMAppleConfigProfileFunc func(ctx context.Context, p fleet.MDMAppleConfigProfile, usesFleetVars []string) (*fleet.MDMAppleConfigProfile, error)
 
 type BulkUpsertMDMAppleConfigProfilesFunc func(ctx context.Context, payload []*fleet.MDMAppleConfigProfile) error
 
@@ -1074,7 +1074,7 @@ type NewMDMWindowsConfigProfileFunc func(ctx context.Context, cp fleet.MDMWindow
 
 type SetOrUpdateMDMWindowsConfigProfileFunc func(ctx context.Context, cp fleet.MDMWindowsConfigProfile) error
 
-type BatchSetMDMProfilesFunc func(ctx context.Context, tmID *uint, macProfiles []*fleet.MDMAppleConfigProfile, winProfiles []*fleet.MDMWindowsConfigProfile, macDeclarations []*fleet.MDMAppleDeclaration, profilesVariables map[string]map[string]struct{}) (updates fleet.MDMProfilesUpdates, err error)
+type BatchSetMDMProfilesFunc func(ctx context.Context, tmID *uint, macProfiles []*fleet.MDMAppleConfigProfile, winProfiles []*fleet.MDMWindowsConfigProfile, macDeclarations []*fleet.MDMAppleDeclaration, profilesVariables []fleet.MDMProfileIdentifierFleetVariables) (updates fleet.MDMProfilesUpdates, err error)
 
 type NewMDMAppleDeclarationFunc func(ctx context.Context, declaration *fleet.MDMAppleDeclaration) (*fleet.MDMAppleDeclaration, error)
 
@@ -3205,7 +3205,7 @@ type DataStore struct {
 	RenewMDMManagedCertificatesFunc        RenewMDMManagedCertificatesFunc
 	RenewMDMManagedCertificatesFuncInvoked bool
 
-	ListHostMDMManagedCertificatesFunc ListHostMDMManagedCertificatesFunc
+	ListHostMDMManagedCertificatesFunc        ListHostMDMManagedCertificatesFunc
 	ListHostMDMManagedCertificatesFuncInvoked bool
 
 	UpsertSecretVariablesFunc        UpsertSecretVariablesFunc
@@ -5927,7 +5927,7 @@ func (s *DataStore) IsCVEKnownToFleet(ctx context.Context, cve string) (bool, er
 	return s.IsCVEKnownToFleetFunc(ctx, cve)
 }
 
-func (s *DataStore) NewMDMAppleConfigProfile(ctx context.Context, p fleet.MDMAppleConfigProfile, usesFleetVars map[string]struct{}) (*fleet.MDMAppleConfigProfile, error) {
+func (s *DataStore) NewMDMAppleConfigProfile(ctx context.Context, p fleet.MDMAppleConfigProfile, usesFleetVars []string) (*fleet.MDMAppleConfigProfile, error) {
 	s.mu.Lock()
 	s.NewMDMAppleConfigProfileFuncInvoked = true
 	s.mu.Unlock()
@@ -6998,7 +6998,7 @@ func (s *DataStore) SetOrUpdateMDMWindowsConfigProfile(ctx context.Context, cp f
 	return s.SetOrUpdateMDMWindowsConfigProfileFunc(ctx, cp)
 }
 
-func (s *DataStore) BatchSetMDMProfiles(ctx context.Context, tmID *uint, macProfiles []*fleet.MDMAppleConfigProfile, winProfiles []*fleet.MDMWindowsConfigProfile, macDeclarations []*fleet.MDMAppleDeclaration, profilesVariables map[string]map[string]struct{}) (updates fleet.MDMProfilesUpdates, err error) {
+func (s *DataStore) BatchSetMDMProfiles(ctx context.Context, tmID *uint, macProfiles []*fleet.MDMAppleConfigProfile, winProfiles []*fleet.MDMWindowsConfigProfile, macDeclarations []*fleet.MDMAppleDeclaration, profilesVariables []fleet.MDMProfileIdentifierFleetVariables) (updates fleet.MDMProfilesUpdates, err error) {
 	s.mu.Lock()
 	s.BatchSetMDMProfilesFuncInvoked = true
 	s.mu.Unlock()
diff --git a/server/service/apple_mdm.go b/server/service/apple_mdm.go
index d4364e3b8d9b..78eca28de9cc 100644
--- a/server/service/apple_mdm.go
+++ b/server/service/apple_mdm.go
@@ -11,6 +11,7 @@ import (
 	"errors"
 	"fmt"
 	"io"
+	"maps"
 	"mime/multipart"
 	"net/http"
 	"net/url"
@@ -431,7 +432,7 @@ func (svc *Service) NewMDMAppleConfigProfile(ctx context.Context, teamID uint, r
 		// TODO what happens if mode is not set?s
 	}
 
-	newCP, err := svc.ds.NewMDMAppleConfigProfile(ctx, *cp, profileVars)
+	newCP, err := svc.ds.NewMDMAppleConfigProfile(ctx, *cp, slices.Collect(maps.Keys(profileVars)))
 	if err != nil {
 		var existsErr endpoint_utils.ExistsErrorInterface
 		if errors.As(err, &existsErr) {
diff --git a/server/service/mdm.go b/server/service/mdm.go
index 01dd102e6f81..ae2f6db52b6a 100644
--- a/server/service/mdm.go
+++ b/server/service/mdm.go
@@ -10,9 +10,11 @@ import (
 	"errors"
 	"fmt"
 	"io"
+	"maps"
 	"mime/multipart"
 	"net/http"
 	"path/filepath"
+	"slices"
 	"strconv"
 	"strings"
 	"time"
@@ -1718,10 +1720,17 @@ func (svc *Service) BatchSetMDMProfiles(
 		return nil
 	}
 
-	profilesVariablesByIdentifier, err := validateFleetVariables(ctx, appCfg, appleProfiles, windowsProfiles, appleDecls)
+	profilesVariablesByIdentifierMap, err := validateFleetVariables(ctx, appCfg, appleProfiles, windowsProfiles, appleDecls)
 	if err != nil {
 		return err
 	}
+	profilesVariablesByIdentifier := make([]fleet.MDMProfileIdentifierFleetVariables, 0, len(profilesVariablesByIdentifierMap))
+	for identifier, variables := range profilesVariablesByIdentifierMap {
+		profilesVariablesByIdentifier = append(profilesVariablesByIdentifier, fleet.MDMProfileIdentifierFleetVariables{
+			Identifier:     identifier,
+			FleetVariables: slices.Collect(maps.Keys(variables)),
+		})
+	}
 
 	// Now that validation is done, we remove the exposed secret variables from the profiles
 	appleProfilesSlice := make([]*fleet.MDMAppleConfigProfile, 0, len(appleProfiles))