Repository: score-spec/score-compose
Pull request #65 (2024-03-10 16:36:37)

feat: added generate subcommand and default provisioners with volume+redis
Depends on https://github.com/score-spec/score-compose/pull/66

This PR adds the `score generate` subcommand:

Demo here https://www.loom.com/share/8524876286ca40da96a077a9b1a56deb?sid=8512c9e3-0dab-4a8e-8695-0c5b0ed67d1b

```
$ score generate --help
The generate command will convert Score files in the current Score compose project into a combined Docker compose manifest. All resources and links between Workloads will be resolved and provisioned as required.

By default this command looks for score.yaml in the current directory, but can take explicit file names as positional arguments.

"score-compose init" MUST be run first. An error will be thrown if the project directory is not present.

Usage:
  score-compose generate [flags]

Examples:

  # Specify Score files
  score-compose generate score.yaml *.score.yaml

  # Regenerate without adding new score files
  score-compose generate

  # Provide overrides when one score file is provided
  score-compose generate score.yaml --override-file=./overrides.score.yaml --override-property=metadata.key=value

Flags:
  -h, --help                            help for generate
  -o, --output string                   The output file to write the composed compose file to (default "compose.yaml")
      --override-property stringArray   An optional set of path=key overrides to set or remove
      --overrides-file string           An optional file of Score overrides to merge in

Global Flags:
      --quiet           Mute any logging output
  -v, --verbose count   Increase log verbosity and detail by specifying this flag one or more times
```

This command generates a docker compose file from the current project state, optionally adding a new score file with any overrides or properties it may need.

This loads a default provisioners yaml file from the state directory which provides an initial set of resource providers. The default resource set only includes a volume and redis provider as an example, but this will be expanded to support more types in following PRs for project Melody.





mathieu-benoit: Maybe shortcut for:
- `--override-property` --> `-p`?
- `--overrides-file` --> `-f`?

Also, I don't know if it's just in the description of this PR, but it's `--override-property` versus `--overrides-file` (without versus with `s`), should we be consistent here?
astromechza: > Quick feedback here about `score-compose generate`, maybe shortcut for:
> 
>     * `--override-property` --> `-p`?
> 
>     * `--overrides-file` --> `-f`?
> 
> 
> Also, I don't know if it's just in the description of this PR, but it's `--override-property` versus `--overrides-file` (without versus with `s`), should we be consistent here?

@mathieu-benoit I was thinking about short codes for these, but I'm not a fan of `-f` and `-p` because they aren't obvious what they do at first glance. I recommend that we get it out first with the long-form version and then make it shorter if we find that customers are struggling with it or commands are getting "too long".

On the plurals, I think the existing form is more accurate and more gramatically correct, the file contains multiple _overrides_, and the property option just overrides a single property.

Patch:
diff --git a/internal/command/default.provisioners.yaml b/internal/command/default.provisioners.yaml
new file mode 100644
index 0000000..870af06
--- /dev/null
+++ b/internal/command/default.provisioners.yaml
@@ -0,0 +1,78 @@
+# The default volume provisioner provided by score-compose allows basic volume resources to be created in the resources
+# system. The volume resource just creates an ephemeral Docker volume with a random string as the name, and source
+# attribute that we can reference.
+- uri: template://default-provisioners/volume
+  # By default, match all classes and ids of volume. If you want to override this, create another provisioner definition
+  # with a higher priority.
+  type: volume
+  init: |
+    randomVolumeName: {{ .Id | replace "." "-" }}-{{ randAlphaNum 6 }}
+  # Store the random volume name if we haven't chosen one yet, otherwise use the one that exists already
+  state: |
+    name: {{ dig "name" .Init.randomVolumeName .State }}
+  # Return a source value with the volume name. This can be used in volume resource references now.
+  outputs: |
+    source: {{ .State.name }}
+  # Add a volume to the docker compose file. We assume our name is unique here. We also apply a label to help ensure
+  # that we can track the volume back to the workload and resource that created it.
+  volumes: |
+    {{ .State.name }}:
+      name: {{ .State.name }}
+      driver: local
+      labels:
+        dev.score.compose.res.uid: {{ .Uid }}
+
+# The default redis provisioner adds a redis service to the project which returns a host, port, username, and password.
+- uri: template://default-provisioners/redis
+  # By default, match all redis types regardless of class and id. If you want to override this, create another
+  # provisioner definition with a higher priority.
+  type: redis
+  # Init template has the default port and a random service name and password if needed later
+  init: |
+    port: 6379
+    randomServiceName: redis-{{ randAlphaNum 6 }}
+    randomPassword: {{ randAlphaNum 16 | quote }}
+  # The only state we need to persist is the chosen random service name and password
+  state: |
+    serviceName: {{ dig "serviceName" .Init.randomServiceName .State | quote }}
+    password: {{ dig "password" .Init.randomPassword .State | quote }}
+  # Return the outputs schema that consumers expect
+  outputs: |
+    host: {{ .State.serviceName }}
+    port: {{ .Init.port }}
+    username: default
+    password: {{ .State.password | quote }}
+  # write the config file to the mounts directory
+  files: |
+    {{ .State.serviceName }}/redis.conf: |
+      requirepass {{ .State.password }}
+      port {{ .Init.port }}
+      save 60 1
+      loglevel warning
+  # add a volume for persistence of the redis data
+  volumes: |
+    {{ .State.serviceName }}-data:
+      name: {{ .State.serviceName }}-data
+      driver: local
+      labels:
+        dev.score.compose.res.uid: {{ .Uid }}
+  # And the redis service itself with volumes bound in
+  services: |
+    {{ .State.serviceName }}:
+      labels:
+        dev.score.compose.res.uid: {{ .Uid }}
+      image: redis:7
+      entrypoint: ["redis-server"]
+      command: ["/usr/local/etc/redis/redis.conf"]
+      volumes:
+      - type: bind
+        source: {{ .MountsDirectory }}/{{ .State.serviceName }}/redis.conf
+        target: /usr/local/etc/redis/redis.conf
+        read_only: true
+      - type: volume
+        source: {{ .State.serviceName }}-data
+        target: /data
+        volume:
+          nocopy: true
+  info_logs: |
+    - "To connect to redis: \"docker run -it --network {{ .ComposeProjectName }}_default --rm redis redis-cli -h {{ .State.serviceName | squote }} -a {{ .State.password | squote }}\""
diff --git a/internal/command/generate.go b/internal/command/generate.go
new file mode 100644
index 0000000..de817e1
--- /dev/null
+++ b/internal/command/generate.go
@@ -0,0 +1,291 @@
+package command
+
+import (
+	"context"
+	"fmt"
+	"log/slog"
+	"os"
+	"slices"
+	"strings"
+
+	"github.com/compose-spec/compose-go/v2/types"
+	"github.com/imdario/mergo"
+	"github.com/score-spec/score-go/loader"
+	"github.com/score-spec/score-go/schema"
+	score "github.com/score-spec/score-go/types"
+	"github.com/spf13/cobra"
+	"gopkg.in/yaml.v3"
+
+	"github.com/score-spec/score-compose/internal/compose"
+	"github.com/score-spec/score-compose/internal/project"
+	"github.com/score-spec/score-compose/internal/provisioners"
+	"github.com/score-spec/score-compose/internal/provisioners/envprov"
+	provloader "github.com/score-spec/score-compose/internal/provisioners/loader"
+)
+
+const (
+	generateCmdOverridesFileFlag    = "overrides-file"
+	generateCmdOverridePropertyFlag = "override-property"
+)
+
+var generateCommand = &cobra.Command{
+	Use:   "generate",
+	Args:  cobra.ArbitraryArgs,
+	Short: "Convert one or more Score files into a Docker compose manifest",
+	Long: `The generate command will convert Score files in the current Score compose project into a combined Docker compose
+manifest. All resources and links between Workloads will be resolved and provisioned as required.
+
+By default this command looks for score.yaml in the current directory, but can take explicit file names as positional
+arguments.
+
+"score-compose init" MUST be run first. An error will be thrown if the project directory is not present.
+`,
+	Example: `
+  # Specify Score files
+  score-compose generate score.yaml *.score.yaml
+
+  # Regenerate without adding new score files
+  score-compose generate
+
+  # Provide overrides when one score file is provided
+  score-compose generate score.yaml --override-file=./overrides.score.yaml --override-property=metadata.key=value`,
+
+	// don't print the errors - we print these ourselves in main()
+	SilenceErrors: true,
+
+	RunE: func(cmd *cobra.Command, args []string) error {
+		cmd.SilenceUsage = true
+
+		inputFiles := args
+		slices.Sort(inputFiles)
+		slog.Debug("Input Score files", "files", inputFiles)
+
+		// first load all the score files, parse them with a dummy yaml decoder to find the workload name, reject any
+		// with invalid or duplicate names.
+		workloadNames, workloadSpecs, err := loadRawScoreFiles(inputFiles)
+		if err != nil {
+			return err
+		}
+		slog.Debug("Input Workload names", "names", workloadNames)
+
+		// Now read and apply any overrides files to the score files
+		if v, _ := cmd.Flags().GetString(generateCmdOverridesFileFlag); v != "" {
+			if len(workloadNames) == 0 {
+				return fmt.Errorf("--%s cannot be used without providing a score file", generateCmdOverridesFileFlag)
+			} else if len(workloadNames) > 1 {
+				return fmt.Errorf("--%s cannot be used when multiple score files are provided", generateCmdOverridesFileFlag)
+			}
+			if err := parseAndApplyOverrideFile(v, generateCmdOverridesFileFlag, workloadSpecs[workloadNames[0]]); err != nil {
+				return err
+			}
+		}
+
+		// Now read, parse, and apply any override properties to the score files
+		if v, _ := cmd.Flags().GetStringArray(generateCmdOverridePropertyFlag); len(v) > 0 {
+			if len(workloadNames) == 0 {
+				return fmt.Errorf("--%s cannot be used without providing a score file", generateCmdOverridesFileFlag)
+			} else if len(workloadNames) > 1 {
+				return fmt.Errorf("--%s cannot be used when multiple score files are provided", generateCmdOverridesFileFlag)
+			}
+			for _, overridePropertyEntry := range v {
+				if err := parseAndApplyOverrideProperty(overridePropertyEntry, generateCmdOverridePropertyFlag, workloadSpecs[workloadNames[0]]); err != nil {
+					return err
+				}
+			}
+		}
+
+		sd, ok, err := project.LoadStateDirectory(".")
+		if err != nil {
+			return fmt.Errorf("failed to load existing state directory: %w", err)
+		} else if !ok {
+			return fmt.Errorf("state directory does not exist, please run \"score-compose init\" first")
+		}
+		slog.Info(fmt.Sprintf("Loaded state directory with docker compose project '%s'", sd.State.ComposeProjectName))
+
+		currentState := &sd.State
+
+		// Now validate with score spec
+		for workloadName, spec := range workloadSpecs {
+			// Ensure transforms are applied (be a good citizen)
+			if changes, err := schema.ApplyCommonUpgradeTransforms(spec); err != nil {
+				return fmt.Errorf("failed to upgrade spec: %w", err)
+			} else if len(changes) > 0 {
+				for _, change := range changes {
+					slog.Info(fmt.Sprintf("Applying backwards compatible upgrade to '%s': %s", workloadName, change))
+				}
+			}
+			if err := schema.Validate(spec); err != nil {
+				return fmt.Errorf("validation errors in workload '%s': %w", workloadName, err)
+			}
+			slog.Info(fmt.Sprintf("Validated workload '%s'", workloadName))
+
+			var out score.Workload
+			if err := loader.MapSpec(&out, spec); err != nil {
+				return fmt.Errorf("failed to convert '%s' to structure: %w", workloadName, err)
+			}
+
+			currentState, err = currentState.WithWorkload(&out, nil)
+			if err != nil {
+				return fmt.Errorf("failed to add workload '%s': %w", workloadName, err)
+			}
+		}
+
+		if len(currentState.Workloads) == 0 {
+			return fmt.Errorf("the project is empty, please provide a score file to generate from")
+		}
+
+		loadedProvisioners, err := provloader.LoadProvisionersFromDirectory(sd.Path, provloader.DefaultSuffix)
+		if err != nil {
+			return fmt.Errorf("failed to load provisioners: %w", err)
+		} else if len(loadedProvisioners) > 0 {
+			slog.Info(fmt.Sprintf("Successfully loaded %d resource provisioners", len(loadedProvisioners)))
+		}
+
+		// append the env var provisioner
+		environmentProvisioner := new(envprov.Provisioner)
+		loadedProvisioners = append(loadedProvisioners, environmentProvisioner)
+
+		currentState, err = currentState.WithPrimedResources()
+		if err != nil {
+			return fmt.Errorf("failed to prime resources: %w", err)
+		}
+
+		superProject := &types.Project{
+			Name:     sd.State.ComposeProjectName,
+			Services: make(types.Services, 0),
+			Volumes:  map[string]types.VolumeConfig{},
+			Networks: map[string]types.NetworkConfig{},
+		}
+
+		currentState, err = provisioners.ProvisionResources(context.Background(), currentState, loadedProvisioners, superProject)
+		if err != nil {
+			return fmt.Errorf("failed to provision: %w", err)
+		} else if len(currentState.Resources) > 0 {
+			slog.Info(fmt.Sprintf("Provisioned %d resources", len(currentState.Resources)))
+		}
+
+		for workloadName, workloadState := range currentState.Workloads {
+			outputFunctions, err := currentState.GetResourceOutputForWorkload(workloadName)
+			if err != nil {
+				return err
+			}
+
+			slog.Info(fmt.Sprintf("Converting workload '%s' to Docker compose", workloadName))
+			converted, err := compose.ConvertSpec(&workloadState.Spec, outputFunctions)
+			if err != nil {
+				return fmt.Errorf("failed to convert workload '%s' to Docker compose: %w", workloadName, err)
+			}
+
+			for serviceName, service := range converted.Services {
+				if _, ok := superProject.Services[serviceName]; ok {
+					return fmt.Errorf("failed to add converted workload '%s': duplicate service name '%s'", workloadName, serviceName)
+				}
+				superProject.Services[serviceName] = service
+			}
+			for volumeName, volume := range converted.Volumes {
+				if _, ok := superProject.Volumes[volumeName]; ok {
+					return fmt.Errorf("failed to add converted workload '%s': duplicate volume name '%s'", workloadName, volumeName)
+				}
+				superProject.Volumes[volumeName] = volume
+			}
+			for networkName, network := range converted.Networks {
+				if _, ok := superProject.Networks[networkName]; ok {
+					return fmt.Errorf("failed to add converted workload '%s': duplicated network name '%s'", workloadName, networkName)
+				}
+				superProject.Networks[networkName] = network
+			}
+		}
+
+		sd.State = *currentState
+		if err := sd.Persist(); err != nil {
+			return fmt.Errorf("failed to persist updated state directory: %w", err)
+		}
+
+		raw, _ := yaml.Marshal(superProject)
+
+		v, _ := cmd.Flags().GetString("output")
+		if v == "" {
+			return fmt.Errorf("no output file specified")
+		} else if v == "-" {
+			_, _ = fmt.Fprint(cmd.OutOrStdout(), string(raw))
+		} else if err := os.WriteFile(v+".temp", raw, 0755); err != nil {
+			return fmt.Errorf("failed to write output file: %w", err)
+		} else if err := os.Rename(v+".temp", v); err != nil {
+			return fmt.Errorf("failed to complete writing output file: %w", err)
+		}
+		return nil
+	},
+}
+
+// loadRawScoreFiles loads raw score specs as yaml from the given files and finds all the workload names. It throws
+// errors if it failed to read, load, or if names are duplicated.
+func loadRawScoreFiles(fileNames []string) ([]string, map[string]map[string]interface{}, error) {
+	workloadNames := make([]string, 0, len(fileNames))
+	workloadToRawScore := make(map[string]map[string]interface{}, len(fileNames))
+
+	for _, fileName := range fileNames {
+		var out map[string]interface{}
+		raw, err := os.ReadFile(fileName)
+		if err != nil {
+			return nil, nil, fmt.Errorf("failed to read '%s': %w", fileName, err)
+		} else if err := yaml.Unmarshal(raw, &out); err != nil {
+			return nil, nil, fmt.Errorf("failed to decode '%s' as yaml: %w", fileName, err)
+		}
+
+		var workloadName string
+		if meta, ok := out["metadata"].(map[string]interface{}); ok {
+			workloadName, _ = meta["name"].(string)
+			if _, ok := workloadToRawScore[workloadName]; ok {
+				return nil, nil, fmt.Errorf("workload name '%s' in file '%s' is used more than once", workloadName, fileName)
+			}
+		}
+		workloadNames = append(workloadNames, workloadName)
+		workloadToRawScore[workloadName] = out
+	}
+	return workloadNames, workloadToRawScore, nil
+}
+
+func init() {
+	generateCommand.Flags().StringP("output", "o", "compose.yaml", "The output file to write the composed compose file to")
+	generateCommand.Flags().String(generateCmdOverridesFileFlag, "", "An optional file of Score overrides to merge in")
+	generateCommand.Flags().StringArray(generateCmdOverridePropertyFlag, []string{}, "An optional set of path=key overrides to set or remove")
+	rootCmd.AddCommand(generateCommand)
+}
+
+func parseAndApplyOverrideFile(entry string, flagName string, spec map[string]interface{}) error {
+	if raw, err := os.ReadFile(entry); err != nil {
+		return fmt.Errorf("--%s '%s' is invalid, failed to read file: %w", flagName, entry, err)
+	} else {
+		slog.Info(fmt.Sprintf("Applying overrides from %s to workload", entry))
+		var out map[string]interface{}
+		if err := yaml.Unmarshal(raw, &out); err != nil {
+			return fmt.Errorf("--%s '%s' is invalid: failed to decode yaml: %w", flagName, entry, err)
+		} else if err := mergo.Merge(&spec, out, mergo.WithOverride); err != nil {
+			return fmt.Errorf("--%s '%s' failed to apply: %w", flagName, entry, err)
+		}
+	}
+	return nil
+}
+
+func parseAndApplyOverrideProperty(entry string, flagName string, spec map[string]interface{}) error {
+	parts := strings.SplitN(entry, "=", 2)
+	if len(parts) != 2 {
+		return fmt.Errorf("--%s '%s' is invalid, expected a =-separated path and value", flagName, entry)
+	}
+	if parts[1] == "" {
+		slog.Info(fmt.Sprintf("Overriding '%s' in workload", parts[0]))
+		if err := writePathInStruct(spec, parseDotPathParts(parts[0]), true, nil); err != nil {
+			return fmt.Errorf("--%s '%s' could not be applied: %w", flagName, entry, err)
+		}
+	} else {
+		var value interface{}
+		if err := yaml.Unmarshal([]byte(parts[1]), &value); err != nil {
+			return fmt.Errorf("--%s '%s' is invalid, failed to unmarshal value as json: %w", flagName, entry, err)
+		}
+		slog.Info(fmt.Sprintf("Overriding '%s' in workload", parts[0]))
+		if err := writePathInStruct(spec, parseDotPathParts(parts[0]), false, value); err != nil {
+			return fmt.Errorf("--%s '%s' could not be applied: %w", flagName, entry, err)
+		}
+	}
+	return nil
+}
diff --git a/internal/command/init.go b/internal/command/init.go
index 12353c5..526c5b5 100644
--- a/internal/command/init.go
+++ b/internal/command/init.go
@@ -1,6 +1,7 @@
 package command
 
 import (
+	_ "embed"
 	"errors"
 	"fmt"
 	"log/slog"
@@ -11,6 +12,7 @@ import (
 	"github.com/spf13/cobra"
 
 	"github.com/score-spec/score-compose/internal/project"
+	"github.com/score-spec/score-compose/internal/provisioners/loader"
 )
 
 const DefaultScoreFileContent = `# Score provides a developer-centric and platform-agnostic 
@@ -50,6 +52,9 @@ service:
 resources: {}
 `
 
+//go:embed default.provisioners.yaml
+var defaultProvisionersContent string
+
 var initCmd = &cobra.Command{
 	Use:   "init",
 	Args:  cobra.NoArgs,
@@ -103,7 +108,7 @@ acts as a namespace when multiple score files and containers are used.
 
 			slog.Info(fmt.Sprintf("Writing new state directory '%s'", project.DefaultRelativeStateDirectory))
 			wd, _ := os.Getwd()
-			sd := &project.StateDirectory{
+			sd = &project.StateDirectory{
 				Path: project.DefaultRelativeStateDirectory,
 				State: project.State{
 					Workloads:          map[string]project.ScoreWorkloadState{},
@@ -120,6 +125,12 @@ acts as a namespace when multiple score files and containers are used.
 			if err := sd.Persist(); err != nil {
 				return fmt.Errorf("failed to persist new compose project name: %w", err)
 			}
+
+			dst := "99-default" + loader.DefaultSuffix
+			slog.Info(fmt.Sprintf("Writing default provisioners yaml file '%s'", dst))
+			if err := os.WriteFile(filepath.Join(sd.Path, dst), []byte(defaultProvisionersContent), 0644); err != nil {
+				return fmt.Errorf("failed to write provisioners: %w", err)
+			}
 		}
 
 		if _, err := os.ReadFile(initCmdScoreFile); err != nil {
@@ -135,6 +146,13 @@ acts as a namespace when multiple score files and containers are used.
 		} else {
 			slog.Info(fmt.Sprintf("Found existing Score file '%s'", initCmdScoreFile))
 		}
+
+		if provs, err := loader.LoadProvisionersFromDirectory(sd.Path, loader.DefaultSuffix); err != nil {
+			return fmt.Errorf("failed to load existing provisioners: %w", err)
+		} else {
+			slog.Debug(fmt.Sprintf("Successfully loaded %d resource provisioners", len(provs)))
+		}
+
 		slog.Info(fmt.Sprintf("Read more about the Score specification at https://docs.score.dev/docs/"))
 
 		return nil
diff --git a/internal/command/override_utils.go b/internal/command/override_utils.go
new file mode 100644
index 0000000..fd832d1
--- /dev/null
+++ b/internal/command/override_utils.go
@@ -0,0 +1,106 @@
+package command
+
+import (
+	"fmt"
+	"slices"
+	"strconv"
+	"strings"
+)
+
+func parseDotPathParts(input string) []string {
+	// support escaping dot's to insert elements with a . in them.
+	input = strings.ReplaceAll(input, "\\\\", "\x01")
+	input = strings.ReplaceAll(input, "\\.", "\x00")
+	parts := strings.Split(input, ".")
+	for i, part := range parts {
+		part = strings.ReplaceAll(part, "\x00", ".")
+		part = strings.ReplaceAll(part, "\x01", "\\")
+		parts[i] = part
+	}
+	return parts
+}
+
+func writePathInStruct(input map[string]interface{}, path []string, isDelete bool, value interface{}) error {
+	if len(path) == 0 {
+		return fmt.Errorf("cannot change root node")
+	}
+
+	// the current position in the tree
+	var current interface{} = input
+
+	// a reference to the map that holds current
+	var parentMap map[string]interface{}
+	var parentKey string
+
+	// first traverse to the right location
+	for _, s := range path[:len(path)-1] {
+		switch currentType := current.(type) {
+		case map[string]interface{}:
+			parentMap = currentType
+			parentKey = s
+
+			next, ok := currentType[s]
+			if ok {
+				current = next
+			} else {
+				currentType[s] = make(map[string]interface{})
+				current = currentType[s]
+			}
+		case []interface{}:
+			parentMap = nil
+
+			idx, err := strconv.Atoi(s)
+			if err != nil {
+				return fmt.Errorf("cannot index '%s' in array", s)
+			} else if idx < 0 || idx >= len(currentType) {
+				return fmt.Errorf("cannot set '%s' in array: out of range", s)
+			}
+			current = currentType[idx]
+		default:
+			return fmt.Errorf("cannot lookup property or index '%s' in %T", s, currentType)
+		}
+	}
+	// then apply the change
+
+	key := path[len(path)-1]
+	switch currentType := current.(type) {
+	case map[string]interface{}:
+		if isDelete {
+			delete(currentType, key)
+		} else {
+			currentType[key] = value
+		}
+	case []interface{}:
+		// This is where the bulk of the complexity comes from. Parsing validating and then navigating the slices.
+		idx, err := strconv.Atoi(key)
+		if err != nil {
+			return fmt.Errorf("cannot index '%s' in array", key)
+		} else if idx < -1 || idx >= len(currentType) {
+			return fmt.Errorf("cannot set '%s' in array: out of range", key)
+		} else if isDelete {
+			if idx == -1 {
+				return fmt.Errorf("cannot delete '%s' in array", key)
+			} else {
+				if parentMap != nil {
+					parentMap[parentKey] = slices.Delete(currentType, idx, idx+1)
+				} else {
+					return fmt.Errorf("override in nested arrays is not supported")
+				}
+			}
+		} else {
+			if idx == -1 {
+				if parentMap != nil {
+					parentMap[parentKey] = append(currentType, value)
+				} else {
+					return fmt.Errorf("override in nested arrays is not supported")
+				}
+			} else {
+				currentType[idx] = value
+			}
+		}
+	default:
+		return fmt.Errorf("cannot lookup property or index '%s' in %T", key, currentType)
+	}
+
+	return nil
+}


Test patch:
diff --git a/internal/command/generate_test.go b/internal/command/generate_test.go
new file mode 100644
index 0000000..a556ab0
--- /dev/null
+++ b/internal/command/generate_test.go
@@ -0,0 +1,157 @@
+package command
+
+import (
+	"context"
+	"os"
+	"path/filepath"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/score-spec/score-compose/internal/project"
+)
+
+func TestGenerateHelp(t *testing.T) {
+	stdout, stderr, err := executeAndResetCommand(context.Background(), rootCmd, []string{"generate", "--help"})
+	assert.NoError(t, err)
+	assert.Equal(t, `The generate command will convert Score files in the current Score compose project into a combined Docker compose
+manifest. All resources and links between Workloads will be resolved and provisioned as required.
+
+By default this command looks for score.yaml in the current directory, but can take explicit file names as positional
+arguments.
+
+"score-compose init" MUST be run first. An error will be thrown if the project directory is not present.
+
+Usage:
+  score-compose generate [flags]
+
+Examples:
+
+  # Specify Score files
+  score-compose generate score.yaml *.score.yaml
+
+  # Regenerate without adding new score files
+  score-compose generate
+
+  # Provide overrides when one score file is provided
+  score-compose generate score.yaml --override-file=./overrides.score.yaml --override-property=metadata.key=value
+
+Flags:
+  -h, --help                            help for generate
+  -o, --output string                   The output file to write the composed compose file to (default "compose.yaml")
+      --override-property stringArray   An optional set of path=key overrides to set or remove
+      --overrides-file string           An optional file of Score overrides to merge in
+
+Global Flags:
+      --quiet           Mute any logging output
+  -v, --verbose count   Increase log verbosity and detail by specifying this flag one or more times
+`, stdout)
+	assert.Equal(t, "", stderr)
+
+	stdout2, stderr, err := executeAndResetCommand(context.Background(), rootCmd, []string{"help", "generate"})
+	assert.NoError(t, err)
+	assert.Equal(t, stdout, stdout2)
+	assert.Equal(t, "", stderr)
+}
+
+func changeToTempDir(t *testing.T) string {
+	t.Helper()
+	td := t.TempDir()
+	wd, _ := os.Getwd()
+	require.NoError(t, os.Chdir(td))
+	t.Cleanup(func() {
+		require.NoError(t, os.Chdir(wd))
+	})
+	return td
+}
+
+func TestGenerateWithoutInit(t *testing.T) {
+	_ = changeToTempDir(t)
+	stdout, _, err := executeAndResetCommand(context.Background(), rootCmd, []string{"generate"})
+	assert.EqualError(t, err, "state directory does not exist, please run \"score-compose init\" first")
+	assert.Equal(t, "", stdout)
+}
+
+func TestGenerateWithoutScoreFiles(t *testing.T) {
+	_ = changeToTempDir(t)
+	stdout, _, err := executeAndResetCommand(context.Background(), rootCmd, []string{"init"})
+	assert.NoError(t, err)
+	assert.Equal(t, "", stdout)
+	stdout, _, err = executeAndResetCommand(context.Background(), rootCmd, []string{"generate"})
+	assert.EqualError(t, err, "the project is empty, please provide a score file to generate from")
+	assert.Equal(t, "", stdout)
+}
+
+func TestInitAndGenerateWithBadFile(t *testing.T) {
+	td := changeToTempDir(t)
+	stdout, _, err := executeAndResetCommand(context.Background(), rootCmd, []string{"init"})
+	assert.NoError(t, err)
+	assert.Equal(t, "", stdout)
+
+	assert.NoError(t, os.WriteFile(filepath.Join(td, "thing"), []byte(`"blah"`), 0644))
+
+	stdout, _, err = executeAndResetCommand(context.Background(), rootCmd, []string{"generate", "thing"})
+	assert.EqualError(t, err, "failed to decode 'thing' as yaml: yaml: unmarshal errors:\n  line 1: cannot unmarshal !!str `blah` into map[string]interface {}")
+	assert.Equal(t, "", stdout)
+}
+
+func TestInitAndGenerateWithBadScore(t *testing.T) {
+	td := changeToTempDir(t)
+	stdout, _, err := executeAndResetCommand(context.Background(), rootCmd, []string{"init"})
+	assert.NoError(t, err)
+	assert.Equal(t, "", stdout)
+
+	assert.NoError(t, os.WriteFile(filepath.Join(td, "thing"), []byte(`{}`), 0644))
+
+	stdout, _, err = executeAndResetCommand(context.Background(), rootCmd, []string{"generate", "thing"})
+	assert.EqualError(t, err, "validation errors in workload '': jsonschema: '' does not validate with https://score.dev/schemas/score#/required: missing properties: 'apiVersion', 'metadata', 'containers'")
+	assert.Equal(t, "", stdout)
+}
+
+func TestInitAndGenerate_with_sample(t *testing.T) {
+	td := changeToTempDir(t)
+	stdout, _, err := executeAndResetCommand(context.Background(), rootCmd, []string{"init"})
+	assert.NoError(t, err)
+	assert.Equal(t, "", stdout)
+
+	// write overrides file
+	assert.NoError(t, os.WriteFile(filepath.Join(td, "overrides.yaml"), []byte(`{"resources": {"foo": {"type": "environment"}}}`), 0644))
+	// generate
+	stdout, _, err = executeAndResetCommand(context.Background(), rootCmd, []string{
+		"generate", "-o", "compose-output.yaml",
+		"--overrides-file", "overrides.yaml",
+		"--override-property", "containers.hello-world.variables.THING=${resources.foo.THING}",
+		"--", "score.yaml",
+	})
+	assert.NoError(t, err)
+	assert.Equal(t, "", stdout)
+	raw, err := os.ReadFile(filepath.Join(td, "compose-output.yaml"))
+	assert.NoError(t, err)
+	expectedOutput := `name: "001"
+services:
+    example-hello-world:
+        environment:
+            EXAMPLE_VARIABLE: example-value
+            THING: ${THING}
+        image: nginx:latest
+        ports:
+            - target: 80
+              published: "8080"
+`
+	assert.Equal(t, expectedOutput, string(raw))
+	// generate again just for luck
+	stdout, _, err = executeAndResetCommand(context.Background(), rootCmd, []string{"generate", "-o", "compose-output.yaml"})
+	assert.NoError(t, err)
+	assert.Equal(t, "", stdout)
+	raw, err = os.ReadFile(filepath.Join(td, "compose-output.yaml"))
+	assert.NoError(t, err)
+	assert.Equal(t, expectedOutput, string(raw))
+
+	// check that state was persisted
+	sd, ok, err := project.LoadStateDirectory(td)
+	assert.NoError(t, err)
+	assert.True(t, ok)
+	assert.Len(t, sd.State.Workloads, 1)
+	assert.Len(t, sd.State.Resources, 1)
+}
diff --git a/internal/command/override_utils_test.go b/internal/command/override_utils_test.go
new file mode 100644
index 0000000..c89109f
--- /dev/null
+++ b/internal/command/override_utils_test.go
@@ -0,0 +1,117 @@
+package command
+
+import (
+	"encoding/json"
+	"fmt"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestParseDotPathParts(t *testing.T) {
+	for _, tc := range []struct {
+		Input    string
+		Expected []string
+	}{
+		{"", []string{""}},
+		{"a", []string{"a"}},
+		{"a.b", []string{"a", "b"}},
+		{"a.-1", []string{"a", "-1"}},
+		{"a.b\\.c", []string{"a", "b.c"}},
+		{"a.b\\\\.c", []string{"a", "b\\", "c"}},
+	} {
+		t.Run(tc.Input, func(t *testing.T) {
+			assert.Equal(t, tc.Expected, parseDotPathParts(tc.Input))
+		})
+	}
+}
+
+func TestWritePathInStruct(t *testing.T) {
+	for _, tc := range []struct {
+		Name          string
+		Spec          string
+		Path          []string
+		Delete        bool
+		Value         interface{}
+		Expected      string
+		ExpectedError error
+	}{
+		{
+			Name:     "simple object set",
+			Spec:     `{"a":{"b":[{}]}}`,
+			Path:     []string{"a", "b", "0", "c"},
+			Value:    "hello",
+			Expected: `{"a":{"b":[{"c":"hello"}]}}`,
+		},
+		{
+			Name:     "simple object delete",
+			Spec:     `{"a":{"b":[{"c":"hello"}]}}`,
+			Path:     []string{"a", "b", "0", "c"},
+			Delete:   true,
+			Expected: `{"a":{"b":[{}]}}`,
+		},
+		{
+			Name:     "simple array set",
+			Spec:     `{"a":[{}]}`,
+			Path:     []string{"a", "0"},
+			Value:    "hello",
+			Expected: `{"a":["hello"]}`,
+		},
+		{
+			Name:     "simple array append",
+			Spec:     `{"a":["hello"]}`,
+			Path:     []string{"a", "-1"},
+			Value:    "world",
+			Expected: `{"a":["hello","world"]}`,
+		},
+		{
+			Name:     "simple array delete",
+			Spec:     `{"a":["hello", "world"]}`,
+			Path:     []string{"a", "0"},
+			Delete:   true,
+			Expected: `{"a":["world"]}`,
+		},
+		{
+			Name:     "build object via path",
+			Spec:     `{}`,
+			Path:     []string{"a", "b"},
+			Value:    "hello",
+			Expected: `{"a":{"b":"hello"}}`,
+		},
+		{
+			Name:          "bad index str",
+			Spec:          `{"a":[]}`,
+			Path:          []string{"a", "b"},
+			Value:         "hello",
+			ExpectedError: fmt.Errorf("cannot index 'b' in array"),
+		},
+		{
+			Name:          "index out of range",
+			Spec:          `{"a": [0]}`,
+			Path:          []string{"a", "2"},
+			Value:         "hello",
+			ExpectedError: fmt.Errorf("cannot set '2' in array: out of range"),
+		},
+		{
+			Name:          "no append nested arrays",
+			Spec:          `{"a":[[0]]}`,
+			Path:          []string{"a", "0", "-1"},
+			Value:         "hello",
+			ExpectedError: fmt.Errorf("override in nested arrays is not supported"),
+		},
+	} {
+		t.Run(tc.Name, func(t *testing.T) {
+			var inSpec map[string]interface{}
+			assert.NoError(t, json.Unmarshal([]byte(tc.Spec), &inSpec))
+			err := writePathInStruct(inSpec, tc.Path, tc.Delete, tc.Value)
+			if tc.ExpectedError != nil {
+				assert.EqualError(t, err, tc.ExpectedError.Error())
+			} else {
+				if assert.NoError(t, err) {
+					outSpec, _ := json.Marshal(inSpec)
+					assert.JSONEq(t, tc.Expected, string(outSpec))
+				}
+			}
+		})
+	}
+}
diff --git a/internal/command/root_test.go b/internal/command/root_test.go
index 9d226d3..01cec4d 100644
--- a/internal/command/root_test.go
+++ b/internal/command/root_test.go
@@ -20,6 +20,7 @@ Usage:
 
 Available Commands:
   completion  Generate the autocompletion script for the specified shell
+  generate    Convert one or more Score files into a Docker compose manifest
   help        Help about any command
   init        Initialise a new score-compose project with local state directory and score file
   run         Translate the SCORE file to docker-compose configuration
diff --git a/internal/command/run_test.go b/internal/command/run_test.go
index b2e423b..8c5b79f 100644
--- a/internal/command/run_test.go
+++ b/internal/command/run_test.go
@@ -42,7 +42,11 @@ func executeAndResetCommand(ctx context.Context, cmd *cobra.Command, args []stri
 		subCmd.SetContext(nil)
 		subCmd.SilenceUsage = false
 		subCmd.Flags().VisitAll(func(f *pflag.Flag) {
-			_ = f.Value.Set(f.DefValue)
+			if f.Value.Type() == "stringArray" {
+				_ = f.Value.(pflag.SliceValue).Replace(nil)
+			} else {
+				_ = f.Value.Set(f.DefValue)
+			}
 		})
 	}
 	return nowOut.String(), nowErr.String(), err