Repository: score-spec/score-compose
Pull request #172 (2024-07-19 21:35:46)

feat(generate): added --publish port CLI flag
This removes the need for some of the weird workarounds used in local development to publish private ports from workloads or resources that are usually only exposed to services within the docker network.

Now to access a workload port that doesn't have a dns+route to expose it, you can now use `--publish <host port>:<workload name>:<container port>` to add a port publish to the output compose.yaml file. This does not require that the port is set as a Score service port, therefore this can be used for things like debugging ports. For example, `--publish 8080:my-workload:80` would publish a HTTP port to port 8080.

Now to access a resource port like a postgres, rabbitmq, or other resource that wouldn't be published externally you can use `--publish <host port>:<resource uid>.output:<container port>`. The expression in the middle allows you to select a resource service that usually has a dynamic hostname. For example `--publish 5432:postgres#my-workload.db.host:5432` would expose the database `db` from workload `my-workload`.

Note, that these publish calls must be on the LAST request to `score-compose generate` since they apply to the final compose manifest.

I think this is better than the existing workarounds for doing this like:

1. Putting compose.score.dev/publish-port annotations in your score file
2. Using jq to examine and modify the compose file in place
3. Adding additional `socat` services to the compose.yaml



Patch:
diff --git a/README.md b/README.md
index 5ccb835..77cfc76 100644
--- a/README.md
+++ b/README.md
@@ -111,6 +111,12 @@ Examples:
   # Provide overrides when one score file is provided
   score-compose generate score.yaml --override-file=./overrides.score.yaml --override-property=metadata.key=value
 
+  # Publish a port exposed by a workload for local testing
+  score-compose generate score.yaml --publish 8080:my-workload:80
+
+  # Publish a port from a resource host and port for local testing, the middle expression is RESOURCE_ID.OUTPUT_KEY
+  score-compose generate score.yaml --publish 5432:postgres#my-workload.db.host:5432
+
 Flags:
       --build stringArray               An optional build context to use for the given container --build=container=./dir or --build=container={"context":"./dir"}
       --env-file string                 Location to store a skeleton .env file for compose - this will override existing content
@@ -119,6 +125,7 @@ Flags:
   -o, --output string                   The output file to write the composed compose file to (default "compose.yaml")
       --override-property stringArray   An optional set of path=key overrides to set or remove
       --overrides-file string           An optional file of Score overrides to merge in
+      --publish stringArray             An optional set of HOST_PORT:<ref>:CONTAINER_PORT to publish on the host system.
 
 Global Flags:
       --quiet           Mute any logging output
@@ -127,6 +134,19 @@ Global Flags:
 
 **NOTE**: The `score-compose run` command still exists but is hidden and should be considered deprecated as it does not support resource provisioning.
 
+### Using the `--publish` flag
+
+`score-compose` installs all workloads and resource services into the compose docker network but does not publish ports on the host by default. To access ports inside the network, the user must either exec into a target container, run a new `socat` container with published ports, use a `route` resource that publishes "public" ports, or modify the compose.yaml directly.
+
+The `--publish` flag on the `generate` command can be used to automatically add published ports to the services in the compose.yaml file in a safe and dynamic way. Note that this is an _ephemeral_ flag not stored in the state file. As such it should be added to the _last_ invocation of `score-compose generate` to avoid it being overridden by subsequent calls.
+
+- To publish a container port from a workload to the host, use `--publish HOST_PORT:<workload>:CONTAINER_PORT`.
+- To publish a a port from one of the resource services, use `--publish HOST_PORT:<resource uid>.<output key>:CONTAINER_PORT`. Examples of this are:
+  - `15432:postgres#my-workload.res.host:5432` - `host` is the output on the `postgres.default#my-workload.res` Postgres resource that contains the service hostname.
+  - `9001:s3.default#storage.service:9001` - `service` is the service name output on the `s3.default#storage` S3 resource.
+
+This deprecates the use of the `compose.score.dev/publish-port` resource metadata annotation in most cases. 
+
 ## Testing
 
 Run the tests using `go test -v ./... -race`. If you do not have `docker` CLI installed locally or want the tests to run
diff --git a/examples/06-resource-provisioning/README.md b/examples/06-resource-provisioning/README.md
index efde7f1..df50f5e 100644
--- a/examples/06-resource-provisioning/README.md
+++ b/examples/06-resource-provisioning/README.md
@@ -112,6 +112,22 @@ This makes it much easier to use the outputs in your integration tests and CI pi
 
 **NOTE**: experts may wish to look at the `.score-compose/state.yaml` file directly.
 
+## Publishing resource ports
+
+You can use the `--publish` flag on the `generate` command to expose the publish the redis port on the docker host to access it from external tests, web browsers, or IDEs. In this case the resource uid for the `main-cache` is `redis.default#main-cache` and it has a `host` output. So the publish command may look like:
+
+```
+$ score-compose generate score.yaml score2.yaml --publish '6379:redis.default#main-cache.host:6379'
+```
+
+This adds the following section to the dynamic redis service:
+
+```yaml
+ports:
+  - target: 6379
+    published: "6379"
+```
+
 ## The `*.provisioners.yaml` files
 
 When you run `score-compose init`, a [99-default.provisioners.yaml](https://github.com/score-spec/score-compose/blob/main/internal/command/default.provisioners.yaml) file is created, which is a YAML file holding the definition of the built-in provisioners.
diff --git a/examples/08-service-port-resource/README.md b/examples/08-service-port-resource/README.md
index 5e58b78..17c026d 100644
--- a/examples/08-service-port-resource/README.md
+++ b/examples/08-service-port-resource/README.md
@@ -72,3 +72,17 @@ services:
 And when run, the logs show `workload-b` requesting the index page from the nginx server every 5 seconds.
 
 **NOTE**: no dependency relationship is created between the workloads, because Score assumes these workloads may start or restart in any order. Like all good software, the services should be implemented in a way that allows them to start up without the dependency being immediately available. In this case we use `|| true` in the wget statement to ensure `workload-b` retries the request.
+
+## Accessing workload ports
+
+Normally, the ports exposed by a workload, including the `service` ports are not accessible on the docker host. However the `--publish` flag exists to allow this.
+
+The flag can take the form `HOST_PORT:<workload name>:CONTAINER_PORT`, where the container port can be _any_ port on the container you wish to map to the host. This skips any service port mappings.
+
+In the example above, we can publish the `workload-a` port, by calling `generate` with `--publish 8080:workload-a:80`. This modifies the output `compose.yaml` file with an addition ports section:
+
+```yaml
+ports:
+  - target: 80
+    published: "8080"
+```
diff --git a/internal/command/generate.go b/internal/command/generate.go
index 951eedd..3722814 100644
--- a/internal/command/generate.go
+++ b/internal/command/generate.go
@@ -20,6 +20,7 @@ import (
 	"log/slog"
 	"os"
 	"slices"
+	"strconv"
 	"strings"
 
 	composeloader "github.com/compose-spec/compose-go/v2/loader"
@@ -46,6 +47,7 @@ const (
 	generateCmdBuildFlag            = "build"
 	generateCmdOutputFlag           = "output"
 	generateCmdEnvFileFlag          = "env-file"
+	generateCmdPublishFlag          = "publish"
 )
 
 var generateCommand = &cobra.Command{
@@ -59,6 +61,7 @@ By default this command looks for score.yaml in the current directory, but can t
 arguments.
 
 "score-compose init" MUST be run first. An error will be thrown if the project directory is not present.
+
 `,
 	Example: `
   # Specify Score files
@@ -68,7 +71,13 @@ arguments.
   score-compose generate
 
   # Provide overrides when one score file is provided
-  score-compose generate score.yaml --override-file=./overrides.score.yaml --override-property=metadata.key=value`,
+  score-compose generate score.yaml --override-file=./overrides.score.yaml --override-property=metadata.key=value
+
+  # Publish a port exposed by a workload for local testing
+  score-compose generate score.yaml --publish 8080:my-workload:80
+
+  # Publish a port from a resource host and port for local testing, the middle expression is RESOURCE_ID.OUTPUT_KEY
+  score-compose generate score.yaml --publish 5432:postgres#my-workload.db.host:5432`,
 
 	// don't print the errors - we print these ourselves in main()
 	SilenceErrors: true,
@@ -262,6 +271,80 @@ arguments.
 			}
 		}
 
+		if v, err := cmd.Flags().GetStringArray(generateCmdPublishFlag); err != nil {
+			return fmt.Errorf("failed to read publish property: %w", err)
+		} else {
+			for i, k := range v {
+				parts := strings.Split(k, ":")
+				if len(parts) <= 2 {
+					return fmt.Errorf("--%s[%d] expected 3 :-separated parts", generateCmdPublishFlag, i)
+				}
+				// raw host port
+				rhp := parts[0]
+				// raw ref
+				rr := strings.Join(parts[1:len(parts)-1], ":")
+				// raw container port
+				rcp := parts[len(parts)-1]
+
+				hp, err := strconv.Atoi(rhp)
+				if err != nil {
+					return fmt.Errorf("--%s[%d] could not parse host port '%s' as integer", generateCmdPublishFlag, i, rhp)
+				} else if hp <= 1 {
+					return fmt.Errorf("--%s[%d] host port must be > 1", generateCmdPublishFlag, i)
+				}
+
+				cp, err := strconv.Atoi(rcp)
+				if err != nil {
+					return fmt.Errorf("--%s[%d] could not parse container port '%s' as integer", generateCmdPublishFlag, i, rcp)
+				} else if cp <= 1 {
+					return fmt.Errorf("--%s[%d] container port must be > 1", generateCmdPublishFlag, i)
+				}
+
+				if strings.Contains(rr, "#") {
+					parts := strings.Split(rr, ".")
+					if len(parts) < 2 {
+						return fmt.Errorf("--%s[%d] must match RES_UID.OUTPUT", generateCmdPublishFlag, i)
+					}
+					rr = strings.Join(parts[0:len(parts)-1], ".")
+					outputKey := parts[len(parts)-1]
+
+					resUid := parseResourceUid(rr)
+					res, ok := currentState.Resources[resUid]
+					if !ok {
+						return fmt.Errorf("--%s[%d] failed to find a resource with uid '%s'", generateCmdPublishFlag, i, rr)
+					}
+
+					if v, ok := res.Outputs[outputKey]; !ok {
+						return fmt.Errorf("--%s[%d] resource '%s' has no output '%s'", generateCmdPublishFlag, i, resUid, outputKey)
+					} else if sv, ok := v.(string); !ok {
+						return fmt.Errorf("--%s[%d] resource '%s' output '%s' is not a string", generateCmdPublishFlag, i, resUid, outputKey)
+					} else if config, ok := superProject.Services[sv]; !ok {
+						return fmt.Errorf("--%s[%d] host '%s' does not exist", generateCmdPublishFlag, i, sv)
+					} else {
+						config.Ports = append(config.Ports, types.ServicePortConfig{
+							Published: strconv.Itoa(hp),
+							Target:    uint32(cp),
+						})
+						superProject.Services[sv] = config
+						slog.Info(fmt.Sprintf("Published port %d of service '%s' to host port %d", cp, sv, hp))
+					}
+				} else if _, ok := currentState.Workloads[rr]; !ok {
+					return fmt.Errorf("--%s[%d] failed to find a workload named '%s'", generateCmdPublishFlag, i, rr)
+				} else {
+					for sv, config := range superProject.Services {
+						if config.Hostname == rr {
+							config.Ports = append(config.Ports, types.ServicePortConfig{
+								Published: strconv.Itoa(hp),
+								Target:    uint32(cp),
+							})
+							superProject.Services[sv] = config
+							slog.Info(fmt.Sprintf("Published port %d of service '%s' to host port %d", cp, sv, hp))
+						}
+					}
+				}
+			}
+		}
+
 		sd.State = *currentState
 		if err := sd.Persist(); err != nil {
 			return fmt.Errorf("failed to persist updated state directory: %w", err)
@@ -296,6 +379,21 @@ arguments.
 	},
 }
 
+func parseResourceUid(raw string) framework.ResourceUid {
+	parts := strings.SplitN(raw, "#", 2)
+	firstParts := strings.SplitN(parts[0], ".", 2)
+	secondParts := strings.SplitN(parts[1], ".", 2)
+	resType := firstParts[0]
+	var resClass *string
+	if len(firstParts) > 1 {
+		resClass = &firstParts[1]
+	}
+	if len(secondParts) == 1 {
+		return framework.NewResourceUid("", "", resType, resClass, &parts[1])
+	}
+	return framework.NewResourceUid(secondParts[0], secondParts[1], resType, resClass, nil)
+}
+
 // loadRawScoreFiles loads raw score specs as yaml from the given files and finds all the workload names. It throws
 // errors if it failed to read, load, or if names are duplicated.
 func loadRawScoreFiles(fileNames []string) ([]string, map[string]map[string]interface{}, error) {
@@ -331,6 +429,7 @@ func init() {
 	generateCommand.Flags().String(generateCmdImageFlag, "", "An optional container image to use for any container with image == '.'")
 	generateCommand.Flags().StringArray(generateCmdBuildFlag, []string{}, "An optional build context to use for the given container --build=container=./dir or --build=container={\"context\":\"./dir\"}")
 	generateCommand.Flags().String(generateCmdEnvFileFlag, "", "Location to store a skeleton .env file for compose - this will override existing content")
+	generateCommand.Flags().StringArray(generateCmdPublishFlag, []string{}, "An optional set of HOST_PORT:<ref>:CONTAINER_PORT to publish on the host system.")
 	rootCmd.AddCommand(generateCommand)
 }
 


Test patch:
diff --git a/internal/command/generate_examples_test.go b/internal/command/generate_examples_test.go
index dab35c0..e36298a 100644
--- a/internal/command/generate_examples_test.go
+++ b/internal/command/generate_examples_test.go
@@ -28,9 +28,10 @@ import (
 )
 
 type exampleTestCase struct {
-	subDir   string
-	adds     []string
-	expected string
+	subDir           string
+	adds             []string
+	expected         string
+	expectedContains string
 }
 
 func TestExample(t *testing.T) {
@@ -153,7 +154,12 @@ volumes:
 		},
 		{
 			subDir: "06-resource-provisioning",
-			adds:   []string{"score.yaml", "score2.yaml"},
+			adds:   []string{"score.yaml", "score2.yaml", "--publish 6379:redis#main-cache.host:6379"},
+			expectedContains: `
+        ports:
+            - target: 6379
+              published: "6379"
+`,
 		},
 		{
 			subDir: "07-overrides",
@@ -171,7 +177,7 @@ services:
 		},
 		{
 			subDir: "08-service-port-resource",
-			adds:   []string{"scoreA.yaml", "scoreB.yaml"},
+			adds:   []string{"scoreA.yaml", "scoreB.yaml", "--publish 8080:workload-a:80"},
 			expected: `name: 08-service-port-resource
 services:
     workload-a-example:
@@ -179,6 +185,9 @@ services:
             compose.score.dev/workload-name: workload-a
         hostname: workload-a
         image: nginx
+        ports:
+            - target: 80
+              published: "8080"
     workload-b-example:
         annotations:
             compose.score.dev/workload-name: workload-b
@@ -239,10 +248,13 @@ services:
 				assert.Equal(t, "", stdout)
 			}
 
-			if tc.expected != "" {
+			if tc.expected != "" || tc.expectedContains != "" {
 				raw, err := os.ReadFile("compose.yaml")
 				require.NoError(t, err)
-				assert.Equal(t, tc.expected, string(raw))
+				if tc.expected != "" {
+					assert.Equal(t, tc.expected, string(raw))
+				}
+				assert.Contains(t, string(raw), tc.expectedContains)
 			}
 
 			if os.Getenv("NO_DOCKER") == "" {
diff --git a/internal/command/generate_test.go b/internal/command/generate_test.go
index 87a579f..4fd0f3d 100644
--- a/internal/command/generate_test.go
+++ b/internal/command/generate_test.go
@@ -52,6 +52,12 @@ Examples:
   # Provide overrides when one score file is provided
   score-compose generate score.yaml --override-file=./overrides.score.yaml --override-property=metadata.key=value
 
+  # Publish a port exposed by a workload for local testing
+  score-compose generate score.yaml --publish 8080:my-workload:80
+
+  # Publish a port from a resource host and port for local testing, the middle expression is RESOURCE_ID.OUTPUT_KEY
+  score-compose generate score.yaml --publish 5432:postgres#my-workload.db.host:5432
+
 Flags:
       --build stringArray               An optional build context to use for the given container --build=container=./dir or --build=container={"context":"./dir"}
       --env-file string                 Location to store a skeleton .env file for compose - this will override existing content
@@ -60,6 +66,7 @@ Flags:
   -o, --output string                   The output file to write the composed compose file to (default "compose.yaml")
       --override-property stringArray   An optional set of path=key overrides to set or remove
       --overrides-file string           An optional file of Score overrides to merge in
+      --publish stringArray             An optional set of HOST_PORT:<ref>:CONTAINER_PORT to publish on the host system.
 
 Global Flags:
       --quiet           Mute any logging output
@@ -1288,3 +1295,67 @@ resources:
 		assert.NoError(t, cmd.Run())
 	})
 }
+
+func TestGeneratePublishPorts(t *testing.T) {
+	td := changeToTempDir(t)
+	stdout, _, err := executeAndResetCommand(context.Background(), rootCmd, []string{"init"})
+	assert.NoError(t, err)
+	assert.Equal(t, "", stdout)
+	assert.NoError(t, os.WriteFile(filepath.Join(td, "score.yaml"), []byte(`
+apiVersion: score.dev/v1b1
+metadata:
+  name: example
+containers:
+  hello:
+    image: foo
+resources:
+  db1:
+    type: postgres
+  db2:
+    type: postgres
+    id: thing
+`), 0644))
+
+	stdout, _, err = executeAndResetCommand(context.Background(), rootCmd, []string{
+		"generate", "score.yaml",
+		"--publish", "8080:example:80",
+		"--publish", "42:postgres#example.db1.host:13",
+		"--publish", "43:postgres.default#example.db1.host:14",
+		"--publish", "44:postgres.default#thing.host:15",
+	})
+	assert.NoError(t, err)
+	assert.Equal(t, "", stdout)
+
+	t.Run("validate compose spec", func(t *testing.T) {
+		if os.Getenv("NO_DOCKER") != "" {
+			t.Skip("NO_DOCKER is set")
+			return
+		}
+		dockerCmd, err := exec.LookPath("docker")
+		require.NoError(t, err)
+		cmd := exec.Command(dockerCmd, "compose", "-f", "compose.yaml", "convert", "--quiet", "--dry-run")
+		cmd.Dir = td
+		cmd.Stdout = os.Stdout
+		cmd.Stderr = os.Stderr
+		assert.NoError(t, cmd.Run())
+	})
+
+	for _, tc := range [][2]string{
+		{"something", "--publish[0] expected 3 :-separated parts"},
+		{"x:foo:y", "--publish[0] could not parse host port 'x' as integer"},
+		{"8080:foo:y", "--publish[0] could not parse container port 'y' as integer"},
+		{"42:thing:13", "--publish[0] failed to find a workload named 'thing'"},
+		{"42:x#y:13", "--publish[0] must match RES_UID.OUTPUT"},
+		{"42:x#y.z:13", "--publish[0] failed to find a resource with uid 'x#y'"},
+		{"42:postgres#thing.foo:13", "--publish[0] resource 'postgres.default#thing' has no output 'foo'"},
+	} {
+		t.Run("invalid publish "+tc[0], func(t *testing.T) {
+			stdout, _, err = executeAndResetCommand(context.Background(), rootCmd, []string{
+				"generate", "score.yaml",
+				"--publish", tc[0],
+			})
+			assert.EqualError(t, err, tc[1])
+			assert.Equal(t, "", stdout)
+		})
+	}
+}