Repository: artefactual-labs/enduro
Pull request #633: Add one Task queue and worker per AM Pipeline
Author: login: DanielCosme
Created: 2025-04-28T13:42:10Z | Merged: 2025-04-30T20:23:43Z

This MR will create one Temporal Task queue and worker per each AM Pipeline. Previously all SIPs were ingested into the same global Task queue, now every distinct AM pipeline that gets configured gets it's own Queue and Worker, when starting a `processing` workflow.

Notes: 

- The pipeline selection was previously done during the `processing` workflow, now it occurs before. That logic has been extracted into a standalone function.
- Batch and Bulk workflows will still run in the Global Task Queue.
- Each Pipeline object now has a `TaskQueue` field.

Fixes #618 

Linked issue #618: Problem: Pipelines remaining idle even when there are many packages queued 
Enduro in the current DPS setup watches three directories:

    dpj-ts
    epj-ts
    other-ts

It queues the packages as they appear in the directories. What seems to be an issue though, is the way these packages are distributed to the transfer slots. For example, if there are many epj-s queued before a batch of dpj-s, the dpj-s aren't picked up, even if the dpj pipeline is completeley idle.

The behaviour we're observing won't give us full utilization of all pipelines unless we somehow manage to distribute dpj-s and epj-s evenly in the queue, which is not really realistic. Consider the following scenario:

    a dpj submission is queued
    an epj submission is sent to archiving
    the sip containing the “avleveringsliste” is queued in enduro, after all the dpj-s
    epj sip creation does not start until the avl is archived, basically when the dpj submission is finished

The result is the epj and dpj  submissions are prevented from running in parallell, and even if we manage to tweak it so both epj and dpj sips are created, we have to make sure they get queued evenly.

Ideally, Enduro should trigger transfers for a certain type of package, as long as that type of pipeline has free transfer slots, ignoring the transfer's overall position in the queue.
Linked issue comments: user: login: sevein
created_at: 2024-12-15T06:46:30Z
body: If I understand this correctly, this issue arised from the changes introduced in https://github.com/artefactual-labs/enduro/pull/616, which effectively adds a global queue processing tasks in FIFO order without considering the availability of individual pipelines. If I remember correctly, we had to do this to address scalability challenges when running Temporal and Enduro workers in constrained environments where a high number of concurrent sessions were causing timeout errors in Temporal.

Users can mitigate the negative impact of the global queue by adjusting `MaxConcurrentSessionExecutionSize` and `MaxConcurrentWorkflowExecutionSize` to higher values (the default used to be `5000`). While this won't eliminate the issue, it can reduce its frequency (more packages would be dispatched directly to pipeline-specific queues) - however doing so requires provisioning additional compute resources for Temporal and Enduro to handle the increased load.

An alternative worth exploring is to push the existing pipeline-specific queues to operate outside the scope of worker sessions, assuming that worker sessions are resource-intensive and the root issue. For example, could we start a worker session only when a pipeline slot has been acquired? This approach could reduce the number of concurrent worker sessions to the maximum combined capacity of all pipelines. For instance, with 4 pipelines each having a capacity of 5, the system would operate at a maximum of 20 concurrent sessions, as the queuing would occur beforehand. This idea is relatively simple to implement and could provide significant efficiency gains.
author_association: MEMBER
user: login: DanielCosme
created_at: 2024-12-17T18:04:27Z
body: @sevein You are spot on in your understanding of the problem. My "solution" to the previous scalability problem created the problem described in this issue. Technically we could revert back to using generous values for `MaxConcurrentSessionExecutionSize` and `MaxConcurrentWorkflowExecutionSize` if we give the Temporal service an Isolated VM with generous resources (mainly to avoid it competing for disk IO with AM and Enduro). I say it because I tested that as a brute force alternative, and it worked. Maybe even pointing Temporal's DB to be in it's own block device in the same VM could help immensely (Separate from AM's DB). I'm not saying is a perfect solution, but it can temporarily mitigate this Limitation at the operational level.

I actually did some tests with your proposed solution (a while ago). I faced some unknown (to me) problems in which transfers would stop being sent to AM, I have high hopes that it's a small bug in Enduro we could actually solve. Sadly I did not look further. Let me know if it would be useful to setup a scalability testing environment for Enduro. By scalability testing in this particular case I mean:

 - 3 watched directories
 - 2+ AM pipelines (powered by docker in the same host)
 - 15k Transfer batches being queued at any given time.
 - 1 Instance of Enduro

Also, I was able to successfully reproduce this issue locally in my environment, emulating the client setup.
author_association: CONTRIBUTOR

Diff:
diff --git a/internal/batch/service.go b/internal/batch/service.go
index 72811771..d86a540c 100644
--- a/internal/batch/service.go
+++ b/internal/batch/service.go
@@ -143,7 +143,8 @@ func (s *batchImpl) InitProcessingWorkflow(ctx context.Context, req *collection.
 	req.ValidationConfig = validation.Config{}
 	// req.MetadataConfig = metadata.Config{}
 	tr := noop.NewTracerProvider().Tracer("")
-	err := collection.InitProcessingWorkflow(ctx, tr, s.cc, s.taskQueue, req)
+
+	err := collection.InitProcessingWorkflow(ctx, tr, s.cc, req)
 	if err != nil {
 		s.logger.Error(err, "Error initializing processing workflow.")
 	}
diff --git a/internal/batch/workflow.go b/internal/batch/workflow.go
index 35d833b0..313ffaa5 100644
--- a/internal/batch/workflow.go
+++ b/internal/batch/workflow.go
@@ -60,11 +60,6 @@ func NewBatchActivity(batchsvc Service) *BatchActivity {
 }
 
 func (a *BatchActivity) Execute(ctx context.Context, params BatchWorkflowInput) error {
-	pipelines := []string{}
-	if params.PipelineName != "" {
-		pipelines = append(pipelines, params.PipelineName)
-	}
-
 	if params.Depth < 0 {
 		params.Depth = 0
 	}
@@ -97,7 +92,7 @@ func (a *BatchActivity) Execute(ctx context.Context, params BatchWorkflowInput)
 			BatchDir:           filepath.Dir(path),
 			Key:                entry.Name(),
 			IsDir:              entry.IsDir(),
-			PipelineNames:      pipelines,
+			PipelineName:       params.PipelineName,
 			ProcessingConfig:   params.ProcessingConfig,
 			CompletedDir:       params.CompletedDir,
 			RetentionPeriod:    params.RetentionPeriod,
diff --git a/internal/batch/workflow_test.go b/internal/batch/workflow_test.go
index b41e4193..7e773595 100644
--- a/internal/batch/workflow_test.go
+++ b/internal/batch/workflow_test.go
@@ -37,14 +37,14 @@ func TestBatchActivityStartsProcessingWorkflows(t *testing.T) {
 		BatchDir:         batchPath,
 		Key:              "transfer1",
 		IsDir:            true,
-		PipelineNames:    []string{"am"},
+		PipelineName:     "am",
 		ProcessingConfig: "automated",
 	})
 	serviceMock.EXPECT().InitProcessingWorkflow(ctx, &collection.ProcessingWorkflowRequest{
 		BatchDir:         batchPath,
 		Key:              "transfer2",
 		IsDir:            true,
-		PipelineNames:    []string{"am"},
+		PipelineName:     "am",
 		ProcessingConfig: "automated",
 	})
 
diff --git a/internal/collection/goa.go b/internal/collection/goa.go
index 6862b248..b207095a 100644
--- a/internal/collection/goa.go
+++ b/internal/collection/goa.go
@@ -267,7 +267,7 @@ func (w *goaWrapper) Retry(ctx context.Context, payload *goacollection.RetryPayl
 	req.WorkflowID = *goacol.WorkflowID
 	req.CollectionID = goacol.ID
 	tr := noop.NewTracerProvider().Tracer("")
-	if err := InitProcessingWorkflow(ctx, tr, w.cc, w.taskQueue, req); err != nil {
+	if err := InitProcessingWorkflow(ctx, tr, w.cc, req); err != nil {
 		return fmt.Errorf("error starting the new workflow instance: %w", err)
 	}
 
diff --git a/internal/collection/workflow.go b/internal/collection/workflow.go
index 028db3dc..30c44768 100644
--- a/internal/collection/workflow.go
+++ b/internal/collection/workflow.go
@@ -12,6 +12,7 @@ import (
 	temporalsdk_client "go.temporal.io/sdk/client"
 
 	"github.com/artefactual-labs/enduro/internal/metadata"
+	"github.com/artefactual-labs/enduro/internal/pipeline"
 	"github.com/artefactual-labs/enduro/internal/validation"
 )
 
@@ -28,10 +29,7 @@ type ProcessingWorkflowRequest struct {
 	// Name of the watcher that received this blob.
 	WatcherName string
 
-	// Pipelines that are available for processing. The workflow will choose one
-	// (randomly picked for now). If the slice is empty, it will be
-	// automatically populated from the list of all configured pipelines.
-	PipelineNames []string
+	PipelineName string
 
 	// Period of time to schedule the deletion of the original blob from the
 	// watched data source. nil means no deletion.
@@ -72,7 +70,7 @@ type ProcessingWorkflowRequest struct {
 	MetadataConfig metadata.Config
 }
 
-func InitProcessingWorkflow(ctx context.Context, tr trace.Tracer, c temporalsdk_client.Client, taskQueue string, req *ProcessingWorkflowRequest) error {
+func InitProcessingWorkflow(ctx context.Context, tr trace.Tracer, c temporalsdk_client.Client, req *ProcessingWorkflowRequest) error {
 	_, span := tr.Start(ctx, "InitProcessingWorkflow")
 	defer span.End()
 
@@ -89,7 +87,7 @@ func InitProcessingWorkflow(ctx context.Context, tr trace.Tracer, c temporalsdk_
 
 	opts := temporalsdk_client.StartWorkflowOptions{
 		ID:                    req.WorkflowID,
-		TaskQueue:             taskQueue,
+		TaskQueue:             pipeline.TaskQueueName(req.PipelineName),
 		WorkflowIDReusePolicy: temporalsdk_api_enums.WORKFLOW_ID_REUSE_POLICY_ALLOW_DUPLICATE,
 	}
 	_, err := c.ExecuteWorkflow(ctx, opts, ProcessingWorkflowName, req)
diff --git a/internal/pipeline/pipeline.go b/internal/pipeline/pipeline.go
index 33e10fc7..b2f4aa0f 100644
--- a/internal/pipeline/pipeline.go
+++ b/internal/pipeline/pipeline.go
@@ -57,6 +57,7 @@ type Pipeline struct {
 	status          string
 	statusUpdatedAt time.Time
 	statusLock      sync.RWMutex
+	TaskQueue       string
 }
 
 func NewPipeline(logger logr.Logger, config Config) (*Pipeline, error) {
@@ -64,18 +65,19 @@ func NewPipeline(logger logr.Logger, config Config) (*Pipeline, error) {
 	config.ProcessingDir = expandPath(config.ProcessingDir)
 
 	p := &Pipeline{
-		logger: logger,
-		sem:    semaphore.NewWeighted(int64(config.Capacity)),
-		config: &config,
-		client: httpClient(),
+		logger:    logger,
+		sem:       semaphore.NewWeighted(int64(config.Capacity)),
+		config:    &config,
+		client:    httpClient(),
+		TaskQueue: TaskQueueName(config.Name),
 	}
 
 	if config.ID != "" {
 		p.ID = config.ID
 	}
 
-	// init() enriches our record by retrieving the UUID but we still return
-	// the the object in case of errors.
+	// init() enriches our record by retrieving the UUID, but we still return
+	// the object in case of errors.
 	if err := p.init(); err != nil {
 		return p, err
 	}
@@ -83,6 +85,10 @@ func NewPipeline(logger logr.Logger, config Config) (*Pipeline, error) {
 	return p, nil
 }
 
+func TaskQueueName(name string) string {
+	return fmt.Sprintf("pipeline-%s", name)
+}
+
 // init connects with the pipeline to retrieve its identifier, unless one has
 // already been defined via configuration.
 func (p *Pipeline) init() error {
diff --git a/internal/workflow/processing.go b/internal/workflow/processing.go
index f0b5aa05..19915617 100644
--- a/internal/workflow/processing.go
+++ b/internal/workflow/processing.go
@@ -286,27 +286,7 @@ func (w *ProcessingWorkflow) Execute(ctx temporalsdk_workflow.Context, req *coll
 		}
 	}
 
-	// Randomly choose the pipeline from the list of names provided. If the
-	// list is empty then choose one from the list of all configured pipelines.
-	{
-		var pick string
-		if err := temporalsdk_workflow.SideEffect(ctx, func(ctx temporalsdk_workflow.Context) interface{} {
-			names := req.PipelineNames
-			if len(names) < 1 {
-				names = w.pipelineRegistry.Names()
-				if len(names) < 1 {
-					return ""
-				}
-			}
-			src := rand.NewSource(time.Now().UnixNano())
-			rnd := rand.New(src) // #nosec G404 -- not security sensitive.
-			return names[rnd.Intn(len(names))]
-		}).Get(&pick); err != nil {
-			return err
-		}
-
-		tinfo.PipelineName = pick
-	}
+	tinfo.PipelineName = req.PipelineName
 
 	// Load pipeline configuration and hooks.
 	{
@@ -706,3 +686,18 @@ func (w *ProcessingWorkflow) transfer(sessCtx temporalsdk_workflow.Context, tinf
 
 	return nil
 }
+
+// RandomPipeline will randomly choose the pipeline from the list of names provided. If the
+// list is empty then choose one from the list of all configured pipelines.
+func RandomPipeline(pipelineNames []string, registry *pipeline.Registry) string {
+	names := pipelineNames
+	if len(names) < 1 {
+		names = registry.Names()
+		if len(names) < 1 {
+			return ""
+		}
+	}
+	src := rand.NewSource(time.Now().UnixNano())
+	rnd := rand.New(src) // #nosec G404 -- not security sensitive.
+	return names[rnd.Intn(len(names))]
+}
diff --git a/internal/workflow/processing_test.go b/internal/workflow/processing_test.go
index 963e48ec..0b4e46ab 100644
--- a/internal/workflow/processing_test.go
+++ b/internal/workflow/processing_test.go
@@ -71,7 +71,7 @@ func (s *ProcessingWorkflowTestSuite) TestParseErrorIsIgnored() {
 	s.env.ExecuteWorkflow(s.workflow.Execute, &collection.ProcessingWorkflowRequest{
 		CollectionID:     0,
 		WatcherName:      "watcher",
-		PipelineNames:    []string{"pipeline"},
+		PipelineName:     "pipeline",
 		RetentionPeriod:  &retentionPeriod,
 		StripTopLevelDir: true,
 		Key:              "key",
@@ -107,7 +107,7 @@ func (s *ProcessingWorkflowTestSuite) TestParseError() {
 	s.env.ExecuteWorkflow(s.workflow.Execute, &collection.ProcessingWorkflowRequest{
 		CollectionID:     0,
 		WatcherName:      "watcher",
-		PipelineNames:    []string{"pipeline"},
+		PipelineName:     "pipeline",
 		RetentionPeriod:  &retentionPeriod,
 		StripTopLevelDir: true,
 		Key:              "key",
diff --git a/main.go b/main.go
index f3e58207..c256dc66 100644
--- a/main.go
+++ b/main.go
@@ -218,10 +218,18 @@ func main() {
 								attribute.String("key", event.Key),
 								attribute.Bool("dir", event.IsDir),
 							)
-							logger.V(1).Info("Starting new workflow", "watcher", event.WatcherName, "bucket", event.Bucket, "key", event.Key, "dir", event.IsDir)
+							pipelineName := workflow.RandomPipeline(event.PipelineName, pipelineRegistry)
+							logger.V(1).Info(
+								"Starting new workflow",
+								"watcher", event.WatcherName,
+								"bucket", event.Bucket,
+								"key", event.Key,
+								"dir", event.IsDir,
+								"pipeline", pipelineName,
+							)
 							req := collection.ProcessingWorkflowRequest{
 								WatcherName:        event.WatcherName,
-								PipelineNames:      event.PipelineName,
+								PipelineName:       pipelineName,
 								RetentionPeriod:    event.RetentionPeriod,
 								CompletedDir:       event.CompletedDir,
 								StripTopLevelDir:   event.StripTopLevelDir,
@@ -233,7 +241,8 @@ func main() {
 								ValidationConfig:   config.Validation,
 								MetadataConfig:     config.Metadata,
 							}
-							if err := collection.InitProcessingWorkflow(ctx, tracer, temporalClient, config.Temporal.TaskQueue, &req); err != nil {
+
+							if err := collection.InitProcessingWorkflow(ctx, tracer, temporalClient, &req); err != nil {
 								logger.Error(err, "Error initializing processing workflow.")
 							}
 							span.End()
@@ -247,62 +256,36 @@ func main() {
 		}
 	}
 
-	// Workflow and activity worker.
-	{
-		h := hooks.NewHooks(config.Hooks)
-
-		done := make(chan struct{})
-		w := temporalsdk_worker.New(temporalClient, config.Temporal.TaskQueue, temporalsdk_worker.Options{
-			EnableSessionWorker:                    true,
-			MaxConcurrentSessionExecutionSize:      config.Worker.MaxConcurrentSessionExecutionSize,
-			MaxConcurrentWorkflowTaskExecutionSize: config.Worker.MaxConcurrentWorkflowsExecutionsSize,
-			MaxHeartbeatThrottleInterval:           config.Worker.HeartbeatThrottleInterval,
-			DefaultHeartbeatThrottleInterval:       config.Worker.HeartbeatThrottleInterval,
-		})
-		if err != nil {
-			logger.Error(err, "Error creating Temporal worker.")
-			os.Exit(1)
+	for _, p := range pipelineRegistry.List() {
+		if p.Status(ctx) == "active" {
+			logger.Info("Creating worker and task queue", "Pipeline", p.TaskQueue)
+		} else {
+			logger.Error(errors.New("Pipeline not active"), "Cannot create worker and task queue for inactive pipeline")
+			continue
 		}
-
-		w.RegisterWorkflowWithOptions(workflow.NewProcessingWorkflow(h, colsvc, pipelineRegistry, logger, config.Workflow).Execute, temporalsdk_workflow.RegisterOptions{Name: collection.ProcessingWorkflowName})
-		w.RegisterActivityWithOptions(activities.NewAcquirePipelineActivity(pipelineRegistry).Execute, temporalsdk_activity.RegisterOptions{Name: activities.AcquirePipelineActivityName})
-		w.RegisterActivityWithOptions(activities.NewDownloadActivity(h, pipelineRegistry, wsvc).Execute, temporalsdk_activity.RegisterOptions{Name: activities.DownloadActivityName})
-		w.RegisterActivityWithOptions(archive.NewExtractActivity(config.ExtractActivity).Execute, temporalsdk_activity.RegisterOptions{Name: archive.ExtractActivityName})
-		w.RegisterActivityWithOptions(activities.NewBundleActivity().Execute, temporalsdk_activity.RegisterOptions{Name: activities.BundleActivityName})
-		w.RegisterActivityWithOptions(activities.NewValidateTransferActivity().Execute, temporalsdk_activity.RegisterOptions{Name: activities.ValidateTransferActivityName})
-		w.RegisterActivityWithOptions(activities.NewTransferActivity(pipelineRegistry).Execute, temporalsdk_activity.RegisterOptions{Name: activities.TransferActivityName})
-		w.RegisterActivityWithOptions(activities.NewPollTransferActivity(pipelineRegistry).Execute, temporalsdk_activity.RegisterOptions{Name: activities.PollTransferActivityName})
-		w.RegisterActivityWithOptions(activities.NewPollIngestActivity(pipelineRegistry).Execute, temporalsdk_activity.RegisterOptions{Name: activities.PollIngestActivityName})
-		w.RegisterActivityWithOptions(activities.NewCleanUpActivity().Execute, temporalsdk_activity.RegisterOptions{Name: activities.CleanUpActivityName})
-		w.RegisterActivityWithOptions(activities.NewHidePackageActivity(pipelineRegistry).Execute, temporalsdk_activity.RegisterOptions{Name: activities.HidePackageActivityName})
-		w.RegisterActivityWithOptions(activities.NewDeleteOriginalActivity(wsvc).Execute, temporalsdk_activity.RegisterOptions{Name: activities.DeleteOriginalActivityName})
-		w.RegisterActivityWithOptions(activities.NewDisposeOriginalActivity(wsvc).Execute, temporalsdk_activity.RegisterOptions{Name: activities.DisposeOriginalActivityName})
-		w.RegisterActivityWithOptions(activities.NewPopulateMetadataActivity(pipelineRegistry).Execute, temporalsdk_activity.RegisterOptions{Name: activities.PopulateMetadataActivityName})
-
-		w.RegisterActivityWithOptions(workflow.NewAsyncCompletionActivity(colsvc).Execute, temporalsdk_activity.RegisterOptions{Name: workflow.AsyncCompletionActivityName})
-		w.RegisterActivityWithOptions(nha_activities.NewUpdateHARIActivity(h).Execute, temporalsdk_activity.RegisterOptions{Name: nha_activities.UpdateHARIActivityName})
-		w.RegisterActivityWithOptions(nha_activities.NewUpdateProductionSystemActivity(h).Execute, temporalsdk_activity.RegisterOptions{Name: nha_activities.UpdateProductionSystemActivityName})
-
-		w.RegisterWorkflowWithOptions(collection.BulkWorkflow, temporalsdk_workflow.RegisterOptions{Name: collection.BulkWorkflowName})
-		w.RegisterActivityWithOptions(collection.NewBulkActivity(colsvc).Execute, temporalsdk_activity.RegisterOptions{Name: collection.BulkActivityName})
-
-		w.RegisterWorkflowWithOptions(batch.BatchWorkflow, temporalsdk_workflow.RegisterOptions{Name: batch.BatchWorkflowName})
-		w.RegisterActivityWithOptions(batch.NewBatchActivity(batchsvc).Execute, temporalsdk_activity.RegisterOptions{Name: batch.BatchActivityName})
-
-		g.Add(
-			func() error {
-				if err := w.Start(); err != nil {
-					return err
-				}
-				<-done
-				return nil
-			},
-			func(err error) {
-				w.Stop()
-				close(done)
-			},
+		registerWorker(
+			p.TaskQueue,
+			temporalClient,
+			pipelineRegistry,
+			config,
+			colsvc,
+			wsvc,
+			batchsvc,
+			logger,
+			&g,
 		)
 	}
+	registerWorker(
+		config.Temporal.TaskQueue,
+		temporalClient,
+		pipelineRegistry,
+		config,
+		colsvc,
+		wsvc,
+		batchsvc,
+		logger,
+		&g,
+	)
 
 	// Observability server.
 	{
@@ -506,3 +489,66 @@ func initTracerProvider(ctx context.Context, logger logr.Logger, cfg TelemetryCo
 
 	return tp, shutdown, nil
 }
+
+func registerWorker(
+	taskQueue string,
+	temporalClient temporalsdk_client.Client,
+	pipelineRegistry *pipeline.Registry,
+	config configuration,
+	colsvc collection.Service,
+	wsvc watcher.Service,
+	batchsvc batch.Service,
+	logger logr.Logger,
+	g *run.Group,
+) {
+	// Workflow and activity worker.
+	h := hooks.NewHooks(config.Hooks)
+
+	done := make(chan struct{})
+	w := temporalsdk_worker.New(temporalClient, taskQueue, temporalsdk_worker.Options{
+		EnableSessionWorker:                    true,
+		MaxConcurrentSessionExecutionSize:      config.Worker.MaxConcurrentSessionExecutionSize,
+		MaxConcurrentWorkflowTaskExecutionSize: config.Worker.MaxConcurrentWorkflowsExecutionsSize,
+		MaxHeartbeatThrottleInterval:           config.Worker.HeartbeatThrottleInterval,
+		DefaultHeartbeatThrottleInterval:       config.Worker.HeartbeatThrottleInterval,
+	})
+
+	w.RegisterWorkflowWithOptions(workflow.NewProcessingWorkflow(h, colsvc, pipelineRegistry, logger, config.Workflow).Execute, temporalsdk_workflow.RegisterOptions{Name: collection.ProcessingWorkflowName})
+	w.RegisterActivityWithOptions(activities.NewAcquirePipelineActivity(pipelineRegistry).Execute, temporalsdk_activity.RegisterOptions{Name: activities.AcquirePipelineActivityName})
+	w.RegisterActivityWithOptions(activities.NewDownloadActivity(h, pipelineRegistry, wsvc).Execute, temporalsdk_activity.RegisterOptions{Name: activities.DownloadActivityName})
+	w.RegisterActivityWithOptions(archive.NewExtractActivity(config.ExtractActivity).Execute, temporalsdk_activity.RegisterOptions{Name: archive.ExtractActivityName})
+	w.RegisterActivityWithOptions(activities.NewBundleActivity().Execute, temporalsdk_activity.RegisterOptions{Name: activities.BundleActivityName})
+	w.RegisterActivityWithOptions(activities.NewValidateTransferActivity().Execute, temporalsdk_activity.RegisterOptions{Name: activities.ValidateTransferActivityName})
+	w.RegisterActivityWithOptions(activities.NewTransferActivity(pipelineRegistry).Execute, temporalsdk_activity.RegisterOptions{Name: activities.TransferActivityName})
+	w.RegisterActivityWithOptions(activities.NewPollTransferActivity(pipelineRegistry).Execute, temporalsdk_activity.RegisterOptions{Name: activities.PollTransferActivityName})
+	w.RegisterActivityWithOptions(activities.NewPollIngestActivity(pipelineRegistry).Execute, temporalsdk_activity.RegisterOptions{Name: activities.PollIngestActivityName})
+	w.RegisterActivityWithOptions(activities.NewCleanUpActivity().Execute, temporalsdk_activity.RegisterOptions{Name: activities.CleanUpActivityName})
+	w.RegisterActivityWithOptions(activities.NewHidePackageActivity(pipelineRegistry).Execute, temporalsdk_activity.RegisterOptions{Name: activities.HidePackageActivityName})
+	w.RegisterActivityWithOptions(activities.NewDeleteOriginalActivity(wsvc).Execute, temporalsdk_activity.RegisterOptions{Name: activities.DeleteOriginalActivityName})
+	w.RegisterActivityWithOptions(activities.NewDisposeOriginalActivity(wsvc).Execute, temporalsdk_activity.RegisterOptions{Name: activities.DisposeOriginalActivityName})
+	w.RegisterActivityWithOptions(activities.NewPopulateMetadataActivity(pipelineRegistry).Execute, temporalsdk_activity.RegisterOptions{Name: activities.PopulateMetadataActivityName})
+
+	w.RegisterActivityWithOptions(workflow.NewAsyncCompletionActivity(colsvc).Execute, temporalsdk_activity.RegisterOptions{Name: workflow.AsyncCompletionActivityName})
+	w.RegisterActivityWithOptions(nha_activities.NewUpdateHARIActivity(h).Execute, temporalsdk_activity.RegisterOptions{Name: nha_activities.UpdateHARIActivityName})
+	w.RegisterActivityWithOptions(nha_activities.NewUpdateProductionSystemActivity(h).Execute, temporalsdk_activity.RegisterOptions{Name: nha_activities.UpdateProductionSystemActivityName})
+
+	w.RegisterWorkflowWithOptions(collection.BulkWorkflow, temporalsdk_workflow.RegisterOptions{Name: collection.BulkWorkflowName})
+	w.RegisterActivityWithOptions(collection.NewBulkActivity(colsvc).Execute, temporalsdk_activity.RegisterOptions{Name: collection.BulkActivityName})
+
+	w.RegisterWorkflowWithOptions(batch.BatchWorkflow, temporalsdk_workflow.RegisterOptions{Name: batch.BatchWorkflowName})
+	w.RegisterActivityWithOptions(batch.NewBatchActivity(batchsvc).Execute, temporalsdk_activity.RegisterOptions{Name: batch.BatchActivityName})
+
+	g.Add(
+		func() error {
+			if err := w.Start(); err != nil {
+				return err
+			}
+			<-done
+			return nil
+		},
+		func(err error) {
+			w.Stop()
+			close(done)
+		},
+	)
+}


Reviews:
user: login: sevein
body: LGTM
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-28T19:18:36Z
user: login: sevein
body: Please can you describe somewhere what is the problem that you're solving? That'd be a nice thing to have, thank you!
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-28T19:19:51Z
user: login: DanielCosme
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T20:15:25Z
user: login: DanielCosme
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T20:16:19Z
user: login: DanielCosme
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T20:16:25Z
user: login: DanielCosme
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T20:16:43Z

Review comments:
diff_hunk: @@ -506,3 +489,68 @@ func initTracerProvider(ctx context.Context, logger logr.Logger, cfg TelemetryCo
 
 	return tp, shutdown, nil
 }
+
+func RegisterWorker(
path: main.go
user: login: sevein
body: It does not need to be exported.
created_at: 2025-04-28T19:04:58Z
author_association: MEMBER
diff_hunk: @@ -506,3 +489,68 @@ func initTracerProvider(ctx context.Context, logger logr.Logger, cfg TelemetryCo
 
 	return tp, shutdown, nil
 }
+
+func RegisterWorker(
+	taskQueue string,
+	temporalClient temporalsdk_client.Client,
+	pipelineRegistry *pipeline.Registry,
+	config configuration,
+	colsvc collection.Service,
+	wsvc watcher.Service,
+	batchsvc batch.Service,
+	logger logr.Logger,
+	g run.Group,
path: main.go
user: login: sevein
body: Best to use `*run.Group`.
created_at: 2025-04-28T19:07:07Z
author_association: MEMBER
diff_hunk: @@ -506,3 +489,68 @@ func initTracerProvider(ctx context.Context, logger logr.Logger, cfg TelemetryCo
 
 	return tp, shutdown, nil
 }
+
+func RegisterWorker(
+	taskQueue string,
+	temporalClient temporalsdk_client.Client,
+	pipelineRegistry *pipeline.Registry,
+	config configuration,
+	colsvc collection.Service,
+	wsvc watcher.Service,
+	batchsvc batch.Service,
+	logger logr.Logger,
+	g run.Group,
+) {
+	// Workflow and activity worker.
+	{
path: main.go
user: login: sevein
body: The extra nesting using curly brackets makes sense in a big main function but I don't think we need it here. FYI I borrowed that style from Peter Bourgon, e.g.: https://github.com/go-kit/examples/blob/master/addsvc/cmd/addsvc/addsvc.go.
created_at: 2025-04-28T19:09:22Z
author_association: MEMBER
diff_hunk: @@ -706,3 +686,18 @@ func (w *ProcessingWorkflow) transfer(sessCtx temporalsdk_workflow.Context, tinf
 
 	return nil
 }
+
+// RandomPipeline  will randomly choose the pipeline from the list of names provided. If the
path: internal/workflow/processing.go
user: login: sevein
body: Extra space before `will randomly...`.
created_at: 2025-04-28T19:10:59Z
author_association: MEMBER
diff_hunk: @@ -706,3 +686,18 @@ func (w *ProcessingWorkflow) transfer(sessCtx temporalsdk_workflow.Context, tinf
 
 	return nil
 }
+
+// RandomPipeline  will randomly choose the pipeline from the list of names provided. If the
path: internal/workflow/processing.go
user: login: DanielCosme
body: Fixed.
created_at: 2025-04-30T20:15:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -506,3 +489,68 @@ func initTracerProvider(ctx context.Context, logger logr.Logger, cfg TelemetryCo
 
 	return tp, shutdown, nil
 }
+
+func RegisterWorker(
+	taskQueue string,
+	temporalClient temporalsdk_client.Client,
+	pipelineRegistry *pipeline.Registry,
+	config configuration,
+	colsvc collection.Service,
+	wsvc watcher.Service,
+	batchsvc batch.Service,
+	logger logr.Logger,
+	g run.Group,
+) {
+	// Workflow and activity worker.
+	{
path: main.go
user: login: DanielCosme
body: I deleted the extra brackets. I like them, but I agree in this case is not necessary to keep them.
created_at: 2025-04-30T20:16:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -506,3 +489,68 @@ func initTracerProvider(ctx context.Context, logger logr.Logger, cfg TelemetryCo
 
 	return tp, shutdown, nil
 }
+
+func RegisterWorker(
+	taskQueue string,
+	temporalClient temporalsdk_client.Client,
+	pipelineRegistry *pipeline.Registry,
+	config configuration,
+	colsvc collection.Service,
+	wsvc watcher.Service,
+	batchsvc batch.Service,
+	logger logr.Logger,
+	g run.Group,
path: main.go
user: login: DanielCosme
body: Done. Thank you for catching that.
created_at: 2025-04-30T20:16:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -506,3 +489,68 @@ func initTracerProvider(ctx context.Context, logger logr.Logger, cfg TelemetryCo
 
 	return tp, shutdown, nil
 }
+
+func RegisterWorker(
path: main.go
user: login: DanielCosme
body: Done.
created_at: 2025-04-30T20:16:42Z
author_association: CONTRIBUTOR

PR comments:
user: login: DanielCosme
created_at: 2025-04-30T16:39:12Z
body: > Please can you describe somewhere what is the problem that you're solving? That'd be a nice thing to have, thank you!

Yes, there is an Issue for it: #618 
author_association: CONTRIBUTOR
user: login: DanielCosme
created_at: 2025-04-30T20:18:48Z
body: @sevein BTW I added two new commits in order to bring back the default configuration `enduro.toml` to it's default values. No functional changes. 775212853685730369850ea7e551c955e2b3bd1c d3151eed2cd772b2da44fc6e88bddc76ddece410   
Now you should see no changes to `enduro.toml` in the PR Diff.
author_association: CONTRIBUTOR