Repository: diggerhq/digger
Pull request #384 (2023-06-16 11:08:10)

feat: setting to accumulate plans as one comment
https://github.com/diggerhq/digger/issues/370


Patch:
diff --git a/README.md b/README.md
index 335ed9e0..c87e3cca 100644
--- a/README.md
+++ b/README.md
@@ -2,87 +2,63 @@
 
 <h2 align="center">
   <a href="https://join.slack.com/t/diggertalk/shared_invite/zt-1tocl4w0x-E3RkpPiK7zQkehl8O78g8Q">Slack</a> |
-  <a href="https://docs.digger.dev/">Docs</a>
+  <a href="https://docs.digger.dev/">Docs</a> | 
+  <a href="https://www.loom.com/share/51f27994d95f4dc5bb6eea579e1fa8dc?sid=403f161a-6c0b-44ac-af57-cc9b56190f64">Demo Video</a>
 </h2>
 
 CI/CD for Terraform is [tricky](https://itnext.io/pains-in-terraform-collaboration-249a56b4534e). To make life easier, specialised CI systems aka [TACOS](https://itnext.io/spice-up-your-infrastructure-as-code-with-tacos-1a9c179e0783) exist - Terraform Cloud, Spacelift, Atlantis, etc.
 
 But why have 2 CI systems? Why not reuse the async jobs infrastructure with compute, orchestration, logs, etc of your existing CI?
 
-Digger runs terraform natively in your CI and takes care of the other bits - locks, plan artifacts and so on. [Demo video](https://www.loom.com/share/e201e639a73941e0b5508710377a6106)
+Digger runs terraform natively in your CI. This is:
 
+- Secure, because cloud access secrets aren't shared with a third-party
+- Cost-effective, because you are not paying for additional compute just to run your terraform
 
 ## Features
-- 👟 Runner-less. Terraform runs in the compute environment of your existing CI such as Github Actions, Gitlab, Argo etc.
-- 🪶 Minimal / no backend. Digger's own backend is a serverless function; it is only needed for certain CI environments (eg Gitlab)
-- 🔒 Code-level locks. Avoid race conditions across multiple PRs. Similar to Atlantis workflow.
-- ☁️ Multi-cloud. At the moment Digger supports AWS and GCP; Azure support coming in April 2023 (yes, in a few weeks).
-- 💥 Projects. Allow to isolate terraform runs and locks to a specific directory
-- 💥 Terragrunt support
-- 💥 Workspaces support
+- Terraform plan and apply in pull request comments
+- Any VCS - Github, Gitlab, Azure Repos, etc
+- Any CI - Github Actions, Gitlab, Azure DevOps, etc
+- Any cloud provider - AWS, GCP, Azure
+- Private runners - thanks to the fact that there are no separate runners! Your existing CI's compute environment is used
+- Open Policy Agent (OPA) support for RBAC
+- PR-level locks (on top of Terraform native state locks, similar to Atlantis) to avoid race conditions across multiple PRs
+- Terragrunt, Workspaces, multiple Terraform versions, static analysis via Checkov, plan persistence, ...
+- Drift detection - coming soon
+- Cost estimation - coming soon
 
-## Roadmap
+## Getting Started
 
-Need a feature that's not listed? Book a [community feedback call](https://calendly.com/diggerdev/community-feedback) - we ship fast ✅
+- [Github Actions + AWS](https://docs.digger.dev/getting-started/github-actions-+-aws)
+- [Github Actions + GCP](https://docs.digger.dev/getting-started/github-actions-and-gcp)
+- [Gitlab Pipelines + GCP](https://docs.digger.dev/getting-started/gitlab-pipelines-+-aws)
+- [Azure DevOps](https://docs.digger.dev/getting-started/azure-devops)
 
-- ✅ GCP support. Store PR locks in GCP storage buckets. Shipped in [#50](https://github.com/diggerhq/digger/pull/50)
-- ✅ Workspaces support. Allow usage of Terraform CLI Workspaces. Shipped in [#72](https://github.com/diggerhq/digger/pull/72)
-- ✅ Terragrunt support. Config option to run terragrunt wrapper. Shipped in [#76](https://github.com/diggerhq/digger/pull/76)
-- ✅ Azure support using Storage Account Tables WIP: [#122](https://github.com/diggerhq/digger/pull/122)
-- ⌚ AWS CodeBuild support
-- ⌛ Gitlab Support
-- ⌛ Configurable workflows. In addition to Atlantis-style (apply, then merge) also support "apply-only" and "no-lock"
-- ⌛ Bitbucket Support
-- ⌛ Jenkins Support
+## How it works
 
-## How to use
+Digger has 2 main components:
+- CLI that runs inside your CI and calls terraform with the right arguments
+- Orchestrator - a minimal backend (that can also be self-hosted) that triggers CI jobs in response to events such as PR comments
 
-This is demo flow with a sample repo using local state - for real world scenario you'll need to configure remote backend (S3 + DynamoDB) and add a [workflow file](https://github.com/diggerhq/digger_demo/blob/main/.github/workflows/plan.yml) to the root of the repo.
+Digger also stores PR-level locks and plan cache in your cloud account (DynamoDB + S3 on AWS, equivalents in other cloud providers)
 
-1. Fork the [demo repository](https://github.com/diggerhq/digger_demo_multienv)
-2. Enable Actions (by default workflows won't trigger in a fork)
-
-<img width="1441" alt="Screenshot 2023-02-24 at 20 24 08" src="https://user-images.githubusercontent.com/1280498/221291130-6831d45a-008f-452f-91d3-37ba133d7cbb.png">
-
-
-2. In your repository settings > Actions ensure that the Workflow Read and Write permissions are assigned - This will allow the workflow  to post comments on your PRs
-<img width="1017" alt="Screen Shot 2023-03-01 at 12 02 59 PM" src="https://user-images.githubusercontent.com/1627972/222136385-c7cb8f2c-1731-475d-b3a4-78b0d79a3874.png">
-
-3. Add environment variables into your Github Action Secrets (NOTE: This step is optional if you just want to test out the Action with `null_resource`)
-- AWS_ACCESS_KEY_ID
-- AWS_SECRET_ACCESS_KEY 
-4. make a change and create a PR - this will create a lock
-5. comment `digger plan` - terraform plan output will be added as comment. If you don't see a comment (bug) - check out job output
-6. create another PR - plan or apply won’t work in this PR until the first lock is released
-7. you should see `Locked by PR #1` comment. The action logs will display "Project locked" error message.
-
-## Remote backend and state-level locks
-
-Digger does not interfere with your remote backend setup. You could be using [S3 backend](https://developer.hashicorp.com/terraform/language/settings/backends/s3) or TF cloud's [remote backend](https://developer.hashicorp.com/terraform/language/settings/backends/remote) or [some other way](https://developer.hashicorp.com/terraform/language/settings/backends/configuration)
-
-Digger also doesn't differentiate locks based on statefiles - if a PR is locked, it's locked for all "instances" of state (aka [Terraform CLI Workspaces](https://developer.hashicorp.com/terraform/cloud-docs/workspaces#terraform-cloud-vs-terraform-cli-workspaces))
-
-state-level locks will keep working normally because are handled by terraform itself ([same as in Atlantis](https://www.runatlantis.io/docs/locking.html#relationship-to-terraform-state-locking))
-
-
-## Notes
-- We perform anonymous usage tracking. No sensitive or personal / identifyable data is logged. You can see what is tracked in [`pkg/utils/usage.go`](https://github.com/diggerhq/digger/blob/main/pkg/utils/usage.go)
+## Telementry
+No sensitive or personal / identifyable data is logged. You can see what is tracked in [`pkg/utils/usage.go`](https://github.com/diggerhq/digger/blob/main/pkg/utils/usage.go)
 
 ## Contributing
-**If you are considering using digger within your organisation
-please [reach out to us](https://join.slack.com/t/diggertalk/shared_invite/zt-1q6npg7ib-9dwRbJp8sQpSr2fvWzt9aA).**
 
-To contribute to Digger please follow our [Contributing guide](CONTRIBUTING.md)
+We love contributions. Check out our [contribiting guide](CONTRIBUTING.md) to get started. 
+
+Not sure where to get started? You can:
 
-## FAQ
+-   [Book a free, non-pressure pairing session / code walkthrough with one of our teammates](https://calendly.com/diggerdev/digger-pro-demo-clone)!
+-   Join our <a href="https://join.slack.com/t/diggertalk/shared_invite/zt-1tocl4w0x-E3RkpPiK7zQkehl8O78g8Q">Slack</a>, and ask us any questions there.
 
-Q) **Since you're FOSS I assume you plan to monetize by selling support? Or...?**
 
-A) We are a vc-backed startup fully focused on this tool; in terms of monetization - we are currently in the process of launching Digger Pro. Check out the features [here](https://digger.dev/#plans) and feel free to book a [demo](https://bit.ly/diggerpro) if interested.
+## Resources
 
-## Links
-- [The case for a 'Headless Terraform IDP' for terraform self service](https://medium.com/@DiggerHQ/the-case-for-headless-terraform-idp-5bc5a873805f)
-- [Can GitHub actions be used as a CI/CD for Terraform?](https://medium.com/@DiggerHQ/can-github-actions-be-used-as-a-ci-cd-for-terraform-e4ac59a38b0)
-- [Why are people using Terraform Cloud?](https://www.reddit.com/r/Terraform/comments/1132qf3/why_are_people_using_terraform_cloud_i_may_be/)
-- [The Pains of Terraform Collaboration](https://itnext.io/pains-in-terraform-collaboration-249a56b4534e)
-- [Four Great Alternatives to HashiCorp’s Terraform Cloud](https://medium.com/@elliotgraebert/four-great-alternatives-to-hashicorps-terraform-cloud-6e0a3a0a5482)
+- [Docs](https://docs.digger.dev/) for comprehensive documentation and guides
+- [Slack](https://join.slack.com/t/diggertalk/shared_invite/zt-1tocl4w0x-E3RkpPiK7zQkehl8O78g8Q) for discussion with the community and Infisical team.
+- [GitHub](https://github.com/diggerhq/digger) for code, issues, and pull request
+- [Medium](https://medium.com/@DiggerHQ) for terraform automation and collaboration insights, articles, tutorials, and updates.
+- [Roadmap](https://diggerdev.notion.site/Digger-Roadmap-845a90fb17954afca80431580e1b3958?pvs=4) for planned features
diff --git a/action.yml b/action.yml
index bcc0677d..df46e022 100644
--- a/action.yml
+++ b/action.yml
@@ -86,6 +86,10 @@ inputs:
     description: Setup tfenv
     required: false
     default: 'false'
+  post-plans-as-one-comment:
+    description: Post plans as one comment
+    required: false
+    default: 'false'
 outputs:
   output:
     value: ${{ steps.digger.outputs.output }}
@@ -186,6 +190,7 @@ runs:
         POLICY_CHECK_ENABLED: ${{ inputs.policy-check-enabled == 'true' }}
         DIGGER_CLOUD_TOKEN: ${{ inputs.digger-cloud-token }}
         DIGGER_CLOUD_HOSTNAME: ${{ inputs.digger-cloud-hostname }}
+        ACCUMULATE_PLANS: ${{ inputs.post-plans-as-one-comment == 'true' }}
       run: |
           cd ${{ github.action_path }}
           go build -o digger ./cmd/digger
@@ -203,6 +208,7 @@ runs:
         POLICY_CHECK_ENABLED: ${{ inputs.policy-check-enabled == 'true' }}
         DIGGER_CLOUD_TOKEN: ${{ inputs.digger-cloud-token }}
         DIGGER_CLOUD_HOSTNAME: ${{ inputs.digger-cloud-hostname }}
+        ACCUMULATE_PLANS: ${{ inputs.post-plans-as-one-comment == 'true' }}
       id: digger
       shell: bash
       run: |
diff --git a/pkg/core/execution/execution.go b/pkg/core/execution/execution.go
index d32087a6..6cdc4cb6 100644
--- a/pkg/core/execution/execution.go
+++ b/pkg/core/execution/execution.go
@@ -43,10 +43,11 @@ func (d DiggerExecutor) storedPlanFilePath() string {
 	return path.Join(d.ProjectNamespace, d.planFileName())
 }
 
-func (d DiggerExecutor) Plan() (bool, error) {
+func (d DiggerExecutor) Plan() (bool, string, error) {
+	plan := ""
 	locked, err := d.ProjectLock.Lock()
 	if err != nil {
-		return false, fmt.Errorf("error locking project: %v", err)
+		return false, "", fmt.Errorf("error locking project: %v", err)
 	}
 	log.Printf("Lock result: %t\n", locked)
 	if locked {
@@ -68,7 +69,7 @@ func (d DiggerExecutor) Plan() (bool, error) {
 			if step.Action == "init" {
 				_, _, err := d.TerraformExecutor.Init(step.ExtraArgs, d.StateEnvVars)
 				if err != nil {
-					return false, fmt.Errorf("error running init: %v", err)
+					return false, "", fmt.Errorf("error running init: %v", err)
 				}
 			}
 			if step.Action == "plan" {
@@ -76,29 +77,27 @@ func (d DiggerExecutor) Plan() (bool, error) {
 				planArgs = append(planArgs, step.ExtraArgs...)
 				isNonEmptyPlan, stdout, stderr, err := d.TerraformExecutor.Plan(planArgs, d.CommandEnvVars)
 				if err != nil {
-					return false, fmt.Errorf("error executing plan: %v", err)
+					return false, "", fmt.Errorf("error executing plan: %v", err)
 				}
 				if d.PlanStorage != nil {
 					planExists, err := d.PlanStorage.PlanExists(d.storedPlanFilePath())
 					if err != nil {
-						return false, fmt.Errorf("error checking if plan exists: %v", err)
+						return false, "", fmt.Errorf("error checking if plan exists: %v", err)
 					}
 
 					if planExists {
 						err = d.PlanStorage.DeleteStoredPlan(d.storedPlanFilePath())
 						if err != nil {
-							return false, fmt.Errorf("error deleting plan: %v", err)
+							return false, "", fmt.Errorf("error deleting plan: %v", err)
 						}
 					}
 
 					err = d.PlanStorage.StorePlan(d.localPlanFilePath(), d.storedPlanFilePath())
 					if err != nil {
-						return false, fmt.Errorf("error storing plan: %v", err)
+						return false, "", fmt.Errorf("error storing plan: %v", err)
 					}
 				}
-				plan := cleanupTerraformPlan(isNonEmptyPlan, err, stdout, stderr)
-				comment := utils.GetTerraformOutputAsCollapsibleComment("Plan for **"+d.ProjectLock.LockId()+"**", plan)
-				err = d.Reporter.Report(comment)
+				plan = cleanupTerraformPlan(isNonEmptyPlan, err, stdout, stderr)
 				if err != nil {
 					fmt.Printf("error publishing comment: %v", err)
 				}
@@ -112,13 +111,13 @@ func (d DiggerExecutor) Plan() (bool, error) {
 				log.Printf("Running %v for **%v**\n", step.Value, d.ProjectLock.LockId())
 				_, _, err := d.CommandRunner.Run(d.ProjectPath, step.Shell, commands)
 				if err != nil {
-					return false, fmt.Errorf("error running command: %v", err)
+					return false, "", fmt.Errorf("error running command: %v", err)
 				}
 			}
 		}
-		return true, nil
+		return true, plan, nil
 	}
-	return false, nil
+	return false, plan, nil
 }
 
 func (d DiggerExecutor) Apply() (bool, error) {
diff --git a/pkg/digger/digger.go b/pkg/digger/digger.go
index 2e344595..45bf271a 100644
--- a/pkg/digger/digger.go
+++ b/pkg/digger/digger.go
@@ -10,6 +10,7 @@ import (
 	"digger/pkg/core/runners"
 	"digger/pkg/core/storage"
 	"digger/pkg/core/terraform"
+	"digger/pkg/core/utils"
 	"digger/pkg/locking"
 	"digger/pkg/usage"
 	"errors"
@@ -17,6 +18,7 @@ import (
 	"log"
 	"os"
 	"path"
+	"strings"
 	"time"
 )
 
@@ -56,8 +58,22 @@ func DetectCI() CIName {
 
 }
 
-func RunCommandsPerProject(commandsPerProject []models.ProjectCommand, projectNamespace string, requestedBy string, eventName string, prNumber int, ciService ci.CIService, lock core_locking.Lock, reporter reporting.Reporter, planStorage storage.PlanStorage, policyChecker policy.Checker, workingDir string) (bool, bool, error) {
+func RunCommandsPerProject(
+	commandsPerProject []models.ProjectCommand,
+	projectNamespace string,
+	requestedBy string,
+	eventName string,
+	prNumber int,
+	ciService ci.CIService,
+	lock core_locking.Lock,
+	reporter reporting.Reporter,
+	planStorage storage.PlanStorage,
+	policyChecker policy.Checker,
+	workingDir string,
+) (bool, bool, error) {
+	accumulatePlans := os.Getenv("ACCUMULATE_PLANS") == "true"
 	appliesPerProject := make(map[string]bool)
+	plansToPublish := make([]string, 0)
 	for _, projectCommands := range commandsPerProject {
 		for _, command := range projectCommands.Commands {
 			policyInput := map[string]interface{}{"user": requestedBy, "action": command}
@@ -112,12 +128,21 @@ func RunCommandsPerProject(commandsPerProject []models.ProjectCommand, projectNa
 			case "digger plan":
 				usage.SendUsageRecord(requestedBy, eventName, "plan")
 				ciService.SetStatus(prNumber, "pending", projectCommands.ProjectName+"/plan")
-				planPerformed, err := diggerExecutor.Plan()
+				planPerformed, plan, err := diggerExecutor.Plan()
 				if err != nil {
 					log.Printf("Failed to run digger plan command. %v", err)
 					ciService.SetStatus(prNumber, "failure", projectCommands.ProjectName+"/plan")
 					return false, false, fmt.Errorf("failed to run digger plan command. %v", err)
 				} else if planPerformed {
+					comment := utils.GetTerraformOutputAsCollapsibleComment("Plan for **"+projectLock.LockId()+"**", plan)
+					if accumulatePlans {
+						plansToPublish = append(plansToPublish, comment)
+					} else {
+						err = reporter.Report(comment)
+						if err != nil {
+							log.Printf("Failed to report plan. %v", err)
+						}
+					}
 					ciService.SetStatus(prNumber, "success", projectCommands.ProjectName+"/plan")
 				}
 			case "digger apply":
@@ -164,6 +189,13 @@ func RunCommandsPerProject(commandsPerProject []models.ProjectCommand, projectNa
 		}
 	}
 
+	if len(plansToPublish) > 0 {
+		err := reporter.Report(strings.Join(plansToPublish, "\n"))
+		if err != nil {
+			log.Printf("Failed to report plans. %v", err)
+		}
+	}
+
 	allAppliesSuccess := true
 	for _, success := range appliesPerProject {
 		if !success {


Test patch:
diff --git a/pkg/digger/digger_test.go b/pkg/digger/digger_test.go
index 1421e76b..daf597c4 100644
--- a/pkg/digger/digger_test.go
+++ b/pkg/digger/digger_test.go
@@ -241,7 +241,7 @@ func TestCorrectCommandExecutionWhenPlanning(t *testing.T) {
 
 	commandStrings := allCommandsInOrderWithParams(terraformExecutor, commandRunner, prManager, lock, planStorage)
 
-	assert.Equal(t, []string{"Lock ", "Init ", "Plan -out #.tfplan", "PlanExists #.tfplan", "StorePlan #.tfplan", "LockId ", "PublishComment 1 <details>\n  <summary>Plan for ****</summary>\n\n  ```terraform\n\n  ```\n</details>", "LockId ", "Run   echo"}, commandStrings)
+	assert.Equal(t, []string{"Lock ", "Init ", "Plan -out #.tfplan", "PlanExists #.tfplan", "StorePlan #.tfplan", "LockId ", "Run   echo"}, commandStrings)
 }
 
 func allCommandsInOrderWithParams(terraformExecutor *MockTerraformExecutor, commandRunner *MockCommandRunner, prManager *MockPRManager, lock *MockProjectLock, planStorage *MockPlanStorage) []string {