Repository: hashicorp/terraform-cdk

Issue #3622: CDKTF diff/plan not acting as expected in air gapped environment
Created: 2024-05-16T14:14:26Z
Expected Behavior
.terraform.lock.hcl contains only hashes that are in the local mirror and does not reach out to registry.terraform.io
Actual Behavior
.terraform.lock.hcl contains all available hashes for used providers and reaches out to registry.terraform.io
Steps to Reproduce

clone https://github.com/jej2003/sample_cdktf
run docker build -t sample_cdk .
docker run -it --entrypoint bash sample_cdk
cdktf diff test
cat cdktf.out/stacks/test/.terraform.lock.hcl
you'll see all hashes are here vs just the ones configured in the mirror

# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/aws" {
  version     = "5.49.0"
  constraints = "5.49.0"
  hashes = [
    "h1:Y3xvYjzBIwYSbcnZDcs6moiy30uxRoY5oT2ExQHKG5A=",
    "zh:0979b07cdeffb868ea605e4bbc008adc7cccb5f3ba1d3a0b794ea3e8fff20932",
    "zh:2121a0a048a1d9419df69f3561e524b7e8a6b74ba0f57bd8948799f12b6ad3a1",
    "zh:573362042ba0bd18e98567a4f45d91b09eb0d223513518ba04f16a646a906403",
    "zh:57be7a4d6c362be2fa586d270203f4eac1ee239816239a9503b86ebc8fa1fef0",
    "zh:5c72ed211d9234edd70eac9d77c3cafc7bbf819d1c28332a6d77acf227c9a23c",
    "zh:7786d1a9781f8e8c0079bf58f4ed4aeddec0caf54ad7ddcf43c47936d545a04f",
    "zh:82133e7d39787ee91ed41988da71beecc2ecb900b5da94b3f3d77fbc4d4dc722",
    "zh:8cdb1c154dead85be8352afd30eaf41c59249de9e7e0a8eb4ab8e625b90a4922",
    "zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425",
    "zh:ac215fd1c3bd647ae38868940651b97a53197688daefcd70b3595c84560e5267",
    "zh:c45db22356d20e431639061a72e07da5201f4937c1df6b9f03f32019facf3905",
    "zh:c9ba90e62db9a4708ed1a4e094849f88ce9d44c52b49f613b30bb3f7523b8d97",
    "zh:d2be3607be2209995c80dc1d66086d527de5d470f73509e813254067e8287106",
    "zh:e3fa20090f3cebf3911fc7ef122bd8c0505e3330ab7d541fa945fea861205007",
    "zh:ef1b9d5c0b6279323f2ecfc322db8083e141984cfe1bb2f33c0f4934fccb69e3",
  ]
}

Versions
language: typescript
cdktf-cli: 0.20.0
node: 20.11.1
cdktf: 0.20.0
terraform: 1.8.0
arch: x86_64
os: RHEL UBI 8 (docker)
Providers
│ Provider Name      │ Provider Version │ CDKTF   │ Constraint │ Package Name           │ Package Version │
│ aws                │ 5.44.0           │ ^0.20.0 │            │ @cdktf/provider-aws    │ 19.0.0          │
Gist
https://github.com/jej2003/sample_cdktf/blob/main/output.log
Possible Solutions
No response
Workarounds
cd into cdktf.out/stacks/test and run terraform init manually
Anything Else?
For additional context, the requirement I am trying to satisfy is supporting an air gapped install.  Reaching out to registry.terraform.io is not possible from the target environment and a plan can't be done outside of the target environment.  I am looking for the best solution to eliminate the last step that is indicated as follows
test  - Fetching hashicorp/aws 5.49.0 for linux_amd64...
test  - Retrieved hashicorp/aws 5.49.0 for linux_amd64 (signed by HashiCorp)
      - Obtained hashicorp/aws checksums for linux_amd64; Additional checksums for this platform are now tracked in the lock file

interestingly, if I run cdktf diff test && cd cdktf.out/stacks/test && terraform init I don't see terraform reaching out to do this retrieve and the .terraform.lock.hcl file only contains what is in my local mirror.  What is cdktf doing differently?
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/aws" {
  version     = "5.49.0"
  constraints = "5.49.0"
  hashes = [
    "h1:Y3xvYjzBIwYSbcnZDcs6moiy30uxRoY5oT2ExQHKG5A=",
  ]
}

References
No response
Help Wanted

 I'm interested in contributing a fix myself

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Duplicate issue #3737: COMPONENT: CDKTF fetching providers on every deploy
Created: 2024-10-02T18:14:42Z
Expected Behavior
CDKTF should not fetch providers if they already exist
Actual Behavior
CDKTF fetches providers on every deploy, even subsequent deploys that did not change providers
Steps to Reproduce
Not sure how to reliably reproduce.
Versions
 ❯❯❯ npx cdktf debug
cdktf debug
language: typescript
cdktf-cli: 0.20.8
node: v18.17.0
cdktf: 0.20.8
constructs: 10.3.0
jsii: null
terraform: 1.9.3
arch: arm64
os: darwin 23.6.0
providers
aws@~> 5.61 (LOCAL)
        terraform provider version: 5.61.0
kubernetes@ ~> 2.31.0 (LOCAL)
        terraform provider version: 2.31.0
http@ ~> 2.1.0 (LOCAL)
        terraform provider version: 2.1.0
tls@ ~> 4.0.5 (LOCAL)
        terraform provider version: 4.0.5
random@ ~> 3.1.0 (LOCAL)
        terraform provider version: 3.1.3
gavinbunney/kubectl@~> 1.14.0 (LOCAL)
        terraform provider version: 1.14.0
DopplerHQ/doppler@1.1.6 (LOCAL)
        terraform provider version: 1.1.6
helm@undefined (LOCAL)
        terraform provider version: 2.14.0
@cdktf/provider-aws (PREBUILT)
        terraform provider version: 5.61.0
        prebuilt provider version: 19.29.0
        cdktf version: ^0.20.0
@cdktf/provider-kubernetes (PREBUILT)
        terraform provider version: 2.31.0
        prebuilt provider version: 11.6.0
        cdktf version: ^0.20.0

Providers
No response
Gist
No response
Possible Solutions
No response
Workarounds
No response
Anything Else?
Looks like this was addressed in 0.15.3 but I'm on 0.20.0 and I'm still seeing it. Fetching these providers can be very slow. Here are some logs I see
                    - Reusing previous version of hashicorp/random from the dependency lock file
goldsky-infra-prod  - Reusing previous version of hashicorp/aws from the dependency lock file
goldsky-infra-prod  - Reusing previous version of hashicorp/archive from the dependency lock file
goldsky-infra-prod  - Reusing previous version of hashicorp/helm from the dependency lock file
goldsky-infra-prod  - Reusing previous version of hashicorp/tls from the dependency lock file
goldsky-infra-prod  - Reusing previous version of hashicorp/http from the dependency lock file
goldsky-infra-prod  - Reusing previous version of hashicorp/cloudinit from the dependency lock file
goldsky-infra-prod  - Reusing previous version of hashicorp/kubernetes from the dependency lock file
goldsky-infra-prod  - Reusing previous version of dopplerhq/doppler from the dependency lock file
goldsky-infra-prod  - Using previously-installed dopplerhq/doppler v1.1.6
goldsky-infra-prod  - Using previously-installed hashicorp/random v3.1.3
goldsky-infra-prod  - Using previously-installed hashicorp/cloudinit v2.3.5
goldsky-infra-prod  - Using previously-installed hashicorp/helm v2.14.0
goldsky-infra-prod  - Using previously-installed hashicorp/tls v4.0.5
goldsky-infra-prod  - Using previously-installed hashicorp/http v2.1.0
goldsky-infra-prod  - Using previously-installed hashicorp/kubernetes v2.31.0
goldsky-infra-prod  - Using previously-installed hashicorp/aws v5.61.0
goldsky-infra-prod  - Using previously-installed hashicorp/archive v2.6.0
goldsky-infra-prod  Terraform has been successfully initialized!

                    You may now begin working with Terraform. Try running "terraform plan" to see
                    any changes that are required for your infrastructure. All Terraform commands
                    should now work.

                    If you ever set or change modules or backend configuration for Terraform,
                    rerun this command to reinitialize your working directory. If you forget, other
                    commands will detect it and remind you to do so if necessary.
goldsky-infra-prod  - Fetching dopplerhq/doppler 1.1.6 for linux_amd64...
goldsky-infra-prod  - Retrieved dopplerhq/doppler 1.1.6 for linux_amd64 (self-signed, key ID D13E9DC04ACCB5E6)
goldsky-infra-prod  - Fetching hashicorp/http 2.1.0 for linux_amd64...
goldsky-infra-prod  - Retrieved hashicorp/http 2.1.0 for linux_amd64 (signed by HashiCorp)
goldsky-infra-prod  - Fetching hashicorp/random 3.1.3 for linux_amd64...
goldsky-infra-prod  - Retrieved hashicorp/random 3.1.3 for linux_amd64 (signed by HashiCorp)
goldsky-infra-prod  - Fetching hashicorp/archive 2.6.0 for linux_amd64...
goldsky-infra-prod  - Retrieved hashicorp/archive 2.6.0 for linux_amd64 (signed by HashiCorp)
goldsky-infra-prod  - Fetching hashicorp/kubernetes 2.31.0 for linux_amd64...
goldsky-infra-prod  - Retrieved hashicorp/kubernetes 2.31.0 for linux_amd64 (signed by HashiCorp)
goldsky-infra-prod  - Fetching hashicorp/helm 2.14.0 for linux_amd64...
goldsky-infra-prod  - Retrieved hashicorp/helm 2.14.0 for linux_amd64 (signed by HashiCorp)
goldsky-infra-prod  - Fetching hashicorp/cloudinit 2.3.5 for linux_amd64...
goldsky-infra-prod  - Retrieved hashicorp/cloudinit 2.3.5 for linux_amd64 (signed by HashiCorp)
goldsky-infra-prod  - Fetching hashicorp/aws 5.61.0 for linux_amd64...
goldsky-infra-prod  - Retrieved hashicorp/aws 5.61.0 for linux_amd64 (signed by HashiCorp)
goldsky-infra-prod  - Fetching hashicorp/tls 4.0.5 for linux_amd64...
goldsky-infra-prod  - Retrieved hashicorp/tls 4.0.5 for linux_amd64 (signed by HashiCorp)
goldsky-infra-prod  - Obtained hashicorp/cloudinit checksums for linux_amd64; All checksums for this platform were already tracked in the lock file
                    - Obtained hashicorp/archive checksums for linux_amd64; All checksums for this platform were already tracked in the lock file
                    - Obtained hashicorp/http checksums for linux_amd64; All checksums for this platform were already tracked in the lock file
                    - Obtained hashicorp/kubernetes checksums for linux_amd64; All checksums for this platform were already tracked in the lock file
                    - Obtained hashicorp/tls checksums for linux_amd64; All checksums for this platform were already tracked in the lock file
                    - Obtained hashicorp/random checksums for linux_amd64; All checksums for this platform were already tracked in the lock file
                    - Obtained hashicorp/aws checksums for linux_amd64; All checksums for this platform were already tracked in the lock file
                    - Obtained dopplerhq/doppler checksums for linux_amd64; All checksums for this platform were already tracked in the lock file
                    - Obtained hashicorp/helm checksums for linux_amd64; All checksums for this platform were already tracked in the lock file

References

#2525

Help Wanted

 I'm interested in contributing a fix myself

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Closing comment (2024-10-15T21:06:52Z): I'm also experiencing the same issue on my side, it makes the init stage unnecessary slow.
I believe this issue is also related to #3622