Package: tgwrap
tg-wrapThis app simply wraps terragrunt (which is a wrapper around terraform, which is a wrapper around cloud APIs, which is...).Wait, why on earth do we need a wrapper for a wrapper (for a wrapper)?Well, first of all it is pretty opinionated so what works for us, doesn't necessarily work for you.But our reasoning for creating this is as follows:1. Less typingterraform is great, and in combination with terragrunt even greater! But let's face it, terragrunt does not excel in conciseness! The options are pretty long, which leads to lots of typing. We don't like typing!2. Testing modules locallyHowever, more importantly, we are heavily utilisingTERRAGRUNT_SOURCEwhen developing.The thing is that as long as you userun-allyou can use one setting for that variable (and conveniently set it as an environment variable), while if you run a regular command, you need to specify the full path. Which is obviously different for each project.Which leads to (even) more typing, and worse: a higher chance for errors.Luckily you can userun-alland add the appriopriate flags to ensure it behaves like a regular plan|apply|destroy etc. But again, more typing.Nothing abunch a aliasescan't solve though!3. But the original reason was: Errors when using run-all are challengingOne of the main boons of terragrunt is the ability to break up large projects in smaller steps while still retaining the inter-dependencies. However, when working on such a large project and something goes wrong somewhere in the middle is pretty challenging.terragrunt's error messages are pretty massive, and this is extrapolated with every individual project in your dependency chain.And if it fails somewhere at the front, it keeps on trying until the last one, blowing up your terminal in the process.So we wanted a possibility to run the projects step by step, using the dependency graph of terragrunt and have a bit more control over it.And when you can run it step by step, you can make the process also re-startable, which is also pretty handy!And this was not something a bunch of aliases could solve, hence this wrapper was born. And while we we're at it, replacing the aliases with this was then pretty straightforward next step as well.4. Analyzing plan filesWhen using the run-all, analyzing what is about to be changed is not going to be easier. Hence we created thetgwrap analyzefunction that lists all the planned changes and (if a config file is availabe) calculates a drift score and runs aterrasafestyle validation check.It needs a config file as follows:---## Critically of resources as interpreted by 'tgwrap analyze'.# It uses it for a 'terrasafe' like validation if resources can safely be deleted.# On top of that it tries to analyze and quantify the drift impact of the changes,# so that this can be monitored.#low:# defaults:#   terrasafe_level: ignore_deletions#   drift_impact:#     default: minor#     delete: mediumazuread_application.:{}# if we you want to use the defaultsazuread_app_role_assignment:# or if you want to override thesedrift_impact:delete:minor# and so on, and so forthmedium:# defaults:#   terrasafe_level: ignore_deletion_if_recreation#   drift_impact:#     default: medium#     delete: majorazurerm_data_factory_linked_service_key_vault.:{}# and so on, and so forthhigh:# defaults:#   terrasafe_level: unauthorized_deletion#   drift_impact:#     default: major#     update: mediumazuread_group.:drift_impact:create:minorupdate:minorazurerm_application_insights.:{}# and so on, and so forthSpeeding up the performance of analyzeThisanalyzefunction turned out to be pretty slow, where most of the time went into theterragrunt showfunction that is executed for each individual module.This was a bit surprising as the plan file is already available on the file system, but it turns out that terragrunt is taking quite a bit of time for managing the depdencies. Even when you're excluding the external dependencies and are located in a particular module.So, if you add the following to your rootterragrunt.hcl:terraform{after_hook"link_to_current_module"{commands=["init", "plan", "apply", "validate", "destroy"]execute=["bash", "-c", "ln -sf $(pwd) ${get_terragrunt_dir()}/.terragrunt-cache/current"]}}The directory where the plan file is stored (including the other resources that terraform needs) becomes predictable and it becomes possible to run a nativeterraform show(insteadterragrunt show) which dramatically speed up things.Just set the proper value as an environment variable:export TGWRAP_PLANFILE_DIR=".terragrunt-cache/current"Or pass it along with the--planfile-dir|-Poption and it will use that.More than a wrapperOver time, tgwrap became more than a wrapper, blantly violating#1 of the unix philosophy: 'Make each program do one thing well'.For instance, the 'analyze' functionality is already an example, but more features such as deploying a landing zone has crept into the application. It makes sense for how we're using it, but we're fully aware this makes it less generically applicable.Usage#generalhelptgwrap --helptgwrap run -htgwrap run-all -h#runaplantgwrap plan # which is the same as tgwrap run plan#run-allaplantgwrap run-all plan#run-allwithexcludingaparticulardirectorytgwrap run-all plan -E 'excluded-dir/*'#oradirectorysomewherefurtherdowninthepathtgwrap run-all plan -E '**/excluded-dir/**'#ordothesameinstep-by-stepmodetgwrap run-all plan -s#orexcluding(akaignoring)externaldependenciestgwrap run-all plan -sx#ifyouwanttoaddadditionalargumentsitisrecommendedtouse--asseparator(althoughit*might*workwithout)tgwrap output -- -jsonNote: special precautions are needed when passing on parameters that contain quotes. For instance, if you want to move state like below, escape the double quote in the staate address:tgwrap state mv 'azuread_group.this[\"viewers\"]' 'azuread_group.this[\"readers\"]'A word about escaping inputsYour shell is escaping special characters such as*and"before passing it to the program (tgwrapin this case). So some inputsneedto be escaped in order to function properly.For example:#toexcludecertainmodulesfromanaction(suchasanalyze):tgwrap analyze -E 'integrations/\*/\*'#toimportaresourcethathasa" in its address:tgwrap import -a 'azuread_group.this[\"my_group\"]' -i ${GROUP_ID}Deploy manifestsIn order to easily deploy a new version of the terraform (and associated terragrunt) modules, we include a manifest file in the root of the landing zone:---git_repository:ssh://git@gitlab.com/my-org/my-terraform-modules-repo.gitbase_path:terragrunt/my-platform# path where the terragrunt modules are storedconfig_path:terragrunt/config/platform-dev# path (relative to base path) where the config files are storeddeploy:# which modules do you want to deploydtap:applies_to_stages:-dev-tst-acc-prdsource_stage:devsource_dir:platform# optional, if the source modules are not directly in the stage dir, but in <stage>/<source_dir> directorybase_dir:platform# optional, if you want to deploy the base modules in its own dir, side by side with substacksconfig_dir:../../../config# this is relative to where you run the deployconfigs:-my-config.hcl-../my-ss-config-dir# skip_all_modules: True # could be handy if you only want to deploy substacks and no regular modulesexclude_modules:# these modules will always be excluded, can be omitted-my-specific-moduleinclude_modules:{}# omit or use an empty dict for all of them# or specify your modules as follows# base: {} # just a simple include# networking-connected: # or a bit more complicated#  - source: networking#  - target: networking-connectedsubstacks:is01:source:shared-integration/intsvc01target:integration/is01exclude_modules:# a list of modules that will always be excluded, can be omitted-my-specific-moduleconfigs:-my-ss-config.hcl-../my-ss-config-diris02:applies_to_stages:# optional list of stages to include the substack in-devsource:shared-integration/intsvc01target:integration/is02# global configuration files that are deployed as well# note that these files are typically applicable to all landing zones and stages!# this might lead to unexpected behaviour# note that you can exclude syncing these files with a command line switchglobal_config_files:root-terragrunt:source:../../terragrunt.hcl# relative to base_pathtarget:../../terragrunt.hcl# can be omitted, then it is same as source pathterrasafe-config:source:../../terrasafe-config.jsonGenerating change logstgwrap can generate a change log by running:tgwrap change-log [--changelog-file ./CHANGELOG.md]The (optional) change log file will be, if passed along. tgwrap then checks the file for the existance of a start and end markers, in the following format:start_marker='<!-- BEGINNING OF OF TGWRAP CHANGELOG SECTION -->'end_marker='<!-- END OF TGWRAP CHANGELOG SECTION -->'If they exist, everything between these lines will be replaced by the new change log.DevelopmentIn order to develop, you need to apply it to your terragrunt projects. For that you can use the--terragrunt-working-diroption and just run it from the poetry directory. Alternatively you can use thetgwrap-devscript and invoke that from your terragrunt directories. Either put it in yourPATHor create an alias for convenience.