Package: cloudimized
CloudimizedCloudimized is a Google Cloud Platform (GCP) configuration scanning tool. It allows
monitoring changes of selected resources.Cloudimized performs similar function as Oxidized, but for Cloud environment.OverviewCloudimized periodical scans of GCP resources via API calls and dumps them into yaml files. Files are tracked in Git,
so every configuration change is being tracked.
It gathers additional information for each change: changer's identity, related Terraform runs
(optionally), identify change ticket number (optionally).
Optionally it sends Slack notifications for each change.FeaturesProject discovery across GCP organizationIdentifying changer identityManual changes detectionIdentifying Terraform runsIdentifying change ticketsSlack notificationsFeature detailsWorkflowOn each execution Cloudimized performs following actions:Gets configuration from previous execution from Git remoteClears all configuration files from local repoPerforms resource readingFor all projects for all resources executes API call to get resource configuration(optional) Performs configured field and item filtering for each resourceDumps all results in yaml files format in local Git repoChecks local Git repo state and detects all changed resourceFor each detected change performs additional information gathering:Get GCP Logs for change to identify changer's identityIdentifies manual changeschanges performed directly by individual users as opposed to service accounts i.e. changes done outside of Terraform(optional) If change performed via Terraform Service Account, identify related Terraform runs.Get Terraform Runs URL(optional) Get ticket/issue marker from Terraform Run message. Generate ticket URL.Commit each individual change to Git repoContains configuration Git diffContains all additional gathered information(optional) Send Slack notification for each changeContaining same information as Git commitPush new commits to remote repoInstallationInstall Cloudimized withpipx(recommended) or plain pip.pipx install cloudimizedCloudimized for operation requires Git repo for storing GCP configuration files.Set-up empty Git repo in remote location i.e. GitHub or GitLabRunningAfter installation:Perform necessaryconfigurationSchedule periodic, regular script executionThis can be achieved via number of ways i.e.via cronvia automation serverExecute withcloudimized -c <PATH_TO_FILE>/config.yamlConfigurationService accountsGCP Service AccountGCP Service Account is used to perform resources scanning, log reading and project discovery. Below steps show setting up account with organization wide
reading permissions.Create dedicated Service Account in selected GCP projectOn organization level create custom role with permissions:logging.logEntries.list//required to identify changer<resource>.list//for each resource to be scannedi.e.compute.subnetworks.listfor scanning of VPC subnetwork resourceRefer to official Google docs formethodand itspermissionmappingOn organization level create IAM policy that binds Service Account with Roles: Browser and custom role you've createdThis will grant Service Account permission to discover all projects in Organzation, perform scanning of selected
resources in all projects and perform Logs scanning in all projects. If needed it is possible to limit permissions to
selected Folders and/or Projects.See:Google Cloud listing permissionsGIT Service AccountGIT Service Account is used to get configuration repository from remote and pushing to that repo detected configuration changes.
In your GIT remote you need to configure account/access token with proper permissions. Script allows communication via
both HTTPS and SSH.Terraform Service AccountTerraform Account is used to read Terraform Runs. Concept of Service Account doesn't exist in Terraform, so this is
performed in following way:Create Terraform Team with "Runs: Read Runs" permissionsFor each Terraform organization generate Team API token for your TeamSee:Terraform teamsTerraform permissionsTerraform API tokensSlack Service AccountSlack Service Account is used for sending detected change notification to Slack channel. This is done via Slack App.Create Slack app on Slack API pageSelectFrom scratchApp Name - enterCloudimizedWorkspace - select your workspaceInAdd features and funcionalityselectPermissionsInScopes->Bot Token Scopes, clickAdd an OAuth ScopesSelectfiles:writeInOAuth Tokens for Your Woskrpace, clickInstall to WorkspaceClickAllow accessBot User OAuth Tokenhas been generated, take a noteSelect/create channel to which notifications will be sentAdd App bot user by @mentioning App name in selected channelSee:Slack app setupJira Service AccountJira Service Account is used for creating issues(tickets) for selected changes. Depending on your Jira deployement
(Cloud vs Server), you need to setup an account with proper permissions to create Issues of selected type in given
project.See:Jira Cloud API tokenJira Software docsSecretsAll secrets required to run script need to be passed via env variables.GIT secretsCommunication with GIT remote can be both SSH and HTTPS based. If using SSH you need to take care of your SSH keys
configuration. If using HTTPS you need to pass credentials via following env variables:env varGIT_USR- usernameenv varGIT_PSW- password/tokenGoogle Service Account secretsCloudimized authenticates to GCP using GCP Service Account. After creating account you need to download JSON key file.
Passing credentials to script can be achieved either by setting env var or using Google's ADC mechanismenv varGOOGLE_APPLICATION_CREDENTIALSset to file path of service account's JSON key tokenorAuthenticate viagcloud auth application-default loginby providing service account's JSON key tokenSee:Best practices to securely authenticate applications in Google CloudAuthenticating as a service accountgcloud auth application-default loginTerraform secretsCloudimized authenticates to Terraform using Team token for each Terraform organization. Tokens are stored in JSON file
that only contains mapping between organization and team token value. File is passed to script by providing JSON file
location. This can be done by:env varTERRAFORM_READ_TOKENSset to file path of terraform's JSON mapping keyoroptionworkspace_token_filein configuration file set to file path of terraform's JSON mapping keySlack secretsCloudimized authenticates to Slack using Slack's Applications Bot User token. Token is passed to script via env varenv varSLACK_TOKENset to token valueJira secretsCloudimized authenticates to Jira using Username and Password/Token combination. Credentials are passed to script via
env varenv varJIRA_USR- Jira's usernameenv varJIRA_PSW- Jira's password/tokenConfiguration fileExample configuration file:services:# GCP API Service name - https://cloud.google.com/compute/docs/reference/rest/v1#service:-compute.googleapis.com-serviceName:compute# GCP API version - https://cloud.google.com/compute/docs/reference/rest/v1#service:-compute.googleapis.comversion:v1# List of all queries to send for this API Servicequeries:# User defined name of resource-resource:networks# GCP API method name - https://cloud.google.com/compute/docs/reference/rest/v1/networks/listgcp_api_call:networks.list# GCP API method parameters - https://cloud.google.com/compute/docs/reference/rest/v1/networks/list#path-parametersgcp_function_args:# Special Cloudimized value for setting ProjectID - script performs dynamic replacement with real Project IDproject:<PROJECT_ID># resource.type value in GCP Logs entries generated for this resource changes - https://cloud.google.com/logging/docs/api/v2/resource-list#tag_gce_networkgcp_log_resource_type:gce_network# Field name in response where resources are stored, (DEFAULT: items) - https://cloud.google.com/compute/docs/reference/rest/v1/networks/list#response-body# items: items# List of fields to exclude from configuration i.e. for status related fieldsfield_exclude_filter:# From each resource remove field "stateDetails" nested under "peerings"-peerings:# https://cloud.google.com/compute/docs/reference/rest/v1/networks/list#response-body-stateDetails# List of conditions to filter out whole individual resourcesitem_exclude_filter:# From results remove each resource that "name" under "peerings" matches "servicenetwork-googleapis-com"-peerings:name:"servicenetworking-googleapis-com"sortFields:# Define fields used to perform sorting of results## Sort items in results using 'name' field as key-name## Sort inner list (under key 'secondaryIpRanges') in each item using 'name' field as key-secondaryIpRanges:namegit:# Git repo's URL for GCP configuration storingremote_url:https://github.com/<ORG>/<REPO_NAME># Local directory for storing GCP configurationlocal_directory:gcp_config# Perform dynamic discovery of all projects in GCP organizationdiscover_projects:True# Static list of project IDs (alternative to the above approach)#project_list:#  # GCP project ID#  - my-project-ID# List of project IDs to exclude from scanning - for use with dynamic discoeryexcluded_projects:-excluded-project-ID# Number of threads for scanning resourcesthread_count:4# default - 3# Change handling configurationchange_processor:# Interval (in minutes) between each scan - has to match script execution intervalscan_interval:30# Regex to identify service account username - match meaning non-manual changeservice_account_regex:'^(my-terraform-sa-|\d+@|service-\d+).*'# Regex to identify ticket/issue from Terraform Run message i.e. ADR-1234ticket_regex:"^.*?([a-zA-z]{2,3}[-_][0-9]+).*"# Ticket/Issue URL base in ticketing system. Used to create ticket linkticket_sys_url:"https://my-tickets.com/list"# Slack notifications configslack:# Slack channel ID for Cloudimized notificationschannelID:"C123456789A"# Commit URL base in Git system. Used to create commit linkrepoCommitURL:"https://github.com/<ORG>/<REPO_NAME>/commit"# Jira issue creator configjira:# Jira's URLurl:"https://my.jira.com"# Jira's Project Key - Project in which create issueprojectKey:"KEY"# Jira's Issue Type - Issue's type to be created (optional)issueType:"Task"#default# Jira's Issue fields - set values on fields in Issue (optional)fields:field_name:"field_value"# Regex filter for selecting projects for which create issues (optional)filterSet:projectId:".*production-only.*"# Terraform Runs configurationterraform:# Terraform URLurl:"https://app.terraform.io"# Path to JSON file containing Organization to team token mapping#workspace_token_file: "<path-to-file>"# Mapping between GCP Service Account performing changes and Terraform Org/Workspaceservice_workspace_map:# GCP Service Account nameterraform-service-account-no1:# Terraform Organizationorg:my-organization# Terraform workspaces listworkspace:["my-workspace-no1"]Single run modeAllows to run cloudimized only to scan given resource and dump them into text files, without performing and additional
functions (no Git, Terraform, Slack, Jira interaction and no GCP logs lookup).Runningcloudimized --singlerun/-s <RESOURCE_NAME> --output/-o {yaml, csv}

i.e
cloudimized -s addresses -o csvConfigurationResource configurations for single run (<RESOURCE_NAME>parameter) to be scanned are stored insinglerunconfigsdirectory and are selected based on filename. Resource configuration is the same as in main config file. Additional
singe run mode configs can be added to folder as needed.Get info available configs or what will be run with <RESOURCE_NAME> with:cloudimized -s --list
cloudimized -s --describe --name <RESOURCE_NAME>OutputBy default script will dump results in YAML format same as in main mode. If chosen it can dump results in CSV file
format (single file per resource).