Package: cloud2sql
Cloud2SQL 🤩Read infrastructure data from your cloud ☁️ and export it to a SQL database 📋.InstallationInstall via homebrewThis is the easiest way to install Cloud2SQL.
Please note, that the installation process will take a couple of minutes.brewinstallsomeengineering/tap/cloud2sqlInstall via Python pipAlternatively you can install Cloud2SQL as Python package, where Python 3.9 or higher is required.If you only need support for a specific database, instead ofcloud2sql[all]you can choose betweencloud2sql[snowflake],cloud2sql[parquet],cloud2sql[postgresql],cloud2sql[mysql].pip3install--user"cloud2sql[all]"This will install the executable to the user install directory of your platform. Please make sure this installation directory is listed inPATH.UsageThe sources and destinations forcloud2sqlare configured via a configuration file. Create your own configuration by adjusting theconfig template file.You can safely delete the sections that are not relevant to you (e.g. if you do not use AWS, you can delete theawssection).
All sections refer to cloud providers and are enabled if a configuration section is provided.In the next section you will create a YAML configuration file. Once you have created your configuration file, you can runcloud2sqlwith the following command:cloud2sql--configmyconfig.yamlConfigurationCloud2SQL uses a YAML configuration file to define thesourcesanddestinations.SourcesAWSsources:aws:# AWS Access Key ID (null to load from env - recommended)access_key_id:null# AWS Secret Access Key (null to load from env - recommended)secret_access_key:null# IAM role name to assumerole:null# List of AWS profiles to collectprofiles:null# List of AWS Regions to collect (null for all)region:null# Scrape the entire AWS organizationscrape_org:false# Assume given role in current accountassume_current:false# Do not scrape current accountdo_not_scrape_current:falseGoogle Cloudsources:gcp:# GCP service account file(s)service_account:[]# GCP project(s)project:[]Kubernetessources:k8s:# Configure access via kubeconfig files.# Structure:#   - path: "/path/to/kubeconfig"#     all_contexts: false#     contexts: ["context1", "context2"]config_files:[]# Alternative: configure access to k8s clusters directly in the config.# Structure:#   - name: 'k8s-cluster-name'#     certificate_authority_data: 'CERT'#     server: 'https://k8s-cluster-server.example.com'#     token: 'TOKEN'configs:[]DigitalOceansources:digitalocean:# DigitalOcean API tokens for the teams to be collectedapi_tokens:[]# DigitalOcean Spaces access keys for the teams to be collected, separated by colonsspaces_access_keys:[]DestinationsSQLitedestinations:sqlite:database:/path/to/database.dbPostgreSQLdestinations:postgresql:host:127.0.0.1port:5432user:cloud2sqlpassword:changemedatabase:cloud2sqlargs:key:valueMySQLdestinations:mysql:host:127.0.0.1port:3306user:cloud2sqlpassword:changemedatabase:cloud2sqlargs:key:valueMariaDBdestinations:mariadb:host:127.0.0.1port:3306user:cloud2sqlpassword:changemedatabase:cloud2sqlargs:key:valueSnowflakedestinations:snowflake:host:myorg-myaccountuser:cloud2sqlpassword:changemedatabase:cloud2sql/publicargs:warehouse:compute_whrole:accountadminApache Parquetdestinations:file:path:/where/to/write/parquet/files/format:parquetbatch_size:100_000CSVdestinations:file:path:/where/to/write/to/csv/files/format:csvbatch_size:100_000Upload to S3destinations:s3:uri:s3://bucket_name/region:eu-central-1format:csvbatch_size:100_000Upload to Google Cloud Storagedestinations:gcs:uri:gs://bucket_name/format:parquetbatch_size:100_000My database is not listed hereCloud2SQL uses SQLAlchemy to connect to the database. If your database is not listed here, you can check if it is supported inSQLAlchemy Dialects.
Install the relevant driver and use the connection string from the documentation.ExampleWe use a minimal configurationexampleand export the data to a SQLite database.
The example uses our AWS default credentials and the default kubernetes config.cloud2sql--configconfig-example.yamlFor a more in-depth example, check out ourblog post.Local DevelopmentCreate a local development environment with the following command:makesetupsourcevenv/bin/activate