Package: localstack-extension-aws-replicator
AWS Replicator Extension (experimental)A LocalStack extension to replicate AWS resources into your local machine.⚠️ Please note that this extension is experimental and currently under active development.PrerequisitesLocalStack ProDockerPythonOverviewThis extension currently offers two modes of operation: (1) the AWS connection proxy, and (2) the resource replicator CLI.AWS Connection ProxyThe AWS connection proxy can be used to forward certain API calls in LocalStack to real AWS, in order to enable seamless transition between local and remote resources.Warning:Be careful when using the proxy - make sure tonevergive access to production accounts or any critical/sensitive data!Note:The replicator CLI currently works only when installing thelocalstackCLI viapip. If you're downloading thelocalstackCLI as abinary release, then please use the proxy configuration UI described below.UsageCLIFor example, in order to forward all API calls for DynamoDB/S3/Cognito to real AWS, the proxy can be started via the CLI as follows:Start LocalStack via CLI$ localstack start -dEnable LocalStack AWS replicator from the Web Application Extension LibraryAfter installation restart LocalstackInstall the AWS replicator CLI package$ pip install localstack-extension-aws-replicatorConfigure real cloud account credentials in a new terminal session to allow access$ export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=...Start proxy in aforementioned terminal session via the CLI$ localstack aws proxy -s dynamodb,s3,cognito-idpNow, when issuing an API call against LocalStack (e.g., viaawslocal), the invocation gets forwarded to real AWS and should return data from your real cloud resources.Proxy Configuration UIStart Localstack with extra CORSEXTRA_CORS_ALLOWED_ORIGINS=https://aws-replicator.localhost.localstack.cloud:4566 localstack start -dEnable Localstack AWS replicator from the Web Application Extension LibraryOnce the extension is installed, it will expose a small configuration endpoint in your LocalStack container under the following endpoint:http://localhost:4566/_localstack/aws-replicator/index.html.Use this Web UI to define the proxy configuration (in YAML syntax), as well as the AWS credentials (AWS access key ID, secret access key, and optionally session token) and save configuration. The proxy should report enabled state and on the host a proxy container should spawn.Now we can communicate with the real AWS cloud resources, directly via LocalStack.To clean up the running proxy container simply click "disable" on the Replicator UI.Resource-specific proxyingAs an alternative to forwardingallrequests for a particular service, you can also proxy only requests forspecificresources to AWS.For example, assume we own an S3 bucketmy-s3-bucketin AWS, then we can use the following configuration to forward any requests tos3://my-s3-bucketto real AWS, while still handling requests to all other buckets locally in LocalStack:services:
  s3:
    resources:
      # list of ARNs of S3 buckets to proxy to real AWS
      - '.*:my-s3-bucket'
    operations:
      # list of operation name regex patterns (optional)
      - 'Get.*'
      - 'Put.*'
    # optionally, specify that only read requests should be allowed (Get*/List*/Describe*, etc)
    read_only: falseStore the configuration above to a file namedproxy_config.yml, then we can start up the proxy via:localstack aws proxy -c proxy_config.ymlIf we then perform local operations against the S3 bucketmy-s3-bucket, the proxy will forward the request and will return the results from real AWS:$ awslocal s3 ls s3://my-s3-bucket
2023-05-14 15:53:40        148 my-file-1.txt
2023-05-15 10:24:43         22 my-file-2.txtAny other S3 requests targeting other buckets will be run against the local state in LocalStack itself, for example:$ awslocal s3 mb s3://test123
make_bucket: test123
...A more comprehensive sample, involving local Lambda functions combined with remote SQS queues and S3 buckets, can be found in theexamplefolder of this repo.ConfigurationIn addition to the proxy services configuration shown above, the following configs can be used to customize the behavior of the extension itself (simply pass them as environment variables to the main LocalStack container):REPLICATOR_CLEANUP_PROXY_CONTAINERS: whether to clean up (remove) the proxy Docker containers once they shut down (default1). Can be set to0to help debug issues, e.g., if a proxy container starts up and exits immediately.REPLICATOR_LOCALSTACK_HOST: the target host to use when the proxy container connects to the LocalStack main container (automatically determined by default)REPLICATOR_PROXY_DOCKER_FLAGS: additional flags that should be passed when creating the proxy Docker containersResource Replicator CLIThe figure below illustrates how the extension can be used to replicate the state, e.g., an SQS queue and the messages contained in it, from AWS into your LocalStack instance.To use the resource replicator, make sure that you have access to AWS configured in your terminal. Note: the extension will only talk to AWS in read-only mode, and willnotmake any changes to your real AWS account.The following command can be used to replicate SQS queues (incl. their messages) into your LocalStack instance:$ localstack aws replicate -s sqsOnce the command has completed, you should be able to list and interact with the queue that was replicated into your local account:$ awslocal sqs list-queues
...
$ awslocal sqs receive-message --queue-url ...
...InstallingTo install the CLI extension, use the followingpipcommand:pipinstall"git+https://github.com/localstack/localstack-extensions/#egg=localstack-extension-aws-replicator&subdirectory=aws-replicator"To install the extension itself (server component running inside LocalStack), use the followingextensionscommand:localstackextensionsinstall"git+https://github.com/localstack/localstack-extensions/#egg=localstack-extension-aws-replicator&subdirectory=aws-replicator"Change Log0.1.10: AddREPLICATOR_PROXY_DOCKER_FLAGSoption to pass custom flags to proxy Docker containers0.1.9: Enhance proxy networking and addREPLICATOR_LOCALSTACK_HOSTconfig option0.1.8: AddREPLICATOR_CLEANUP_PROXY_CONTAINERSoption to skip removing proxy containers for debugging0.1.7: Add rolo dependency to tests0.1.6: Adjust config to supportLOCALSTACK_AUTH_TOKENin addition to legacy API keys0.1.5: Minor fix to accommodate recent upstream changes0.1.4: Fix imports ofbootstrap.authmodules for v3.0 compatibility0.1.3: Adjust code imports for recent LocalStack v3.0 module changes0.1.2: Remove deprecated ProxyListener for starting local aws-replicator proxy server0.1.1: Add simple configuration Web UI0.1.0: Initial version of extensionLicenseThis extension is published under the Apache License, Version 2.0.
By using it, you also agree to the LocalStackEnd-User License Agreement (EULA).