Package: echofish-aws-raw-to-zarr-lambda
echofish-aws-raw-to-zarr-lambdaSetting up the Python EnvironmentMacOSInstall pyenv (https://github.com/pyenv/pyenv#set-up-your-shell-environment-for-pyenv)brew updatebrew install pyenvIn ~/.bashrc addexport PYENV_ROOT="$HOME/.pyenv"export PATH="$PYENV_ROOT/bin:$PATH"eval "$(pyenv init -)"brew install openssl readline sqlite3 xz zlib tcl-tkInstall pyenv-virtualenv (https://github.com/pyenv/pyenv-virtualenv)brew install pyenv-virtualenvIn ~/.bashrc addeval "$(pyenv virtualenv-init -)"Open a new terminalInstall Python versionenv PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.9.16Optional: set global version of Pythonenv PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv global 3.9.16Create virtual envpyenv virtualenv 3.9.16 echofish-aws-raw-to-zarr-lambdaSet local version of python (if not done already)change directory to root of projectpyenv local echofish-aws-raw-to-zarr-lambdaOther OSGet a MacSee Above MacOSSetting up IntelliJInstall the IntelliJ Python pluginSet up pyenvFile -> Project Structure or CMD + ;SDKs -> + -> Add Python SDK -> Virtual EnvironmentSelect Existing EnvironmentChoose ~/.pyenv/versions/echofish-aws-raw-to-zarr-lambda/bin/pythonSet up Python Facet (not sure if this is required)File -> Project Structure or CMD + ;Facets -> + -> PythonSet interpreterInstalling DependenciesAdd dependencies with versions to requirements.txtpip install --upgrade pip && pip install -r requirements_dev.txtMavenThis project can use Apache Maven to easily build this project.  However, this is not required, but recommended.
Maven can be easily installed by downloading it from the Maven site or using sdkman.Build with MavenAfter setting up your pyenv run:mvn clean installDOTENVexport AWS_DEFAULT_PROFILE=echofishFunctional TestingTo run a functional test, developer will need to save the associated environment variables in .env for the lambda to
use. The ACCESS_KEY and SECRET_ACCESS_KEY can be generated in IAM, users, rudy-dev, security credentials,
access keys, create access key. Copy other values as needed from deployed CloudFormation Template.exportOUTPUT_BUCKET=rudy-dev-echofish-118234403147-echofish-dev-outputexportTABLE_NAME=rudy-dev-echofish-EchoFish-File-InfoexportOUTPUT_BUCKET_ACCESS_KEY="XXX"exportOUTPUT_BUCKET_SECRET_ACCESS_KEY="YYY"exportTOPIC_ARN="arn:aws:sns:us-west-2:118234403147:rudy-dev-echofish-processing-finished"Deploying new CodePipeline via CloudformationBuild project w 'mvn clean install', navigate to the 'target/codepipeline' directory. You will need to
create a private ECR repo in us-east-1 w name of the project for builds to push to. Then:aws --profile mggdev cloudformation create-stack --stack-name build-echofish-aws-raw-to-zarr-lambda-trunk --template-body file://codepipeline.yaml --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPANDElastic Container RegistryNote that the main repository is built and curated in the mgg-dev account in us-west-2. The derived docker image is built and pushed to the echofish us-east-1 ECR.
That image is then replicated to the echofish us-west-2 account. You will need to go to the echofish ECR -> Repositories -> [repo] -> Permissions and copy the values from an existing project (for both regions).In the echofish aws account:IAM -> Roles -> AWSServiceRoleForECRReplicationTo Run Testspython -m pytest