Package: unoffical-athena-federation-sdk
(Unofficial) Python SDK for Athena FederationThis is anunofficialPython SDK for Athena Federation.OverviewThe Python SDK makes it easy to create new Amazon Athena Data Source Connectors using Python. It is under active development so the API may change from version to version.You can see an example implementation thatqueries Google Sheets using Athena.Current LimitationsPartitions are not supported, so Athena will not parallelize the query using partitions.Splits are not supported (but coming soon), so Athena will only make 1 request to query your data.Spill to S3 is not supported, so responses must be under 6MB.Local DevelopmentEnsure you've got thebuildmodule install and SDK dependencies.pip install build
pip install -r requirements.txtNow make a wheel.python-mbuildThis will create a file indist/:dist/unoffical_athena_federation_sdk-0.0.0-py3-none-any.whlCopy that file to your example repo and you can include it in yourrequirements.txtlike so:unoffical-athena-federation-sdk @ file:///unoffical_athena_federation_sdk-0.0.0-py3-none-any.whlValidating your connectorYou can test your Lambda function locally using Lambda Docker images.First, build our Docker image and run it.dockerbuild-tlocal/athena-python-example.
dockerrun--rm-p9000:8080local/athena-python-exampleThen, we can execute a samplePingRequest.curl-XPOST"http://localhost:9000/2015-03-31/functions/function/invocations"-d'{"@type": "PingRequest", "identity": {"id": "UNKNOWN", "principal": "UNKNOWN", "account": "123456789012", "arn": "arn:aws:iam::123456789012:root", "tags": {}, "groups": []}, "catalogName": "athena_python_sdk", "queryId": "1681559a-548b-4771-874c-2aa2ea7c39ab"}'{"@type":"PingResponse","catalogName":"athena_python_sdk","queryId":"1681559a-548b-4771-874c-2aa2ea7c39ab","sourceType":"athena_python_sdk","capabilities":23}We can also list schemas.curl-XPOST"http://localhost:9000/2015-03-31/functions/function/invocations"-d'{"@type": "ListSchemasRequest", "identity": {"id": "UNKNOWN", "principal": "UNKNOWN", "account": "123456789012", "arn": "arn:aws:iam::123456789012:root", "tags": {}, "groups": []}, "catalogName": "athena_python_sdk", "queryId": "1681559a-548b-4771-874c-2aa2ea7c39ab"}'{"@type":"ListSchemasResponse","catalogName":"athena_python_sdk","schemas":["sampledb"],"requestType":"LIST_SCHEMAS"}Creating your Lambda function💁Please note these are manual instructions until aserverless applicationcan be built.First, let's define some variables we need throughout.exportSPILL_BUCKET=<BUCKET_NAME>exportAWS_ACCOUNT_ID=123456789012exportAWS_REGION=us-east-1exportIMAGE_TAG=v0.0.1Create an S3 bucket that this Lambda function will use for Spill dataawss3mb${SPILL_BUCKET}Create an ECR repository for this imageawsecrcreate-repository--repository-nameathena_example--image-scanning-configurationscanOnPush=truePush tag the image with the repo name and push it updockertaglocal/athena-python-example${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/athena_example:${IMAGE_TAG}awsecrget-login-password|dockerlogin--usernameAWS--password-stdin${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com
dockerpush${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/athena_example:${IMAGE_TAG}Create an IAM role that will allow your Lambda function to executeNote theArnof the role that's returnedawsiamcreate-role\--role-nameathena-example-execution-role\--assume-role-policy-document'{"Version": "2012-10-17","Statement": [{ "Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]}'awsiamattach-role-policy\--role-nameathena-example-execution-role\--policy-arnarn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRoleGrant the IAM role access to your S3 bucketawsiamcreate-policy--policy-nameathena-example-s3-access--policy-document'{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Action": ["s3:ListBucket"],"Resource": ["arn:aws:s3:::'${SPILL_BUCKET}'"]},{"Effect": "Allow","Action": ["s3:PutObject","s3:GetObject","s3:DeleteObject"],"Resource": ["arn:aws:s3:::'${SPILL_BUCKET}'/*"]}]}'awsiamattach-role-policy\--role-nameathena-example-execution-role\--policy-arnarn:aws:iam::${AWS_ACCOUNT_ID}:policy/athena-example-s3-accessNow create your function pointing to the created repository imageawslambdacreate-function\--function-nameathena-python-example\--rolearn:aws:iam::${AWS_ACCOUNT_ID}:role/athena-example-execution-role\--codeImageUri=${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/athena_example:${IMAGE_TAG}\--environment'Variables={TARGET_BUCKET=<BUCKET_NAME>}'\--description"Example Python implementation for Athena Federated Queries"\--timeout60\--package-typeImageConnect with Athena!Choose "Data sources" on the top navigation bar in the Athena console and then click "Connect data source"Choose the Lambda function you just created and clickConnect!Updating the Lambda functionIf you update the Lambda function, re-run the build and push steps (updating theIMAGE_TAGvariable) and then update the Lambda function:awslambdaupdate-function-code\--function-nameathena-python-example\--image-uri${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/athena_example:${IMAGE_TAG}