Package: aioftps3
aioftps3FTP in front of AWS S3, usingasyncio, andaiohttp. Only a subset of the FTP protocol is supported, with implicit TLS and PASV mode; connections will fail otherwise.Installationpipinstallaioftps3An SSL key and certificate must be present$HOME/ssl.keyand$HOME/ssl.crtrespectively. To create a self-signed certificate, you can use openssl.opensslreq-new-newkeyrsa:2048-days3650-nodes-x509-subj/CN=selfsigned\-keyout$HOME/ssl.key\-out$HOME/ssl.crtRunningpython-maioftps3.server_mainConfigurationConfiguration is through environment variablesVaraiableDescriptionExampleAWS_AUTH_MECHANISMHow requests to AWS are authenticated. Can besecret_access_keyorecs_role. Ifecs_roleit is expected that the server runs in an ECS container.secret_access_keyAWS_ACCESS_KEY_IDThe ID of the  AWS access key, ifAWS_AUTH_MECHANISMissecret_access_key.ommittedAWS_SECRET_ACCESS_KEYThe secret part of the  AWS access key, ifAWS_AUTH_MECHANISMissecret_access_keyommittedAWS_S3_BUCKET_REGIONThe region of the S3 bucket that stores the files.eu-west-1AWS_S3_BUCKET_HOSTThe hostname used to communicate with S3.s3-eu-west-1.amazonaws.comAWS_S3_BUCKET_NAMEThe name of the bucket files are stored in.my-bucket-nameAWS_S3_BUCKET_DIR_SUFFIXThe suffix of the keys created in order to simulate a directory. Must start with a forward slash, but does not need to be longer./FTP_USERS__i__LOGINForiany integer, the username of an FTP user that can login.my-userFTP_USERS__i__PASSWORD_HASHEDForiany integer, the hash, as generated bycreate_password.py, of the password of an FTP user that can login, using the salt inFTP_USERS__i__PASSWORD_SALTommittedFTP_USERS__i__PASSWORD_SALTSeeFTP_USERS__i__PASSWORD_HASHEDommittedFTP_COMMAND_PORTThe port that the server listens on for command connections.8021FTP_DATA_PORTS_FIRSTThe first data port in the range for PASV mode data transfers.4001FTP_DATA_PORTS_COUNTThe number of ports used afterFTP_DATA_PORTS_FIRST.30FTP_DATA_CIDR_TO_DOMAINS__i__CIDRForiany integer, a CIDR range used to match the IP of incoming command connections. If a match is found, the IP of the corresponding domain or IP address inFTP_DATA_CIDR_TO_DOMAINS__i__DOMAINis returned to the client in response to PASV mode requests. Some clients will respond toFTP_DATA_CIDR_TO_DOMAINS__i__DOMAINbeing0.0.0.0by making PASV mode data connections to the same IP as the original command connection, but not all.0.0.0.0/0FTP_DATA_CIDR_TO_DOMAINS__i__DOMAINSeeFTP_DATA_CIDR_TO_DOMAINS__i__CIDR.ftp.my-domain.comHEALTHCHECK_PORTThe port the server listens on for healthcheck requests, such as from an AWS network load balancer.8022Advanced usageThe code inaioftps3.server_mainsatisfies a very particular use case, which may not be useful to most. However, the bulk of the code can be used for other cases: you will have to write your own aioftps3.server_main-equivalent, using the functionsaioftps3.server.on_client_connectandaioftps3.server_socket.server. For example, you couldStore credentials, appropriately hashed, differently, .e.g. in a database.Have the credentials hashed differently.Allow/deny PASV mode data connections based on some condition.See the source ofaioftps3.server_mainfor how these functions can be used.Creating a password and saltpython./create_password.pyRunning testsCertificates must be created, and Minio, which emulates S3 locally, must be started./certificates-create.sh&&./minio-start.shand then to run the tests themselves../tests.shFeatures / Design / LimitationsCan upload files bigger than 2G: usesmultipart uploadunder the hood.Does not store uploading files in memory before uploading them to S3: i.e. it is effectively a streaming upload. However, it's not completely streaming: each part of multipart upload is stored in memory before it begins to transfer to S3, in order to be able to hash its content and determine its length.For uploading files, hashes are computed incrementally as data comes in in order to not block the event loop just before uploads to S3.As few dependencies as is reasonable: aiohttp and its dependencies. Boto 3 isnotused.May not behave well if upload to the server is faster than its upload to S3.There is some locking to deal with the same files being operated on concurrently. However....... it does nothing to deal witheventual consistency of S3, and so some operations may appear to not have an immediate effect.Building and running locallydockerbuild-tftps-s3.&&\dockerrun--rm-p8021-8042:8021-8042\-eAWS_AUTH_MECHANISM=secret_access_key\-eAWS_ACCESS_KEY_ID=ommitted\-eAWS_SECRET_ACCESS_KEY=ommitted\-eAWS_S3_BUCKET_REGION=eu-west-1\-eAWS_S3_BUCKET_HOST=s3-eu-west-1.amazonaws.com\-eAWS_S3_BUCKET_NAME=my-bucket-name\-eAWS_S3_BUCKET_DIR_SUFFIX=/\-eFTP_USERS__1__LOGIN=user\-eFTP_USERS__1__PASSWORD_HASHED=ommitted\-eFTP_USERS__1__PASSWORD_SALT=ommitted\-eFTP_COMMAND_PORT=8021\-eFTP_DATA_PORTS_FIRST=4001\-eFTP_DATA_PORTS_COUNT=2\-eFTP_DATA_CIDR_TO_DOMAINS__1__CIDR=0.0.0.0/0\-eFTP_DATA_CIDR_TO_DOMAINS__1__DOMAIN=0.0.0.0\-eHEALTHCHECK_PORT=8022ftps-s3Building and pushing to Quaydockerbuild-tftps-s3.&&\dockertagftps-s3:latestquay.io/uktrade/ftps-s3:latest&&\dockerpushquay.io/uktrade/ftps-s3:latestBuilding and pushing healthcheck application to Quaydockerbuild-tftps-s3-healthcheck.-fDockerfile-healthcheck&&\dockertagftps-s3-healthcheck:latestquay.io/uktrade/ftps-s3-healthcheck:latest&&\dockerpushquay.io/uktrade/ftps-s3-healthcheck:latestBuilding and pushing Minio, used for testing, to Quaydockerbuild-tftps-s3-minio.-fDockerfile-minio&&\dockertagftps-s3-minio:latestquay.io/uktrade/ftps-s3-minio:latest&&\dockerpushquay.io/uktrade/ftps-s3-minio:latest