Package: dtool-s3
GitHub:https://github.com/jic-dtool/dtool-S3PyPI:https://pypi.python.org/pypi/dtool-S3Free software: MIT LicenseFeaturesCopy datasets to and from S3 object storageList all the datasets in a S3 bucketCreate datasets directly in S3InstallationTo install the dtool-S3 package:pip install dtool-s3ConfigurationInstall theawsclient, for details seehttps://docs.aws.amazon.com/cli/latest/userguide/installing.html. In short:pip install awscli --upgrade --userConfigure the credentials using:aws configureThese are needed for theboto3library, for more details seehttps://boto3.readthedocs.io/en/latest/guide/quickstart.html.Configuring custom endpointsIt is possible to configure buckets to make use of custom endpoints. This is useful if one wants to make use of S3 storage not hosted in AWS.Create the file.config/dtool/dtool.jsonand add the s3 storage account details
using the format below:{
   "DTOOL_S3_ENDPOINT_<BUCKET NAME>": "<ENDPOINT URL HERE>",
   "DTOOL_S3_ACCESS_KEY_<BUCKET NAME>": "<USER NAME HERE>",
   "DTOOL_S3_SECRET_ACCESS_KEY_<BUCKET NAME>": "<KEY HERE>"
}For example:{
   "DTOOL_S3_ENDPOINT_my-bucket": "http://blueberry.famous.uni.ac.uk",
   "DTOOL_S3_ACCESS_KEY_ID_my-bucket": "olssont",
   "DTOOL_S3_SECRET_ACCESS_KEY_my-bucket": "some-secret-token"
}The configuration can also be done using your environment variables. For example on Linux/Mac:env 'DTOOL_S3_ENDPOINT_my-bucket=http://blueberry.famous.uni.ac.uk' \
    'DTOOL_S3_ACCESS_KEY_ID_my-bucket=olssont' \
    'DTOOL_S3_SECRET_ACCESS_KEY_my_bucket=some-secret-token' bashNote that hyphens in environment variable names do not adhere to thePOSIX standardexportwill not allow such names, hence the aboveworkaroundviaenvmay
be necessary to modify the environment.UsageTo copy a dataset from local disk (my-dataset) to a S3 bucket
(/data_raw) one can use the command below:dtool copy ./my-dataset s3://data_rawTo list all the datasets in a S3 bucket one can use the command below:dtool ls s3://data_rawSee thedtool documentationfor more detail.Publishing datasetsIt is possible to make datasets stored in S3 publicly accessible using thedtool publishcommand. The S3 storage broker supports making datasets
accessible to the world by setting the ACL topublic-read(the default) as
well as giving limited access to datasets using presigned URLS.To publish a dataset using a presigned URL one needs to set theDTOOL_S3_PUBLISH_EXPIRYto the number of seconds one wants to make the
dataset accessible for. For example by adding this setting to the~/.config/dtool/dtool.jsonfile or by exporting it as an environment
variable.export DTOOL_S3_PUBLISH_EXPIRY=3600Path prefix and access controlThe S3 plugin supports an endpoint-specific configurable prefix to the path.
This can be used for access control to the dataset. For example:env 'DTOOL_S3_DATASET_PREFIX_my-bucket=u/olssont' bashAlternatively one can edit the~/.config/dtool/dtool.jsonfile:{
   ...,
   "DTOOL_S3_DATASET_PREFIX_my-bucket": "u/olssont"
}Use the following S3 access to policy to that allows reading all data
in the bucket but only writing to the prefixu/<username>anddtool-:{
  "Statement": [
    {
      "Sid": "AllowReadonlyAccess",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:ListBucketVersions",
        "s3:GetObject",
        "s3:GetObjectTagging",
        "s3:GetObjectVersion",
        "s3:GetObjectVersionTagging"
      ],
      "Resource": [
        "arn:aws:s3:::my-bucket",
        "arn:aws:s3:::my-bucket/*"
      ]
    },
    {
      "Sid": "AllowPartialWriteAccess",
      "Effect": "Allow",
      "Action": [
        "s3:DeleteObject",
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Resource": [
        "arn:aws:s3:::my-bucket/dtool-*",
        "arn:aws:s3:::my-bucket/u/${aws:username}/*"
      ]
    },
    {
      "Sid": "AllowListAllBuckets",
      "Effect": "Allow",
      "Action": [
        "s3:ListAllMyBuckets",
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::*"
    }
  ]
}The user also needs write access to toplevel objects that start withdtool-.
Those are the registration keys that are not stored under the configured
prefix. The registration keys contain the prefix where the respective dataset
is found. They are empty if no prefix is configured.TestingLinux/MacAll tests need the S3_TEST_BASE_URI environment variable set.export S3_TEST_BASE_URI="s3://your-dtool-s3-test-bucket"For thetests/test_custom_endpoint_config.pytest one also needs to specify the S3_TEST_ACCESS_KEY_ID and S3_TEST_SECRET_ACCESS_KEY environment variables.export S3_TEST_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY
export S3_TEST_SECRET_ACCESS_KEY=YOUR_AWS_SECRET_ACCESS_KEYTo run the tests.python setup.py develop
pytestWindows PowerShellAll tests need the S3_TEST_BASE_URI environment variable set.$env:S3_TEST_BASE_URI = "s3://your-dtool-s3-test-bucket"For thetests/test_custom_endpoint_config.pytest one also needs to specify the S3_TEST_ACCESS_KEY_ID and S3_TEST_SECRET_ACCESS_KEY environment variables.$env:S3_TEST_ACCESS_KEY_ID = YOUR_AWS_ACCESS_KEY
$env:S3_TEST_SECRET_ACCESS_KEY = YOUR_AWS_SECRET_ACCESS_KEYTo run the tests.python setup.py develop
pytestWindows DOSAll tests need the S3_TEST_BASE_URI environment variable set.setx S3_TEST_BASE_URI "s3://test-dtool-s3-bucket-to"
python setup.py develop
pytestFor thetests/test_custom_endpoint_config.pytest one also needs to specify the S3_TEST_ACCESS_KEY_ID and S3_TEST_SECRET_ACCESS_KEY environment variables.setx S3_TEST_ACCESS_KEY_ID YOUR_AWS_ACCESS_KEY
setx S3_TEST_SECRET_ACCESS_KEY YOUR_AWS_SECRET_ACCESS_KEYTo run the tests.python setup.py develop
pytestRelated packagesdtoolcoredtool-clidtool-ecsdtool-httpdtool-azuredtool-irodsdtool-smb