Package: cloud-fits
Cloud Optimized Fitscloud-fitsprovides the means to index large FITS files and have them served over HTTP(s) for efficient access. A
scientist or team can index the FITS file-directory, then upload the file-directory to A Static Cloud Provider. Static
Cloud Providers are Amazon WebServices, Google Cloud, Digital Ocean Spaces, or Microsoft Blob Storage. The FITS Cloud
Index can than be checked into a Github Repository, shared, or uploaded to a Static Cloud Providercloud-fitsreturns Astropy Datatypes as much as possibleLets index a FITS files and extract Metadata from itAccessing DataRegistry of Open Data on AWS provides a bucket calledstpubdata. It contains data uploaded from theTransiting
Exoplanet Survey Satellite. The type of data files we'll be working with in this tutorial are
about 44GB. It'll cost about $1.05 to download and index one of these 352 files from the Registry. To save time and
money, we'll only download one and index itPrerequisiteBefore we can download data off the Registry. Setup an AWS account and configure your credentials file. Then installaws-clihttps://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.htmlhttps://pypi.org/project/awscli/TutorialLets create our environment and download one data-file from the Registry$ mkdir -p /tmp/tess-data
$ cd /tmp/tess-data
$ aws s3 cp s3://stpubdata/tess/public/mast/ . --recursive --exclude "*" --include "tess-s0022-4-4-cube.fits" --request-payerWith our data downloaded, its time to create a bucket that'll hold our FITS Cloud Index. In some cases, we might not
have write access to the data we're indexing. In this case, we want to generate the index from a public data-set. Then
well store the index in a Static Cloud Provider of our choosing.cloud-fitscan then provide a Pythonic API
augmenting this abstractionLets create a bucket on AWS S3 and upload the index there$ AWS_DEFAULT_REGION=us-east-1 aws s3api create-bucket --bucket tess-fits-cloud-index
$ pip install cloud-fits -U
$ cloud-fits-index --index-bucket-name tess-fits-cloud-index --fits-files-directory /tmp/tess-data/ --data-bucket-path s3://stpubdata/tess/public/mastThe arguments--index-bucket-nameand--fits-file-directoryare intended to be straight forward with the naming.--data-bucket-pathis used to arugment the file-structure difference between--fits-file-directoryand--data-bucket-path.
For example, the data-cubes are located intess/public/mast, but this data isn't captured in--fits-file-directory. So,--data-bucket-pathwas introduced to augment the paths used to download sections of the fileOkay, great. We have everything we need. Now lets do some science. Start python and enter the following,from cloud_fits.bucket_operations import download_index
from cloud_fits.datatypes import FitsCloudIndex

from pprint import pprint

BUCKET_NAME: str = 'tess-fits-cloud-index'

index: FitsCloudIndex = download_index(BUCKET_NAME)
bintable_index = index.headers[2]
print(bintable_index[0, 10])Feature MapAmazon Web Services S3Pythonic API Refinement ( Planned Update )Remote Indexing for all Static Cloud Providers ( Planned Support )Digital Ocean Spaces ( Planned Support )Google Object Storage ( Planned Support )Microsoft Azure ( Planned Support )Development and DeploymentBuilding Docs Locallybuild.shhas a script that'll generate docs locally into/tmp/docs. The static HTML files can then be served from
that directory using these commands$ bash build.sh docs
$ cd /tmp/docs
$ python3 -m http.serverPYPI Automationbuild.shhas a script that'll generate the python wheel, sdist, and associated files. Then it'll upload it to
PYPI or PYPI-Test according to how you invoke thebuild.shfile$ bash build.sh publish-test