Package: meracan-s3netcdf
s3-netcdfS3-NetCDF is a Python library to read / write NetCDF files to S3. This library partitions large NetCDF files into smaller chunks to retrieve data from s3 cost-effectively.Installationpipinstallmeracan-s3netcdfFrom local foldergitclonehttps://github.com/meracan/s3-netcdf.git
pipinstall-e./s3-netcdfWith conda env and testingcondacreate-ns3netcdfpython=3.8
condaactivates3netcdf
gitclonehttps://github.com/meracan/s3-netcdf.git
pipinstall-e./s3-netcdfMethodologyS3-NetCDF creates a master file ".nca" from an input object. The input contains s3 info, metadata, dimensions, partition group, variables, etc. Data is stored in the partition files (.nc) (no data is stored in the master file).Variables need to be stored in a partition group. Each partition group has unique variable's dimensions. Multiple variables can be stored under the same partition group (if they have the same dimensions).The maximum size of partition file (umcompressed) is set using the option inputncSize=1.0(MB). The size is approximative depending on the shape of the array. The partional files are automatically compressed (~100 smaller). The attributeleast_significant_digit={number}can be added in the variable object to further reduce file size. Rememberf4andf8contains 7 digits 16 digits, respectively. S3 http compression (gzip) is not used since partition files are already compressed.InputThe input for creating a master file contains s3 info, metadata, dimensions, partition group, variables, etc.Metadata attributes are stored in themetadataobject. It is recommended to usetitle,institution,source,history,references, andcomment.Dimensions, groups and variables are stored in thencaobject.Input JSON file needs to be converted into a python objectimport json; json.loads(filePath). Input example to create a master file:{"name":"input1","cacheLocation":"../s3","localOnly":true,"bucket":"merac-dev","cacheSize":10.0,"ncSize":1.0,"metadata":{"title":"title-input1"},"nca":{"dimensions":{"npe":3,"nelem":500,"nnode":1000,"ntime":2},"groups":{"elem":{"dimensions":["nelem","npe"],"variables":{"elem":{"type":"i4","units":"","standard_name":"Elements","long_name":"Connectivity table (mesh elements)"}}},"time":{"dimensions":["ntime"],"variables":{"time":{"type":"f8","units":"hours since 1970-01-01 00:00:00.0","calendar":"gregorian","standard_name":"Datetime","long_name":"Datetime"}}},"nodes":{"dimensions":["nnode"],"variables":{"bed":{"type":"f4","units":"m","standard_name":"Bed Elevation, m","long_name":"Description of data source"},"friction":{"type":"f4","units":"","standard_name":"Bed Friction (Manning's)","long_name":"Description of data source"}}},"s":{"dimensions":["ntime","nnode"],"variables":{"a":{"type":"f4","units":"m","standard_name":"a variable","long_name":"Description of a"}}},"t":{"dimensions":["nnode","ntime"],"variables":{"a":{"type":"f4","units":"m","standard_name":"a variable","long_name":"Description of a"}}}}}}The input for opening  a master file can be simplified. As a minimum, the input file should containname,cacheLocationandbucket(if using S3).Input example to open a master file:{"name":"input1","cacheLocation":"../s3","bucket":"merac-dev","localOnly":true,"cacheSize":10.0,"ncSize":1.0}S3, caching and localOnlyPartition files are saved locally (caching) while reading and writing. By default, thecacheLocation={path}is the current working directory.The input optioncacheSize=1.0defines the maximum cache size in MB. If exceeded, oldest cached partition files are removed.The input optionlocalOnly=Truewill ignore all S3 & caching commands. This is used for testing.The name of thebucket={str}in the input if files are uploaded to S3.UsageBasicfroms3netcdfimportNetCDF2D# Create/Open master filenetcdf2d=NetCDF2D(input)# Writingnetcdf2d["{groupname}","{variablename}",{...indices...}]=np.array(...)# Readingnetcdf2d["{groupname}","{variablename}",{...indices...}]Assigning values to indexed arrays is the same asnumpy. Note: string values was not tested.Commands# Get information inside the master filenetcdf2d.info()# Get group dimensional shapenetcdf2d.groups["{groupname}"].shape# Get group dimensional partition shapenetcdf2d.groups["{groupname}"].child# Get variable's attributesnetcdf2d.groups["{groupname}"].attributes["{variablename}")Caching commands# List partition files locallynetcdf2d.cache.getNCs()# Clear/Delete all partition files locally# Warning!netcdf2d.cache.clearNCs()# Delete NetCDF locally# Warning!# Delete master file and partitions filesnetcdf2d.cache.delete()S3 commands# List master and partition files, including metedatanetcdf2d.s3.list()# Clear/Delete all partition files in S3# Warning!netcdf2d.s3.clearNCs()# Delete NetCDF in S3# Warning!# Delete master file and partitions filesnetcdf2d.s3.delete()Testingcondainstallpytest
mkdir../s3
pytestFor developers and debugging:mkdir../s3PYTHONPATH=../s3-netcdf/python3test/test_netcdf2d_func.pyPYTHONPATH=../s3-netcdf/python3test/test_netcdf2d1.pyPYTHONPATH=../s3-netcdf/python3test/test_netcdf2d2.pyAWS S3 CredentialsCredentials (for example), AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION needs to be save in environment variables. For more information, checklink.The credentials needs access toget,putanddelete(if deleting is required) to the bucket.Performance and BenchmarkTODORevise code on the value parsing side: compare shape, value type etc, Should be in different function and not in dataWrapper.Check operation when index assigning: + - * /Fix bench folder and create better performance testsFind optimize shape to uploadtravis-ci and encryption keysComplete documentation in code