Package: pygcdm
pygcdmA python API for transferringCommon Data Model (CDM)files usingRemote Procedure Calls.InstallationUsageTransfer Files using GRPCEncode/Decode APIMessage Typesvariable_specDefinitionUtilitiesRebuilding the Package LocallyCommitting the Package to PyPiGenerating Python Source Code fromprotoFilesRun Tests LocallyInstallationCreate a new virtual environment usingconda. If you don't havecondainstalled on your machine, install if followingthese instructions. Note thatpygcdmrequirespython>=3.9so we specify that explicitly. We can then usepipfor installation after activating thecondaenvironment.conda create --name [environment name] python=3.9
conda activate [environment name]
pip install pygcdmUsageTransfer Files using gRPCThis demonstrates how to implement the python encode/decode API and transmit netCDF files via gRPC. Start by opening up two separate tabs in your terminal. Make sure that whatever python environment you installed the requirements into is active in both. Navigate to theexamplesfolder in this repo.In the first tab, start the gRPC server by running the following command:python grpc_netcdf_server.pyIf it is working properly you should see something that saysstarting server....In the second tab, make a client request by running the following command:python grpc_netcdf_client.pySomething resembling an xarray dataset should print to the terminal if everything is working. Feel free to modify thelocandvariable_specvariables ingrpc_netcdf_client.pyto modify what data is transmitted.Encode/Decode APIThis demonstrates how to encode/decode gRPC messages within a python shell:# import/instantiate encoder object
from pygcdm.netcdf_encode import netCDF_Encode
encoder = netCDF_Encode()

# import and define header request
import pygcdm.protogen.gcdm_netcdf_pb2 as grpc_msg
file_loc = "test/data/test3.nc"
header_request = grpc_msg.HeaderRequest(location=file_loc)
header_response = encoder.generate_header_from_request(header_request)

# define data request
var_spec = "analysed_sst"
data_request = grpc_msg.DataRequest(location=file_loc, variable_spec=var_spec)
data_response = encoder.generate_data_from_request(data_request)

# decode header/data into xarray object
from pygcdm.netcdf_decode import netCDF_Decode
decoder = netCDF_Decode()
ds = decoder.generate_file_from_response(header_response, data_response)Message Typespygcdmhas two message typesHeaderRequestandDataRequestwhich have corresponding response types (HeaderResponse,DataResponserespectively). These messages can be learned abouthereand are defined inprotos/pygcdm/protogen/gcdm_netcdf.proto. The main idea is that you pack your request messages with the information you want and unpack the response messages (pygcdmdoes this packing/unpacking for you). The.protomessages are pretty human readable but a description is provided here:HeaderRequest:location(string): file location where netCDF file is found on machine delivering responseDataRequest:location(string): file location where netCDF file is found on machine delivering responsevariable_spec(string): desctribes what netCDF variable/variable slices you want to request (seevariable_specsection in this README)variable_specDefinitionvariable_specis how you define which variable/data slices you want from the remote netCDF file. It followsBackus-Naur formaccording to the following definition, which was adopted fromnetcdf-javawhich is documentedhere:section specification := selector | selector '.' selector
selector := varName ['(' dims ')']
varName := ESCAPED_STRING


dims := dim | dim, dims
dim := ':' | slice | start ':' end | start ':' end ':' stride
slice := INTEGER
start := INTEGER
stride := INTEGER
end := INTEGER
ESCAPED_STRING : must escape characters = ".("In practice, this means thatvariable_speccan look like the following examples. Slicing notation is analogous to numpy slicing (but not identical).analysed_sstanalysed_sst(:, :, :)analysed_sst(0,0:719:10,0:1439:10)analysed_sst(0,200:700:10,1300:1400:10)Note that these examples are derived from some of the tests intest/.UtilitiesRebuilding the Package LocallyIf you modify the package locally and want to rebuild the package for testing you can run the following command from the root folder:pip install .(You can verify installation by runningpip show -f pygcdm).Committing the Package to PyPiIf you have the priviledges to do so, you can commit the package to PyPi by running the following command in the root folder:python setup.py uploadwhich will prompt you what type of upload you want:testorreal, wheretestuploads tohttps://test.pypi.org/project/pygcdm/andrealuploads tohttps://pypi.org/project/pygcdm/. You'll then be prompted to include your login credentials.Generating Python Source Code fromprotoFilesIn order to generate the grpc python source code from the.protofiles inprotos/src/protogen, we need to use theprotoccompiler (the reason for the weird subfolder structure is so thatprotocgenerates files with proper import namespaces). The python package withprotoccan be installed by following the instructionshere.To compile our code, and put the resulting functions intosrc/protogen/, we can use the following commands (from the root directory where this README is located):$ python -m grpc_tools.protoc -I protos/ --python_out=. --grpc_python_out=. protos/pygcdm/protogen/*.protoMore information on theprotoccompiler can be found by loading the module in python and using thehelpcommand:$ python
>>> import grpc_tools.protoc
>>> help(grpc_tools.protoc)Run Tests LocallyIf you want to run the tests locally, make sure you have thepytestmodule installed. Then, from the root directory run the following command:pytest test/*.pyIf you are testing to verify any changes make sure you rebuild the package locally using the instructions above.