Package: grpc-helper
grpc-helperMiscellaneous GRPC helpers (API versioning, retry, config, etc...)Python APIProvided classes in this module API help to deal with RPC servers/clients handling.RPC ServerTheRpcServerclass handles the lifecycle of a GRPC server. To initialize, it basically needs:a port on which to server RPC requestsa list ofRpcServiceDescriptorobjectsan optionalFoldersinstance (usually provided by theRpcCliParserCLI parsing)an optional dict of string:string, providing config items defaults initialized from CLI (usually provided by theRpcCliParserCLI parsing)an optional list ofConfigorConfigHolderinstances, which will be this server'sstaticconfig itemsan optional list ofConfigorConfigHolderinstances, which will be this server'suserconfig itemsa boolean flag stating if the events service has to be started in this server instance (default:False)a boolean flag stating if the debug signal has to be listened (see below; default:True)TheFoldersclass handle the different folders used by the server:asystemfolder, used to store config shared by multiple users and applications. This folder doesn't need to be writable by the server running user.auserfolder, used to store config shared by multiple applications for the server running user.aworkspacefolder, used to store config and other persistent information by the current server.TheRpcServiceDescriptorclass describes a given service to be hooked in a server instance. Its attributes are:a Python module (from which name and version will be used for the RpcServerService items registration)a service name (used for service identification in RpcServerService items, and for auto-client setup)a version enum:lowest value in the enum will be considered to be the minimum supported version for this service APIhighest value in the enum will be considered to be the current version for this service APIa manager instance, to which all RPC calls will be delegated. When registering a proxied service, this has to be a simple instance of the generated servicer.the GRPC generated hooking method for this servicethe GRPC generated client stub classa boolean stating if this service is a proxied service or notThe manager class must:inherit from the GRPC generated servicer class, in order to use the default implementation if any of the service method is not implemented by this managerfor each implemented service method:declare a single input parameter, which will be the input requestdeclare the return typeA manager class can also inherit fromRpcManagerclass, which provides some usefull features:a_loadmethod, called by the server once all services are alive (typically to perform some internal initializations)a_shutdownmethod, called by the server once it is shutdown (typically to perform some internal finalization operations + interrupt long-running ones)aloggerinstance, to be used for all logging inside this manager and dependenciesalockinstance, to be used to protect manager inner fields against reentranceaclientinstance, initialized to the server own auto-client (see below)afoldersinstance, initialized to the serverFolders(see above)LifecycleThe RPC server will live its life in its own thread. When the application is about to terminate, it is advised to call theshutdownmethod
in order to turn off the RPC server properly. This method can also be called remotely as aRpcServerServiceservive method.When the shutdown method is called:the server will stop accepting new requestsit will notify all managers through their own_shutdownmethodthen it will wait for pending operations to terminate (within therpc-shutdown-graceconfigured timeout)method.if called through RPC, if will wait for the required timeout (using by default therpc-shutdown-timeoutvalue), if this one is >0then the shutdown will be finalized (all non-daemon threads terminated)Note that the code which created the server instance may use thewait_shutdownmethod to block until the RPC server is shutdown.Default servicesNote that the RPC server instance will automatically serve:theserver handling service,
for basic RPC server operations handling.theconfig service,
allowing to remotely get/update/reset user configuration items.thelogger service,
allowing to remotely get/update/reset loggers configuration.API version checksFor requests received from theRpcClientimplementation on a given service, the client api version is checked against the server "supported - current"
range for this service:if the client version is older than the server supported version, the request will raise aResultCode.ERROR_API_CLIENT_TOO_OLDerrorif the client version is newer than the server current version, the request will raise aResultCode.ERROR_API_SERVER_TOO_OLDerrorDebugThe RPC server will hook to the USR2 signal for debug purpose (if required). When receiving this signal, following debug information will be dumped in a file
nameRpcServerDump-YYYYMMDDhhmmss.txt(with dump timestamp) in the logging folder (see below):all the live threads call stacksall the pending RPC requestsAuto-clientAn initialized RPC server instance provides anauto_clientattribute, providing anRpcClientinstance pointing on everything
served by this server. Note that the timeout for this client can be configured through therpc-client-timeoutconfig item.Proxied servicesServices declared as proxied ones (seeRpcServiceDescriptorabove) will be forwarded to the required server/port as soon as this server calls theproxy_registermethod of the proxying server. Note that if this method is not called within therpc-client-timeouttime, the method call will
end with anERROR_PROXY_UNREGISTEREDerror.Configuration of proxied services is persisted in the workspace, in order to restore them when the proxying server restarts.AnRpcProxiedManagerabstract class is available for managers who are designed to be proxied. Such managers will automatically register in main proxy
server, with following details:registered services names list has to be returned by_proxied_servicesmethodregistered services version string has to be returned by_proxied_versionmethodextra stubs available in theself.proxy_clientattribute can be provided by_proxy_stubsmethodif current host IP needs excpilicitely to be used for registration,_proxy_use_current_hostmethod needs to returnTrueWhen handling proxy servers, if the proxying server is started with events service enabled, it will send:anRPC_PROXY_REGISTERevent on eachproxy_registermethod successful callanRPC_PROXY_FORGETevent on eachproxy_forgetmethod successful callProperties of the events will be inherited from the method input message.Rolling logsEach manager of the RPC server through itsloggerattribute, will have his logs persisted in a<name>/<name>.logrolling file (relative to the
logging folder). The complete logs will also be persisted in therootfolder.
Both logging folder and rollover cadency can be configured -- seeConfigurationchapter below.ConfigurationThe RPC server behavior can be configured through the following static configuration items:NameDescriptionTypeDefaultrpc-max-workersMaximum parallel RPC worker threadsPositive integer30rpc-shutdown-graceGrace period for pending calls to be terminated on shutdown (seconds)Positive float30rpc-shutdown-timeoutFinal timeout before real shutdown (i.e. end of process; seconds)Positive float60rpc-logs-folderFolder (absolute or workspace-relative) where to store rolling logsString"logs"rpc-logs-backupBackup log files to be persisted for each manager on rolloverInteger10rpc-logs-interval-unitLogs rollover interval unit (seeTimedRotatingFileHandler documentation)Custom (seedoc)Hrpc-logs-intervalLogs rollover interval (seeTimedRotatingFileHandler documentation)Positive integer1rpc-main-hostMain RPC server host (to be used by proxied services)String"localhost"rpc-main-portMain RPC server port (to be used by proxied services)Positive integer54321rpc-client-timeoutTimeout for RPC client when server is unreachable or proxy not registered yet (seconds)Positive float60rpc-event-retain-timeoutRetain timeout for event queues on interruption (seconds)Positive integer300rpc-event-keepalive-timeoutTimeout for sending keep alive empty events (seconds)Positive integer60Usage exampleimportmy_packagefrommy_package.apiimportMyStatus,MyConfig,MyApiVersionfrommy_package.api.my_pb2_grpcimportadd_MyServiceServicer_to_server,MyServiceServicer,MyServiceStubfromgrpc_helperimportRpcServer,RpcServiceDescriptor,RpcManagerclassMySampleManager(MyServiceServicer,RpcManager):# Custom implementation for sample servicedefupdate(self,request:MyConfig)->MyStatus:# Note that return message *MUST* be explicitely annoted for each implemented method!returnMyStatus()defstart():# Create an RPC server on port 12345srv=RpcServer(12345,[RpcServiceDescriptor(my_package,"my",MyApiVersion,MySampleManager(),add_MyServiceServicer_to_server,MyServiceStub)])# Server is running in its own thread; we need to wait it for shutdown (on Ctrl-C or remote shutdown call)srv.wait_shutdown()RPC ClientTheRpcClientclass provides an access to client side of a GRPC service. To initialize, it basically needs:a host name or IP address for the RPC server to connect toa port for the RPC servera map of service stubs to access:keys are used to define fields names on the generated client object, holding the client stubsvalues are tuple providing:the GRPC generated stub namethe client current API version (coming from the version enum)Optional inputs can also be provided:a timeout if RPC server is unreachable (default:60s)a name allowing to identify the client on the server sidea boolean flag stating if the client shall raise exceptions when receiving non-OKResultCodestatus (default:true)an exception type to be raised (instead ofRpcException) when receiving non-OKResultCodestatusOnce created, a client instance provide as many stubs as configured in the service map. Each of this stubs expose the generated methods of the
corresponding service. These methods take the following arguments:the method input request message (see corresponding proto file)a timeout parameter (in seconds) for this particular request. If set to None (default), there is no timeoutUsage examplefrommy_package.apiimportMyStatus,MyApiVersion,Emptyfrommy_package.api.my_pb2_grpcimportMyServiceStubfromgrpc_helperimportRpcClientdefstart():# Get a client access to my servicec=RpcClient("127.0.0.1",12345,{"my":(MyServiceStub,MyApiVersion.MY_API_CURRENT)},name="myclient")# Use APIs:MyStatus=c.my.list(Empty())Configuration itemsInstances of theConfigclass can be provided to initialize list of static and user configuration items known to a RPC server:staticitems are immutable, and are only initialized once when the RPC server instance is createduseritems can be modified remotely through theconfig serviceTheConfigconstructor arguments are the same than the public APIConfigItemones (seeconfig service).
It also supports an additionalcustom_validatorone, allowing to provide a validation method whenvalidatorargument is set toCONFIG_VALID_CUSTOM.
This method takes two arguments:anamestring: may be useful to raise meaningful validation errorsavaluestring: value to be validated. The method shall raise anRpcExceptionif the value is invalid.Default valueConfiguration items default value are loaded in the following order:hard-coded value (the one provided in the item constructor)system folderconfig.jsonfile value (if system folder is configured)user folderconfig.jsonfile value (if user folder is configured)environment provided value. Environment variable name for a given item is obtained by capitalizing it and replacing hyphens ("-") by underscores ("_").
E.g. the environment variable name for amy-own-settingconfig item will beMY_OWN_SETTING.-c / --configcommand-line option provided valueNote that if the default value fails to be validated with a given item validator (or if it is empty while not allowed), the server will refuse to launch with a
thrown relevant exception.Current valueStatic items current value will be initialized with the default value. User items are initialized to either the default value, or to the user configured value
(thanks to theconfig service) if it was ever modified. User modified values are
persisted in the workspace folderconfig.jsonfile.Current value may be programmatically accessed through one of the following property accessors:item.str_val: raw string valueitem.int_val: value converted as an integerConfig holdersConfiguration items may be defined as attributes of a class inheriting from theConfigHolderone. Such classes can be provided directly toRpcServerconstructor to initialize configuration items (which is more convenient that referencing items one by one).Command-line optionsA ready-to-use CLI parser is provided through theRpcCliParserclass, allowing to rapidly instantiate an RPC server from CLI options.
Constructor arguments are:adescriptionstring, which will be displayed when using the-h / --helpoptionan optionalversionstring, which will be displayed when using the-V / --versionoptionOptionsThe options defined by this parser are:NameDescription--helpdisplays the program help and exit-V / --versiondisplays the program version and exit--system PATHoverrides the default system folder--user PATHoverrides the default user folder-w / --workspace PATHdefines the workspace folder-p / --port PORToverrides the RPC server default listening port-c / --config NAME=VALUEoverrides the default value of a given configuration itemAfter parsing:the system/user/workspace folders are used to initialize aFoldersobject (see above), available in thefoldersargument of the namespace.the configuration items default values are gathered in a dict, available in theconfigargument of the namespace.UsageThe following methods are available to tune the parser behavior:with_rpc_args: adds the options listed above, and returns theRpcCliParserinstance (for fluent API usage). Arguments are:default_port: default RPC listening portdefault_sys: default system folderdefault_usr: default user folderdefault_wks: default workspace folderparse: will parse input arguments, and return the namespace. Arguments are program command-line ones by default, or may be provided as a string list
(e.g. for testing purpose)Following example shows how to use this parser to create anRpcServerinstance:fromgrpc_helperimportRpcCliParser,RpcServerdefstart():# Parse arguments to create RPC server instanceargs=RpcCliParser("My custom RPC server","1.0.0").with_rpc_args(12345,"/etc/my_srv","~/.local/my_srv","my_srv_wks").parse()srv=RpcServer(args.port,[],args.folders,args.config)