Package: service-tools
Python library with several tools for zmq services.
With this library a server with multiple local or distributed workers can be created very easy.Features:local and / or distributed workers and serverssecure authentication with private / public keysautomatic worker startupdynamic add or remove workerseasy worker configuration for new tasksUsageImports:importservice_toolsCreate Certificates:Options:–path=<path_to_certificates>:    path where the certificates are generated–users=<user>: optional;         specify for which users a certificate is generated; available options are: all, server, client; default is client–overwrite=<overwrite>:          optional; if True overwrite existing directory. default is Falsecreate server certificate:python3generate_certificates.py--path=<path_to_certificates>users=server--overwrite=<overwrite>create user certificate:python3generate_certificates.py--path=<path_to_certificates>users=user--overwrite=<overwrite>create server and user certificate:python3generate_certificates.py--path=<path_to_certificates>users=all--overwrite=<overwrite>ServerA server handles all client requests. The server deals the requests to connected workers which process
the request. Several workers can be connected to the server so that requests are computed parallel on
all workers.Server configA server is generated with a config file. Config files are yml-files.fromservice_toolsimportServerserver_config_path=r'resources\server_config.yml'# create a new server objectserver=Server(config_path=server_config_path)# start the serverserver.start()In the config file are the following informations:id: uuid of the servername: name of the serverip: ip address of the server; Example 'localhost', '127.0.0.1'port: port of the server; Example: '8006', '8007' If no port is specified a free port between 6001 and 6050 is
automatically chosen and written in the config file.backend_port: backend_port of the server; Example: '9006', '9007' If no port is specified a free port between
9001 and 9050 is automatically chosen and written in the config file.public_keys_dir: path to public keyssecret_keys_dir: path to secret keysnum_workers: number of workersauto_start: Bool if workers start automatically when server is startedworker_config_paths: list with paths to the worker config files. If only one worker_config_paths is defined but multiple wokers, this
worker_config is copied and a new id for each worker is generated and written in the configworker_script_path: path to script which is executed to start a worker; see also: Python script for automatic worker startlog_dir: directory where logs are createdlogging_mode: logging mode: 'DEBUG' 'INFO' 'WARN' 'ERROR'; see python loggingExample for server config:!ServerConfig_id:6b6f2689-0a1c-44da-be28-eff6bc92f723_name:test server_secure:true_public_keys_dir:resources\public_keys_secret_keys_dir:resources\private_keys_ip:localhost_port:8006_backend_port:9003_num_workers:4_auto_start:true_worker_config_paths:-worker1_config.yml_worker_script_path:test_start_worker.py_log_dir:resources\logging_dir_logging_mode:INFOconfig_path:test_config.yml_Server__update_file:false_Server__on_init:falseThe server config file can also be generated by a method:fromservice_tools.serverimportServerConfig,Servernew_server_config=ServerConfig.create_new(config_path=r'resources\config.yml',name='test server',ip='localhost',port=8006,public_keys_dir=r'resources\public_keys',secret_keys_dir=r'resources\private_keys',log_dir=r'resources\logging_dir',logging_mode='INFO',num_workers=4,auto_start=False,worker_config_paths=[r'resources\worker1_config.yml'],worker_script_path=r'resources\test_start_worker.py')# start the server with the new created server configserver=Server(config_path=new_server_config.config_path)server.start()WorkerWorkers, like servers, are create with a config file:fromservice_toolsimportWorkerworker_config_path=r'resources\worker_config.yml'# create a new workerworker=Worker(config_path=server_config_path)# start the workerworker.start()In the config file are the following information:id: uuid of the workername: name of the workerip: ip address of the worker; Example 'localhost', '127.0.0.1'port: port of the worker; Example: '9006', '9007'public_keys_dir: path to public keys; optionalsecret_keys_dir: path to secret keys; optionalpython_path: path to the python executable with which the worker should be started; optionallog_dir: directory where logs are createdlogging_mode: logging mode: 'DEBUG' 'INFO' 'WARN' 'ERROR'; see python loggingExample for worker config:!WorkerConfig_id:c49b82f2-5eee-495d-92fa-f28a5bdcc7fa_name:test worker_public_keys_dir:_secret_keys_dir:_ip:localhost_port:9049_python_path:python_log_dir:F:\OneDrive\PythonProjects\service_tools\tests\test_outputs\logging_dir_logging_mode:INFOThe worker config file can also be generated by a method:fromservice_tools.workerimportWorkerConfignew_worker_config=WorkerConfig.create_new(config_path=r'resources\worker1_config.ini',name='test worker',ip=None,port=None,public_keys_dir=None,secret_keys_dir=None,log_dir=r'resources\logging_dir',logging_mode='INFO')# start the server with the new created server configworker=Worker(config_path=new_worker_config.config_path)worker.start()Worker functionalityWorkers by default have no functionality. To add functionality create a class inherited from the Worker class. There
are many ways how functionality can be added.One way ist to create a Message class:classMessage(object):def__init__(self,*args,**kwargs):self.method=kwargs.get('method',None)self.args=kwargs.get('args',list())self.kwargs=kwargs.get('args',dict())The client then sends a message with a method specified as string.
The worker receives the message and selects its method to execute with:method=getattr(self,message.method)This method is then executed with the args and kwargs in the message and the return value is returned to the client:return(method(*message.args,**message.kwargs))Example: create a worker with the functionality ‘return_non_sense’ which returns ‘non sense’Worker:fromservice_toolsimportWorkerclassExtendedWorker(Worker):def__init__(self,*args,**kwargs):Worker.__init__(self,*args,**kwargs)defreturn_non_sense(self,*args,**kwargs):return'non sense'Client:importzmqclassMessage(object):def__init__(self,*args,**kwargs):self.method=kwargs.get('method',None)self.args=kwargs.get('args',list())self.kwargs=kwargs.get('args',dict())ctx=zmq.Context.instance()client=ctx.socket(zmq.REQ)client_public,client_secret=zmq.auth.load_certificate(r'resources\private_keys\5520471c-66c1-4605-80dc-a9ff84d959da.key_secret')server_public,_=zmq.auth.load_certificate(r'resources\public_keys\server.key')client.curve_secretkey=client_secretclient.curve_publickey=client_publicclient.curve_serverkey=server_publicclient.connect('tcp://localhost:8006')message=Message(method='return_non_sense')client.send_pyobj(message)return_value=client.recv_pyobj()Python script for automatic worker startHere is a script which starts a worker when executed. The path to the config is given by –config_file argument.fromconfigparserimportConfigParserimportargparseimportosimportuuid# Import a worker. In real case this would be a custom worker with additional functionality.fromsrc.service_tools.workerimportWorker# Import of message is necessary!fromsrc.service_tools.messageimportMessageif__name__=='__main__':parser=argparse.ArgumentParser()parser.add_argument('--config_file',required=True,help="worker config file",type=str)args=parser.parse_args()config_file=args.config_fileprint(f'reading config file:{config_file}')ifnotos.path.isfile(config_file):raiseFileExistsError(f'{config_file}does not exist')config=ConfigParser()config.read(config_file)try:name=config.get('main','name',fallback=None)exceptExceptionase:name=''try:id=uuid.UUID(config.get('main','id',fallback=None))exceptExceptionase:id=''try:ip=config.get('main','ip',fallback=None)exceptExceptionase:print('ip in{self.config_path}does not exist. Assume localhost...')ip='localhost'print(f'starting worker:\nname:{name}\nid:{id}\nip:{ip}')new_worker=Worker(config_path=config_file)new_worker.start()RequirementsPython 3.7+.Windows SupportSummary: On Windows, usepyinstead ofpython3for many of the examples in this
documentation.This package fully supports Windows, along with Linux and macOS, but Python is typicallyinstalled differently on Windows.
Windows users typically access Python through thepylauncher rather than apython3link in theirPATH. Within a virtual environment, all platforms operate the same and use apythonlink to access the Python version used in that virtual environment.DependenciesDependencies are defined in:requirements.inrequirements.txtdev-requirements.indev-requirements.txtVirtual EnvironmentsIt is best practice during development to create an isolatedPython virtual environmentusing thevenvstandard library module. This will keep dependant Python packages from interfering
with other Python projects on your system.On *Nix:$python3-mvenvvenv$sourcevenv/bin/activateOn Windowscmd:>py-mvenvvenv>venv\Scripts\activate.batOnce activated, it is good practice to update core packaging tools (pip,setuptools, andwheel) to the latest versions.(venv)$python-mpipinstall--upgradepipsetuptoolswheelPackagingThis project is designed as a Python package, meaning that it can be bundled up and redistributed
as a single compressed file.Packaging is configured by:pyproject.tomlsetup.pyMANIFEST.inTo package the project as both asource distributionand awheel:(venv)$pythonsetup.pysdistbdist_wheelThis will generatedist/fact-1.0.0.tar.gzanddist/fact-1.0.0-py3-none-any.whl.Read more about theadvantages of wheelsto understand why
generating wheel distributions are important.Upload Distributions to PyPISource and wheel redistributable packages can beuploaded to PyPIor installed
directly from the filesystem usingpip.To upload to PyPI:(venv)$python-mpipinstalltwine(venv)$twineuploaddist/*TestingAutomated testing is performed usingtox.
tox will automatically create virtual environments based ontox.inifor unit testing,
PEP8 style guide checking, and documentation generation.# Run all environments.
#   To only run a single environment, specify it like: -e pep8
# Note: tox is installed into the virtual environment automatically by pip-sync command above.(venv)$toxUnit TestingUnit testing is performed withpytest. pytest has become the defacto
Python unit testing framework. Some key advantages over the built inunittestmodule are:Significantly less boilerplate needed for tests.PEP8 compliant names (e.g.pytest.raises()instead ofself.assertRaises()).Vibrant ecosystem of plugins.pytest will automatically discover and run tests by recursively searching for folders and.pyfiles prefixed withtestfor any functions prefixed bytest.Thetestsfolder is created as a Python package (i.e. there is an__init__.pyfile
within it) because this helpspytestuniquely namespace the test files. Without this,
two test files cannot be named the same, even if they are in different sub-directories.Code coverage is provided by thepytest-covplugin.When running a unit test tox environment (e.g.tox,tox-epy37, etc.), a data file
(e.g..coverage.py37) containing the coverage data is generated. This file is not readable on
its own, but when thecoveragetox environment is run (e.g.toxortox-e-coverage),
coverage from all unit test environments is combined into a single data file and an HTML report is
generated in thehtmlcovfolder showing each source file and which lines were executed during
unit testing. Openhtmlcov/index.htmlin a web browser to view the report. Code coverage
reports help identify areas of the project that are currently not tested.Code coverage is configured inpyproject.toml.To pass arguments topytestthroughtox:(venv)$tox-epy37---kinvalid_factorialCode Style CheckingPEP8is the universally accepted style
guide for Python code. PEP8 code compliance is verified usingflake8.
flake8 is configured in the[flake8]section oftox.ini. Extra flake8 plugins
are also included:pep8-naming: Ensure functions, classes, and variables are named with correct casing.Automated Code FormattingCode is automatically formatted usingblack. Imports are
automatically sorted and grouped usingisort.These tools are configured by:pyproject.tomlTo automatically format code, run:(venv)$tox-efmtTo verify code has been formatted, such as in a CI job:(venv)$tox-efmt-checkGenerated DocumentationDocumentation that includes theREADME.rstand the Python project modules is automatically
generated using aSphinxtox environment. Sphinx is a documentation
generation tool that is the defacto tool for Python documentation. Sphinx uses theRSTmarkup language.This project uses thenapoleonplugin for
Sphinx, which renders Google-style docstrings. Google-style docstrings provide a good mix
of easy-to-read docstrings in code as well as nicely-rendered output."""Computes the factorial through a recursive algorithm.

Args:
    n: A positive input value.

Raises:
    InvalidFactorialError: If n is less than 0.

Returns:
    Computed factorial.
"""The Sphinx project is configured indocs/conf.py.Build the docs using thedocstox environment (e.g.toxortox-edocs). Once built,
opendocs/_build/index.htmlin a web browser.Generate a New Sphinx ProjectTo generate the Sphinx project shown in this project:# Note: Sphinx is installed into the virtual environment automatically by pip-sync command
# above.(venv)$mkdirdocs(venv)$cddocs(venv)$sphinx-quickstart--no-makefile--no-batchfile--extensionssphinx.ext.napoleon# When prompted, select all defaults.Modifyconf.pyappropriately:# Add the project's Python package to the path so that autodoc can find it.importosimportsyssys.path.insert(0,os.path.abspath('../src'))...html_theme_options={# Override the default alabaster line wrap, which wraps tightly at 940px.'page_width':'auto',}Modifyindex.rstappropriately:.. include:: ../README.rst

apidoc/modules.rstProject StructureTraditionally, Python projects place the source for their packages in the root of the project
structure, like:fact
├── fact
│   ├── __init__.py
│   ├── cli.py
│   └── lib.py
├── tests
│   ├── __init__.py
│   └── test_fact.py
├── tox.ini
└── setup.pyHowever, this structure isknownto have bad
interactions withpytestandtox, two standard tools maintaining Python projects. The
fundamental issue is that tox creates an isolated virtual environment for testing. By installing
the distribution into the virtual environment,toxensures that the tests pass even after the
distribution has been packaged and installed, thereby catching any errors in packaging and
installation scripts, which are common. Having the Python packages in the project root subverts
this isolation for two reasons:Callingpythonin the project root (for example,python-mpytest tests/)causes Python
to add the current working directory(the project root) tosys.path, which Python uses to find modules. Because the source
packagefactis in the project root, it shadows thefactpackage installed in the tox
environment.Callingpytestdirectly anywhere that it can find the tests will also add the project root
tosys.pathif thetestsfolder is a a Python package (that is, it contains a__init__.pyfile).pytest adds all folders containing packagestosys.pathbecause it imports the tests like regular Python modules.In order to properly test the project, the source packages must not be on the Python path. To
prevent this, there are three possible solutions:Remove the__init__.pyfile fromtestsand runpytestdirectly as a tox command.Remove the__init__.pyfile from tests and change the working directory ofpython-mpytesttotests.Move the source packages to a dedicatedsrcfolder.The dedicatedsrcdirectory is therecommended solutionbypytestwhen using tox and the solution this blueprint promotes because it is the least
brittle even though it deviates from the traditional Python project structure. It results is a
directory structure like:fact
├── src
│   └── fact
│       ├── __init__.py
│       ├── cli.py
│       └── lib.py
├── tests
│   ├── __init__.py
│   └── test_fact.py
├── tox.ini
└── setup.pyType HintingType hintingallows developers to include
optional static typing information to Python source code. This allows static analyzers such
asPyCharm,mypy, orpytypeto check that functions are used with the correct
types before runtime.ForPyCharm in particular,
the IDE is able to provide much richer auto-completion, refactoring, and type checking while
the user types, resulting in increased productivity and correctness.This project uses the type hinting syntax introduced in Python 3:deffactorial(n:int)->int:Type checking is performed by mypy viatox-emypy. mypy is configured insetup.cfg.LicensingLicensing for the project is defined in:LICENSE.txtsetup.pyThis project uses a common permissive license, the MIT license.You may also want to list the licenses of all of the packages that your Python project depends on.
To automatically list the licenses for all dependencies inrequirements.txt(and their
transitive dependencies) usingpip-licenses:(venv)$tox-elicenses...NameVersionLicensecolorama0.4.3BSDLicenseexitstatus1.3.0MITLicensePyCharm ConfigurationTo configure PyCharm 2018.3 and newer to align to the code style used in this project:Settings | Search “Hard wrap at”Editor | Code Style | General | Hard wrap at: 99Settings | Search “Optimize Imports”Editor | Code Style | Python | Imports☑ Sort import statements☑ Sort imported names in “from” imports☐ Sort plain and “from” imports separately within a group☐ Sort case-insensitivelyStructure of “from” imports◎ Leave as is◉ Join imports with the same source◎ Always split importsSettings | Search “Docstrings”Tools | Python Integrated Tools | Docstrings | Docstring Format: GoogleSettings | Search “Force parentheses”Editor | Code Style | Python | Wrapping and Braces | “From” Import Statements☑ Force parentheses if multilineIntegrate Code FormattersTo integrate automatic code formatters into PyCharm, reference the following instructions:black integrationThe File Watchers method (step 3) is recommended. This will runblackon every save.isort integrationThe File Watchers method (option 1) is recommended. This will runisorton every save.