Package: tno.mpc.communication
TNO PET Lab - secure Multi-Party Computation (MPC) - CommunicationThe TNO PET Lab consists of generic software components, procedures, and
functionalities developed and maintained on a regular basis to facilitate and
aid in the development of PET solutions. The lab is a cross-project initiative
allowing us to integrate and reuse previously developed PET functionalities to
boost the development of new protocols and solutions.The package tno.mpc.communication is part of the TNO Python Toolbox.Limitations in (end-)use: the content of this repository may solely be used for
applications that comply with international export control laws.This implementation of cryptographic software has not been audited. Use at your
own risk.DocumentationDocumentation of the tno.mpc.communication package can be foundhere.InstallEasily install the tno.mpc.communication package using pip:$python-mpipinstalltno.mpc.communicationNote: The package specifies several optional dependency groups:gmpy: Adds support for sending variousgmpy2typestests: Includes all optional libraries required to run the full test suitetls: Required if SSL is neededbitarray: Adds support for sendingbitarraytypesnumpy: Adds support for sendingnumpytypespandas: Adds support for sendingpandastypesSeesending, receiving messagesfor more
information on the supported third party types. Optional dependencies can be
installed by specifying their names in brackets after the package name, e.g.
when usingpip install, usepip install tno.mpc.communication[extra1,extra2]to install the groupsextra1andextra2.UsageThe communication module usesasyncfunctions for sending and receiving. If
you are familiar with the async module, you can skip to thePoolssection.Async explanationWhenasyncfunctions are called, they return what is called acoroutine.
This is a special kind of object, because it is basically a promise that the
code will be run and a result will be given when the coroutine is given to a
so-calledevent loop. For example, see the followingimportasyncioasyncdefadd(a:int,b:int)->int:returna+bdefmain():a,b=1,2coroutine_object=add(a,b)# This is now a coroutine object of type Awaitable[int]event_loop=asyncio.get_event_loop()# This is the event loop that will run the coroutineresult=event_loop.run_until_complete(coroutine_object)# This call starts the coroutine in the event loopprint(result)# this prints 3if__name__=="__main__":main()As you can see from the example, the async methods are defined usingasync def, which tells python that it should return a coroutine. We saw how we
can call an async function from a regular function using the event loop.Note
that you should never redefine the event loop and always retrieve the event loop
as done in the example(unless you know what you are doing). We can also call
async functions from other async functions using theawaitstatement, as is
shown in the following example.importasyncioasyncdefadd_four_numbers(first:int,second:int,third:int,fourth:int)->int:first_second=awaitadd(first,second)# This is blocking, so the function will wait until this is donethird_fourth_coroutine=add(third,fourth)# This is non-blocking, so the code will continue while the add(third,fourth) code starts running# we can do some other stuff hereprint("I am a print statement")third_fourth=awaitthird_fourth_coroutine# we wait until the add(third,fourth) is doneresult=awaitadd(first_second,third_fourth)# here it is important to use await for the result, because then an integer is produced and given# to the return statement instead of a coroutinereturnresultasyncdefadd(a:int,b:int)->int:returna+bdefmain():a,b,c,d=1,2,3,4coroutine_object=add_four_numbers(a,b,c,d)# This is now a coroutine object of type Awaitable[int]event_loop=asyncio.get_event_loop()# This is the event loop that will run the coroutineresult=event_loop.run_until_complete(coroutine_object)# This call starts the coroutine in the event loopprint(result)# this prints 10if__name__=="__main__":main()Note that the type of thecoroutine_objectin themainfunction is anAwaitable[int]. This refers to the fact that the result can be awaited (inside
anasyncfunction) and will return an integer once that is done.PoolsAPoolrepresents a network. A Pool contains a server, which listens for
incoming messages from other parties in the network, and clients for each other
party in the network. These clients are called upon when we want to send or
receive messages.It is also possible to use and initialize the pool without taking care of the
event loop yourself, in that case the template below can be ignored and the
examples can be used as one would regularly do. (An event loop is however still
needed when using theawaitkeyword or when calling anasyncfunction.)TemplateBelow you can find a template for usingPool. Alternatively, you could create
the pool in themainlogic and give it as a parameter to theasync_mainfunction.importasynciofromtno.mpc.communicationimportPoolasyncdefasync_main():pool=Pool()# ...if__name__=="__main__":loop=asyncio.get_event_loop()loop.run_until_complete(async_main())Pool initializationThe following logic works both in regular functions andasyncfunctions.Without SSL/TLS (do not use in production)The following snippet will start a HTTP server and define its clients. Clients
are configured on both the sending and the receiving side. The sending side
needs to know who to send a message to. The receiving side needs to know who it
receives a message from for further handling.By default thePoolobject uses the origin IP and port to identify the client.
However, a more secure and robust identification through SSL/TLS certificates is
also supported and described in sectionWith SSL/TLS (SSL/TLS certificate as client identifier).fromtno.mpc.communicationimportPoolpool=Pool()pool.add_http_server()# default port=80pool.add_http_client("Client 1","192.168.0.101")# default port=80pool.add_http_client("Client 2","192.168.0.102",port=1234)With SSL/TLSA more secure connection can be achieved by using SSL/TLS. APoolobject can
be initialized with paths to key, certificate and CA certificate files that are
passed as arguments to assl.SSLContextobject. More information on the expected files can be found in thePool.__init__docstring and thessldocumentation.fromtno.mpc.communicationimportPoolpool=Pool(key="path/to/keyfile",cert="path/to/certfile",ca_cert="path/to/cafile")pool.add_http_server()# default port=443pool.add_http_client("Client 1","192.168.0.101")# default port=443pool.add_http_client("Client 2","192.168.0.102",port=1234)We do not pose constraints on the certificates that you use in the protocol.
However, your organisation most likely poses minimal security requirements on
the certificates used. As such we do not advocate a method for generating
certificates but rather suggest to contact your system administrator for
obtaining certificates.With SSL/TLS (SSL/TLS certificate as client identifier)This approach does not use the origin of a message (HTTP request) as identifier
of a party, but rather the SSL/TLS certificate of that party. This requires a
priori exchange of the certificates, but is more robust to more complex (docker)
network stacks, proxies, port forwarding, load balancers, IP spoofing, etc.More specifically, we assume that a certificate has a unique combination of
issuer Common Name and S/N and use these components to create a HTTP client
identifier. Our assumption is based on the fact that we trust the issuer (TSL
assumption) and that the issuer is supposed to hand out end-user certificates
with different serial numbers.fromtno.mpc.communicationimportPoolpool=Pool(key="path/to/own/keyfile",cert="path/to/own/certfile",ca_cert="path/to/cafile")pool.add_http_server()# default port=443pool.add_http_client("Client 1","192.168.0.101",port=1234,cert="path/to/client/certfile")Additional dependencies are required in order to load and compare certificates.
These can be installed by installing this package with thetlsextra, e.g.pip install tno.mpc.communication[tls].Adding clientsHTTP clients are identified by an address. The address can be an IP address, but
hostnames are also supported. For example, when communicating between two docker
containers on the same network, the address that is provided topool.add_http_clientcan either be the IP address of the client container or
the name of the client container.Sending, receiving messagesThe library supports sending the following objects through the send and receive
methods:stringsbyte stringsintegersfloatsenum (partially, seeSerializingEnum)(nested) lists/tuples/dictionaries/numpy arrays containing any of the above.
Combinations of these as well.Under the hoodormsgpackis used,
additional options can be activated using theoptionparameter (see,https://github.com/aviramha/ormsgpack#option).Messages can be sent both synchronously and asynchronously. If you do not know
which one to use, use the synchronous methods withawait.# Client 0awaitpool.send("Client 1","Hello!")# Synchronous send message (blocking)pool.asend("Client 1","Hello!")# Asynchronous send message (non-blocking, schedule send task)# Client 1res=awaitpool.recv("Client 0")# Receive message synchronously (blocking)res=pool.arecv("Client 0")# Receive message asynchronously (non-blocking, returns Future if message did not arrive yet)Custom message IDs# Client 0awaitpool.send("Client 1","Hello!","Message ID 1")# Client 1res=awaitpool.recv("Client 0","Message ID 1")Custom serialization logicIt is also possible to define serialization logic in custom classes and load the
logic into the commmunication module. An example is given below. We elaborate on
the requirements for such classes after the example.classSomeClass:defserialize(self,**kwargs:Any)->Dict[str,Any]:# serialization logic that returns a dictionary@staticmethoddefdeserialize(obj:Dict[str,Any],**kwargs:Any)->'SomeClass':# deserialization logic that turns the dictionary produced# by serialize back into an object of type SomeClassThe class needs to contain aserializemethod and adeserializemethod. The
type annotation is necessary and validated by the communication module. Next to
this, the**kwargsargument is also necessary to allow for nested
(de)serialization that makes use of additional optional keyword arguments. It is
not necessary to use any of these optional keyword arguments. If one does not
make use of the**kwargsand also does not make a call to a subsequentSerialization.serialize()orSerialization.deserialize(), it is advised to
write**_kwargs: Anyinstead of**kwargs: Any.To add this logic to the communication module, you have to run the following
command at the start of your script. Thecheck_annotiationsparameter
determines whether the type hints of the serialization code and the presence of
a**kwargsparameter are checked. You should only change this to Falseif you
are exactly sure of what you are doing.fromtno.mpc.communicationimportSerializationif__name__=="__main__":Serialization.set_serialization_logic(SomeClass,check_annotations=True)SerializingEnumTheSerializationmodule can serialize anEnummember; however, only the
value is serialized. The simplest way to work around this limitation is to
convert the deserialized object into anEnummember:fromenumimportEnum,autoclassTestEnum(Enum):A=auto()B=auto()enum_obj=TestEnum.B# Client 0awaitpool.send("Client 1",enum_obj)# Client 1res=awaitpool.recv("Client 0")# 2 <class 'int'>enum_res=TestEnum(res)# TestEnum.B <enum 'TestEnum'>Example codeBelow is a very minimal example of how to use the library. It consists of two
instances, Alice and Bob, who greet each other. Here, Alice runs on localhost
and uses port 61001 for sending/receiving. Bob also runs on localhost, but uses
port 61002.alice.pyimportasynciofromtno.mpc.communicationimportPoolasyncdefasync_main():# Create the pool for Alice.# Alice listens on port 61001 and adds Bob as client.pool=Pool()pool.add_http_server(addr="127.0.0.1",port=61001)pool.add_http_client("Bob",addr="127.0.0.1",port=61002)# Alice sends a message to Bob and waits for a reply.# She prints the reply and shuts down the poolawaitpool.send("Bob","Hello Bob! This is Alice speaking.")reply=awaitpool.recv("Bob")print(reply)awaitpool.shutdown()if__name__=="__main__":loop=asyncio.get_event_loop()loop.run_until_complete(async_main())bob.pyimportasynciofromtno.mpc.communicationimportPoolasyncdefasync_main():# Create the pool for Bob.# Bob listens on port 61002 and adds Alice as client.pool=Pool()pool.add_http_server(addr="127.0.0.1",port=61002)pool.add_http_client("Alice",addr="127.0.0.1",port=61001)# Bob waits for a message from Alice and prints it.# He replies and shuts down his pool instance.message=awaitpool.recv("Alice")print(message)awaitpool.send("Alice","Hello back to you, Alice!")awaitpool.shutdown()if__name__=="__main__":loop=asyncio.get_event_loop()loop.run_until_complete(async_main())To run this example, run each of the files in a separate terminal window. Note
that ifalice.pyis started prior tobob.py, it will throw aClientConnectorError. Namely, Alice tries to send a message to port 61002,
which has not been opened by Bob yet. After startingbob.py, the error
disappears.The outputs in the two terminals will be something similar to the following:>>>pythonalice.py2022-07-0709:36:20,220-tno.mpc.communication.httphandlers-INFO-Servingon127.0.0.1:610012022-07-0709:36:20,230-tno.mpc.communication.httphandlers-INFO-Receivedmessagefrom127.0.0.1:61002
Hellobacktoyou,Bob!2022-07-0709:36:20,232-tno.mpc.communication.httphandlers-INFO-HTTPServer:Shuttingdownservertask2022-07-0709:36:20,232-tno.mpc.communication.httphandlers-INFO-Server127.0.0.1:61001shutdown>>>pythonbob.py2022-07-0709:36:16,915-tno.mpc.communication.httphandlers-INFO-Servingon127.0.0.1:610022022-07-0709:36:20,223-tno.mpc.communication.httphandlers-INFO-Receivedmessagefrom127.0.0.1:61001
HelloBob!ThisisAlicespeaking.2022-07-0709:36:20,232-tno.mpc.communication.httphandlers-INFO-HTTPServer:Shuttingdownservertask2022-07-0709:36:20,256-tno.mpc.communication.httphandlers-INFO-Server127.0.0.1:61002shutdownTest fixturesThetno.mpc.communicationpackage exports several pytest fixtures as pytest
plugins to facilitate the user in testing with pool objects. The fixtures take
care of all configuration and clean-up of the pool objects so that you don't
have to worry about that.Usage:# test_my_module.pyimportpytestfromtypingimportCallablefromtno.mpc.communicationimportPooldeftest_with_two_pools(http_pool_duo:tuple[Pool,Pool])->None:sender,receiver=http_pool_duo# ... your codedeftest_with_three_pools(http_pool_trio:tuple[Pool,Pool,Pool])->None:alice,bob,charlie=http_pool_trio# ... your code@pytest.mark.parameterize("n_players",(2,3,4))deftest_with_variable_pools(n_players:int,http_pool_group_factory:Callable[[int],tuple[Pool,...]],)->None:pools=http_pool_group_factory(n_players)# ... your codeFixture scopeThe scope of the fixtures can be set dynamically through the--fixture-pool-scopeoption to pytest.Note that this will also change the scope of the globalevent_loopfixture
that is provided bypytest-asyncio.By default, in line withpytest_asyncio, the scope of all our fixtures is"function". We advise to
configure a larger scope (e.g."session","package"or"module") when
possible to reduce test set-up and teardown time.Our fixtures passTrueto theport_reuseargument ofaiohttp.web.TCPSite.
Theirdocumentationstates that this option is not supported on Windows (outside of WSL). If you
experience any issues, please disable the plugin by adding-p no:pytest_tno.tno.mpc.communication.pytest_pool_fixturesto your pytest
configuration. Note that withoutport_reusethe tests may crash, as the test
may try to bind to ports which may not have been freed by the operating system.
For more reliable testing, run the tests on a WSL / Linux platform.