Package: tlspyo
tls-python-object (tlspyo):computer: :globe_with_meridians: :computer:A library for easy and secure transfer of python objects over network.:rocket:Quickstart guide:scroll:API documentationtlspyoprovides a simple API to transfer python objects in a robust and safe way viaTLS, between several machines (and/or processes) calledEndpoints.Endpointsare part of one to several groups,Arbitrarily manyEndpointsconnect together via a centralRelay,EachEndpointcanbroadcastorproducepython objects to the desired groups.:information_source:Please carefully read theSecuritysection before usingtlspyoanywhere other than your own secure private network.Quick linksDocumentationPrincipleExample usageGetting startedInstallationTLS setupProducer-consumer exampleSecurityCustom serializationPrincipletlspyoprovides two classes:RelayandEndpoint.TheRelayis the center point of all communication betweenEndpoints,AnEndpointis a node in your network. It connects to theRelayand is part of one to severalgroups.Endpointscan do a multitude of things, including:broadcastpython objects to whole groups ofEndpoints,retrievethe objects broadcast to the group(s) it is part of,producea single object that will be consumed by a singleEndpointof a target group,notifytheRelaythat it is ready to consume a produced object and wait until it receives it.By default,tlspyorelies on Transport Layer Security (TLS) to secure object transfers over network.Example usagefromtlspyoimportRelay,Endpointif__name__=="__main__":# Create a relay to allow connectivity between endpointsre=Relay(port=3000,# this must be the same on your Relay and Endpointspassword="VerySecurePassword",# must be the same on Relay and Endpoints, AND be stronglocal_com_port=3001# needs to be non-overlapping if Relays/Endpoints are on the same machine)# Create an Endpoint in group "producers" (arbitrary name)prod=Endpoint(ip_server='127.0.0.1',# IP of the Relay (here: localhost)port=3000,# must be same port as the Relaypassword="VerySecurePassword",# must be same (strong) password as the Relaygroups="producers",# this endpoint is part of the group "producers"local_com_port=3002)# Create a bunch of other Endpoints in group "consumers" (arbitrary name)cons_1=Endpoint(ip_server='127.0.0.1',port=3000,password="VerySecurePassword",groups="consumers",# this endpoint is part of group "consumers"local_com_port=3003)cons_2=Endpoint(ip_server='127.0.0.1',port=3000,password="VerySecurePassword",groups="consumers",# this endpoint is part of group "consumers"local_com_port=3004,)# Producer broadcasts an object to any and all endpoint in the destination group "consumers"prod.broadcast("I HAVE BEEN BROADCAST","consumers")# Producer sends an object to the shared queue of destination group "consumers"prod.produce("I HAVE BEEN PRODUCED","consumers")# Consumer 1 notifies the Relay that it wants one produced object destined for "consumers"cons_1.notify("consumers")# Consumer 1 is able to retrieve the broadcast AND the consumed object:res=[]whilelen(res)<2:res+=cons_1.receive_all(blocking=True)print(f"Consumer 1 has received:{res}")# Consumer 2 is able to retrieve only the broadcast object:res=cons_2.receive_all(blocking=True)print(f"Consumer 2 has received:{res}")# Let us close everyone gracefully:prod.stop()cons_1.stop()cons_2.stop()re.stop()Getting started:information_source:The machine hosting yourRelaymust be visible to the machines hosting yourEndpointsthrough  the chosenport, via its publicip_server.
When usingtlspyoover the Internet, this typically requires you to configure your router such that it forwardsportto the IP of the machine hosting yourRelayon your local network.InstallationFrom PyPI:pipinstalltlspyoTLS setup::information_source:You can skip this section if you do not want to use TLS.
For instance if you usetlspyoon your own private secure network.
When usingtlspyoover the Internet, you should of course use TLS (read thesecuritysection if you do not understand why).Generate TLS credentials:tlspyomakes the process of generating your TLS credentials straightforward.:arrow_forward: On the machine that will host yourRelay, execute the following command line:python-mtlspyo--generateThis will generate two files in thetlspyo/credentialsdata directory:key.pemandcertificate.pem.:information_source:In case you wish to customize your TLS certificate, add the--customoption in the previous command line.Now, your need to retrieve yourcertificate.pemon the machines that will host yourEndpoints(note: you can skip the following steps if yourEndpointsare on the same machine as yourRelay).This can be achieved via either of the following methods:METHOD 1: manually copy the public certificate (more secure)::arrow_forward: On the machines that will host yourEndpoints, execute:python-mtlspyo--credentialsThis creates and displays the target folder where you need to copy thecertificate.pemthat you generated on the machine that will host theRelay(the source folder was displayed when you executed--generate).METHOD 2: transfer the public certificate via TCP (not secure)::warning:This method is not secure.
In particular, a man-in-the-middle can impersonate the certificate-broadcasting server and send you a fraudulent TLS certificate.
Use with caution.:arrow_forward: On the machine that will host yourRelay, start a certificate-broadcasting server:python-mtlspyo--broadcast--port=<port>where<port>is a port through which other machines will attempt to retrieve your certificate via TCP.:arrow_forward: On the machines that will host yourEndpoints, execute:python-mtlspyo--retrieve--ip=<ip>--port=<port>where<ip>is the public IP of the certificate-broadcasting machine, and<port>is the same as previously.And you are all set! :sunglasses:You can now stop the certificate-broadcasting server by closing the terminal where it runs.A Simple Producer-Consumer ExampleLet us now see how to make basic usage oftlspyo.
In this example, we will create aRelayand twoEndpointson the same machine, and have them transfer objects vialocalhost.
The full script for this example can be foundhere.Import theRelayandEndpointclasses:fromtlspyoimportRelay,EndpointRelayEverytlspyoapplication requires a centralRelay.TheRelaylives on a machine that can be reached by allEndpoints.
Typically, you will want this machine to be accessible to yourEndpointsvia your private local network, or via the Internet throughport forwarding.Note however that, before you make yourRelayvisible to the Internet via, e.g., port forwarding, it is important that you read theSecuritysection.Creating aRelayis straightforward:# Initialize a relay to allow connectivity between endpointsre=Relay(port=3000,# this must be the same on your Relay and Endpointspassword="VerySecurePassword",# this must be the same on Relay and Endpoints, AND be stronglocal_com_port=3001,# this needs to be non-overlapping if Relays/Endpoints live on the same machinesecurity="TLS"# this is the default; replace by None if you do not want to use TLS)As soon as yourRelayis created, it is up and running.
Behind the scenes, it is now waiting for TLS connections fromEndpoints.
This is done in a background process that listens toport3000 in this example.
This process also communicates with yourRelayvialocal_com_port3001 in this example.Usually, you can ignorelocal_com_portand leave it to the default, unless you use severalEndpoints/Relayon the same machine, which we will do.EndpointsNow that ourRelayis ready, let us create a bunch ofEndpoints.
This is also pretty straightforward:# Initialize a producer endpointprod=Endpoint(ip_server='127.0.0.1',# IP of the Relay (here: localhost)port=3000,# must be same port as the Relaypassword="VerySecurePassword",# must be same (strong) password as the Relaygroups="producers",# this endpoint is part of the group "producers"local_com_port=3002,# must be uniquesecurity="TLS"# this is the default; replace by None if you do not want to use TLS)# Initialize  consumer endpointscons_1=Endpoint(ip_server='127.0.0.1',port=3000,password="VerySecurePassword",groups="consumers",# this endpoint is part of group "consumers"local_com_port=3003,# must be uniquesecurity="TLS")cons_2=Endpoint(ip_server='127.0.0.1',port=3000,password="VerySecurePassword",groups="consumers",# this endpoint is part of group "consumers"local_com_port=3004,# must be uniquesecurity="TLS")A nice thing abouttlspyois that all communication is handled behind the scenes.
The above calls have all launched processes in the background which handle connection and communication betweenEndpointsthrough theRelay.Let us now send some objects from the producer to the consumers.
As you may have noticed, we created two different groups here.
We put the producer in a group that we have named "producers", and the consumers in another group that we have called "consumers".
Note thatEndpointcan be created as being part of any number of groups (groupscan take a list of strings).
When communicating between endpoints, you can use those groups to make sure the right endpoints receive the right objects.There are two ways forEndpointsto send objects intlspyo:Broadcastingis used to send an object to all endpoint in a given group.
Furthermore, when anEndpointconnects to theRelay, it receives the last object that was broadcast to each of his groups.# Producer broadcasts an object to any and all endpoint in the destination group "consumers"prod.broadcast("I HAVE BEEN BROADCAST","consumers")Producingis used to send an object to a queue (FIFO) that is shared between allEndpointsof a given group.
The endpoints of the receiving group mustNotifytheRelayto get access to an object that has been put in that shared queue.# Producer sends an object to the shared queue of destination group "consumers"prod.produce("I HAVE BEEN PRODUCED","consumers")# Consumer notifies the Relay that it wants one produced object destined for "consumers"cons_1.notify("consumers")Once objects reach the consumer endpoint, they are stored in a local queue from which you can retrieve objects whenever you want. To do this, there are multiple options:To retrieve from the local queue in a FIFO fashion, usepop(blocking=blocking, max_items=max_items).To retrieve the most recent item(s) in the local queue and discard the rest, useget_last(blocking=blocking, max_items=max_items).To get all items that are currently in the local queue, usereceive_all(blocking=blocking).:information_source:Notes:All calls above return a list of objects. If no objects are returned, the result will be an empty list.IfblockingisTrue, all methods above will block until at least one item is received (default toFalse).Inpopandget_last, usemax_itemsto specify a maximum number of items to be returned (defaults to 1).Now, let our consumers retrieve their loot:# Consumer 1 is able to retrieve the broadcast AND the consumed object:res=[]whilelen(res)<2:res+=cons_1.receive_all(blocking=True)print(f"Consumer 1 has received:{res}")# Consumer 2 is able to retrieve only the broadcast object:res=cons_2.receive_all(blocking=True)print(f"Consumer 2 has received:{res}")which prints:Consumer 1 has received: ['I HAVE BEEN BROADCAST', 'I HAVE BEEN PRODUCED']
Consumer 2 has received: ['I HAVE BEEN BROADCAST']Once we are done, we canstopallEndpoints, and then theRelayfor the sake of a graceful exit:# Let us close everyone gracefully:prod.stop()cons_1.stop()cons_2.stop()re.stop()There you go! You have now sent your first object over the network usingtlspyo.Please check out theAPI documentationfor more advanced usage.SecurityDISCLAIMERWe are doing our best to maketlspyoreasonably securewhen used correctly, but we provide ABSOLUTELY NO GUARANTEE that it is in any sense.
We are a small open-source community, and we greatly appreciate your contribution to tackle any potentially unreasonable security concerns or important missing information.
Please submit a detailed issue if you are aware of any important exploit not covered in this section.Implementationtlspyorelies on theTwistedframework regardingTLSimplementation and network management.Important to know:warning: Objects transferred bytlspyoare serialized withpickleby default, so that you can transfer most python objects easily.NEVER TRANSFER PICKLED OBJECTS OVER A PUBLIC NETWORK WITHOUTtlspyo, as this would make you vulnerable todangerous exploits.
This is because unpickling untrusted pickled objects (i.e., pickled objects created by a malicious user) can lead to arbitrary code execution on your machine.To prevent this from happening,tlspyoprovides two interdependent layers of security:Endpointsauthenticate yourRelayviaTLS, which must useyour own secret key and public certificate.
This ensures yourEndpointsare indeed talking to yourRelayand not to someman-in-the-middle,provided you keep your secret key secure.
This also prevents anyone else fromeavesdroppingthanks to TLS encryption.Every object transfer is protected by a password known to both theRelayand theEndpoints(thepasswordargument).
No object is deserialized without verification of the password.
This ensures that anyone posing as an endpoint will never be able to send undesired objects through your relayunless they know your password.If a malicious user successfully posed as yourRelay, yourEndpointwould send them messages that they could decrypt, including your password (this is prevented by TLS when using your own secret key and public certificate).
If they successfully posed as yourEndpointthey could send malicious pickled objects to yourRelay(this is prevented by them not knowing your password).In a nutshell,when usingtlspyoyou want your password to be as strong as possible, and your TLS secret key to be kept... well, secret:lock:For safety-critical applications, we recommend you ditchpicklealtogether and instead code a secure custom serialization protocol, on top of the TLS layer provided bytlspyo.Custom serializationBy default,tlspyousespicklefor serialization and relies on TLS to prevent attacks.In advanced application, you may want to use another serialization protocol instead.
For instance, you may want to transfer non-picklable objects, further optimize the security of your application, or simply use apickleserialization protocol or your choice instead of your Python's default.In particular, insecurity=Nonemode (i.e., with TLS disabled) over a public network, using your own secure serialization protocol is critical.tlspyomakes this easy.
All you need to do is code your own serialization protocol following thepickle.dumps/pickle.loadssignature, and pass it to theserializer/deserializerarguments of both yourRelayandEndpoints.For instance:importpickleaspklfromtlspyoimportRelay,Endpoint# We define a custom serialization protocol based on pickle for simplicity.# Of course, this is only for illustration.# In practice, you may not want to use pickle here.defmy_custom_serializer(obj):"""Takes a python object as input and outputs a bytestring"""returnb"header"+pkl.dumps(["TEST",pkl.dumps(obj)])defmy_custom_deserializer(bytestring):"""Takes a bytestring as input and outputs a python object"""assertlen(bytestring)>len(b"header")assertbytestring[:len(b"header")]==b"header"bytestring=bytestring[len(b"header"):]tmp=pkl.loads(bytestring)assertisinstance(tmp,list)assertlen(tmp)==2asserttmp[0]=="TEST"obj=pkl.loads(tmp[1])returnobjif__name__=='__main__':re=Relay(port=3000,password="VerySecurePassword",local_com_port=3001,security="TLS",serializer=my_custom_serializer,deserializer=my_custom_deserializer)ep=Endpoint(ip_server='127.0.0.1',port=3000,password="VerySecurePassword",groups="group1",local_com_port=3002,security="TLS",serializer=my_custom_serializer,deserializer=my_custom_deserializer)External linkstlspyois an open-source project hosted atPolytechnique Montreal - MISTlab.
We use it in various projects, ranging from parallelmeta-learningto data transfer between multiplelearning robots.tlspyorelies onTwistedto manage network robustness and security.ContributingContributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make aregreatly appreciated.If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".
Don't forget to give the project a star! Thanks again!Fork the ProjectCreate your Feature Branch (git checkout -b feature/AmazingFeature)Commit your Changes (git commit -m 'Add some AmazingFeature')Push to the Branch (git push origin feature/AmazingFeature)Open a Pull RequestLicenseDistributed under the MIT License. SeeLICENSE.txtfor more information.