Package: emergefs
Serverless, object-oriented, computational file system for pythonemerge combines the notion of an object persistent store and service API mesh into one design as a distributed filesystem with computational features.Whereas with current REST/API type architectural patterns, you would do the following:Create database schemas representing your data objects (data modeling)Retrieve data from databases using separate query language (e.g. SQL)Map data query results back into objectsWrite separate code that operates on the re-hydrated objects (business logic)Write separate code still that exposes said operations over a network (API)Write separate code still that translates from incoming requests to said operations and said objects (HTTP)And keeping everything in sync, probably writing more code to make this maintenance headache easierAll these transformations are slow, difficult to manage and get in the way of actual workWith emerge, you simply:Create a custom python class that encapsulates your data and methods that operate on that data.Store the objects in the emerge filesystemRequest those objects by their path (same as a file)Run methods on retrieved objects (locally or remotely: implicit servicing)Celebrate!BenefitsNo Database schemas or query languagesNoImpedence Mismatchesbetween layersNo visible middlewareNo protocol hasslesNo API scaffoldingUltra fastUltra simplePlain Old Python end-to-endScriptable CLI for interacting with filesystemUsage$emerge
Usage:emerge[OPTIONS]COMMAND[ARGS]...

Options:--debugDebugswitch--helpShowthismessageandexit.

Commands:callCallanobjectmethodcatDisplaycontentsofanobjectcodeListsourcecodeofanobjectcpCopyobjectcommandhelpDisplaydetailsofanobjectsclasslsListfilesinadirectorymethodsDisplayavailablemethodsforanobjectmkdirMakedirectorycommandnodeEmergenodecommandsqueryExecutequerymethodofanobjectrmRemoveobjectcommandExamplesfromemerge.core.clientimportClientfromemerge.core.objectsimportEmergeFile@dataclassclassInventoryItem(EmergeFile):""" Create your own classes on-the-fly """unit_price:float=0.0quantity_on_hand:int=0deftotal_cost(self)->float:returnself.unit_price*self.quantity_on_handitem=InventoryItem(id="widget1",name="widget",path="/inventory",unit_price=3.0,quantity_on_hand=10,data="A widget{}data".format(1),)client=Client("0.0.0.0","5558")# Connect to any node# Stores item under path /inventory/widgetclient.store(item)Execute object methods as-a-serviceNOTE: Method runs on the host containing the objectclient.run("/inventory/widget","total_cost")Retrieve object and run methods on it locallywidget=client.get("/inventory/widget")total_cost=widget.total_cost()Browse FilesystemBrowse the emerge object filesystem from command line:$emergels
inventory
$emergels-lrwxrwxrwx1Aug31202214:08:37inventory$emergels-l/inventory
rwxrwxrwx5.6KAug31202214:08:37widget$emergecat/inventory/widget{"name":"widget","path":"/inventory","id":"widget1","price":3.0,"quantity":10,"perms":"rwxrwxrwx","type":"file","data":"A widget1 data"}$Listing the directory, shows the directory contents length in the size column. In this case "1".
For the widget object, the object size is shown 5.6KExecute Remote MethodsNOTE: Method executes on the server, only results are returned$emergecall/inventory/widgettotal_cost30.0Important to understand that the custom classInventoryItemwith its methods was created in the client, on-the-fly and stored in emerge.
No code update, recompile, redeploy of the emerge network is needed.Execute Local MethodsNOTE: Object (including its state/data) is retrieved from the server and the method invoked in the local process$emergecall--local/inventory/widgettotal_cost30.0Execute A Method on All Objects in a DirectoryNOTE: Execution occurs on the server$emergecall/inventorytotal_cost[30.0,434.10034997376107,137.4442513978148,907.5910237092307,129.6896355950612,705.798940493996,589.6721640143035,1408.5984146794274,1722.7589088095774]Getting Help on an Object$emergehelp/inventory/widget1
HelponInventoryIteminmodule__main__object:

classInventoryItem(emerge.core.objects.EmergeFile)|InventoryItem(id:str,data:str='',name:str='',path:str='/',perms:str='rwxrwxrwx',type:str='file',unit_price:float=0.0,quantity_on_hand:int=0)->None||Classforkeepingtrackofanitemininventory.||Methodresolutionorder:|InventoryItem|emerge.core.objects.EmergeFile|emerge.core.objects.EmergeObject|emerge.data.EmergeData|persistent.Persistent|builtins.object||Methodsdefinedhere:||__eq__(self,other)||__init__(self,id:str,data:str='',name:str='',path:str='/',perms:str='rwxrwxrwx',type:str='file',unit_price:float=0.0,quantity_on_hand:int=0)->None
:Getting Methods of an Object$emergemethods/inventory/widget7
run()total_cost()->floatShowing the Source Code of an Object$emergecode/inventory/widget1
@dataclass
classInventoryItem(EmergeFile):"""Class for keeping track of an item in inventory."""unit_price:float=0.0quantity_on_hand:int=0defrun(self):return"total cost:{}".format(self.total_cost())deftotal_cost(self)->float:importlogginglogging.debug("InventoryItem: total_cost executing")returnself.unit_price*self.quantity_on_handdef__str__(self):importjsonreturnjson.dumps({"name":self.name,"path":self.path,"id":self.id,"unit_price":self.unit_price,"quantity_on_hand":self.quantity_on_hand,"perms":self.perms,"type":self.type,"data":self.data,})Getting Objectsemergeis a distributed and federated namespace platform serving python objects like a filesystem.
When requesting an object by path, you can connect to any node in the filesystem to make this request.
If that node does not host the object, it will consult the broker which will have a reference to where the file path is being held and retrieve it from that node.In the example output below, let's assume we have stored the object/inventory/widget1on the broker node.
The node atlocalhost:5559does not have knowledge of that object.When we attempt to asklocalhost:5559for/inventory/widgetit first checks its own registry for that path, if it doesn't have it, it passes the request to the broker and you get the expected result.The second callemerge cat /inventory/widget1defaults to requesting the object from the broker.$emerge-hlocalhost:5559cat/inventory/widget1{"data":"A widget1 data","date":"Jan 15 2023 03:00:31","id":"widget1","name":"widget1","node":"","path":"/inventory","perms":"rwxrwxrwx","quantity_on_hand":10,"totalcost":30.0,"type":"file","unit_price":3.0,"uuid":"af9720c3-01f8-43bd-b0a4-6988b0061997"}$emergecat/inventory/widget1{"data":"A widget1 data","date":"Jan 15 2023 03:00:31","id":"widget1","name":"widget1","node":"","path":"/inventory","perms":"rwxrwxrwx","quantity_on_hand":10,"totalcost":30.0,"type":"file","unit_price":3.0,"uuid":"af9720c3-01f8-43bd-b0a4-6988b0061997"}Querying the FilesystemQuery With a Lambdaemergedoesn't have a DSL query language, it usesplain-old-python! This removes a lot of layers, mappings, serializers and other complexities used between traditional SQL databases, ORMs, objects etc.
Simpler is better!fromemerge.core.clientimportClientclient=Client("0.0.0.0","5558")results=client.search(lambdao:hasattr(o,'unit_price')ando.unit_price<3.5)print(results)['{"data": "A widget1 data", "id": "widget1", "name": "widget1", "path": "/inventory", "perms": "rwxrwxrwx", "quantity_on_hand": 10, "totalcost": 0.0, "type": "file", "unit_price": 3.0, "uuid": "c959c600-9404-4d94-b8d4-135b8e631da1"}','{"data": "A widget1 data", "id": "widget1", "name": "widget1", "path": "/inventory", "perms": "rwxrwxrwx", "quantity_on_hand": 10, "totalcost": 0.0, "type": "file", "unit_price": 3.0, "uuid": "ec7a3450-c02e-4405-a522-55f56c48106f"}','{"data": "A widget2 data", "id": "widget2", "name": "widget2", "path": "/inventory", "perms": "rwxrwxrwx", "quantity_on_hand": 11, "totalcost": 0.0, "type": "file", "unit_price": 3.4092252235237934, "uuid": "86b6802a-5fad-4305-95ec-ae56c7e1d1ee"}','{"data": "A widget3 data", "id": "widget3", "name": "widget3", "path": "/inventory", "perms": "rwxrwxrwx", "quantity_on_hand": 26, "totalcost": 0.0, "type": "file", "unit_price": 3.2701771680050653, "uuid": "157676d2-80fc-4737-a895-a3ed69809fa6"}','{"data": "A widget1 data", "id": "widget1", "name": "widget1", "path": "/inventory", "perms": "rwxrwxrwx", "quantity_on_hand": 10, "totalcost": 0.0, "type": "file", "unit_price": 3.0, "uuid": "57c2dc38-beef-461e-b77b-a736656a2a61"}']Using a QueryObjectGiven the following object with a definedquerymethod, using thequerycommand will invoke the query method passing in a reference to the filesystem for the method to query.@dataclassclassQueryFile(EmergeFile):importpersistent.listresults=persistent.list.PersistentList()defquery(self,fs):"""This only runs on the server and receives the filesystem object to traverse"""self.results=[]forobjinfs.dir("/inventory"):ifobj.unit_price<15:self.results.append(obj)return[str(result)forresultinself.results]Invoke the query object$emergequery/queries/query1[{"name":"widget1","path":"/inventory","id":"widget1","unit_price":3.0,"quantity_on_hand":10,"perms":"rwxrwxrwx","type":"file","data":"A widget1 data"},{"name":"widget2","path":"/inventory","id":"widget2","unit_price":14.470011665792036,"quantity_on_hand":30,"perms":"rwxrwxrwx","type":"file","data":"A widget2 data"},{"name":"widget3","path":"/inventory","id":"widget3","unit_price":10.57263472290883,"quantity_on_hand":13,"perms":"rwxrwxrwx","type":"file","data":"A widget3 data"},{"name":"widget1","path":"/inventory","id":"widget1","unit_price":3.0,"quantity_on_hand":10,"perms":"rwxrwxrwx","type":"file","data":"A widget1 data"},{"name":"widget2","path":"/inventory","id":"widget2","unit_price":14.470011665792036,"quantity_on_hand":30,"perms":"rwxrwxrwx","type":"file","data":"A widget2 data"},{"name":"widget3","path":"/inventory","id":"widget3","unit_price":10.57263472290883,"quantity_on_hand":13,"perms":"rwxrwxrwx","type":"file","data":"A widget3 data"},{"name":"widget1","path":"/inventory","id":"widget1","unit_price":3.0,"quantity_on_hand":10,"perms":"rwxrwxrwx","type":"file","data":"A widget1 data"},{"name":"widget2","path":"/inventory","id":"widget2","unit_price":14.470011665792036,"quantity_on_hand":30,"perms":"rwxrwxrwx","type":"file","data":"A widget2 data"},{"name":"widget3","path":"/inventory","id":"widget3","unit_price":10.57263472290883,"quantity_on_hand":13,"perms":"rwxrwxrwx","type":"file","data":"A widget3 data"},{"name":"widget1","path":"/inventory","id":"widget1","unit_price":3.0,"quantity_on_hand":10,"perms":"rwxrwxrwx","type":"file","data":"A widget1 data"},{"name":"widget2","path":"/inventory","id":"widget2","unit_price":14.470011665792036,"quantity_on_hand":30,"perms":"rwxrwxrwx","type":"file","data":"A widget2 data"},{"name":"widget3","path":"/inventory","id":"widget3","unit_price":10.57263472290883,"quantity_on_hand":13,"perms":"rwxrwxrwx","type":"file","data":"A widget3 data"}]Retrieving Stored Results LaterThe query class may retain the results as in the example above. This way, another client can simply request the currently stored results without re-executing the queryfromemerge.core.clientimportClientclient=Client("0.0.0.0","5558")# Just retrieve the stored query objectquery=client.getobject("/queries/query1",False)# And access its data like a regular python objectforresultinquery.results:print(round(result.total_cost(),1))Output30.0434.1137.430.0434.1137.430.0434.1137.430.0434.1137.4Using Object ProxiesEmerge can proxy an object you created on-the-fly and stored in the filesystem so you can invoke methods on it directly, rather than using theclientobject to execute methods.fromemerge.core.clientimportClientclient=Client("0.0.0.0","5558")widget1=client.proxy("/inventory/widget1")# Invoke method on serverprint(widget1.total_cost())widgets=client.list("/inventory")print(widgets)proxies=[client.proxy(widget)forwidgetinwidgets]# Executes each method on the serverprint([proxy.total_cost()forproxyinproxies])query=client.proxy("/queries/query1")# Access member as if local. results is lazy loaded from the serverforresultinquery.results:print(result)print(round(result.total_cost(),1))GraphQL queriesemergewill automatically create the required GraphQL schemas and resolvers for your objects!Given the following class you create an object and insert into the filesystem.fromdataclassesimportdataclassfromemerge.core.clientimportClientfromemerge.core.objectsimportEmergeFile@dataclassclassInventoryItem(EmergeFile):"""Class for keeping track of an item in inventory."""unit_price:float=0.0quantity_on_hand:int=0totalcost:float=0foo:str="FOO"defrun(self):return"total cost:{}".format(self.total_cost())deftotal_cost(self)->float:importlogginglogging.debug("InventoryItem: total_cost executing")self.totalcost=self.unit_price*self.quantity_on_handreturnself.totalcostclient=Client("0.0.0.0","5558")item=InventoryItem(id="widget1",name="widget1",path="/inventory",unit_price=3.0,quantity_on_hand=10,data="A widget{}data".format(1),)client.store(item)Onceemergeas received and stored your custom object/class it will generate the GraphQL artifacts needed to query it.
Internally,emergecreates and maintains optimize indicies over your custom object fields dynamically, making it super-fast to execute queries in memory.
Then issue a GraphQL query to find it$catquery1.jsonquery{InventoryItem(name:"widget1"){nameidpath}}$emergegraphql"$(catquery1.json)"{'InventoryItem':{'name':'widget1','id':'widget1','path':'/inventory'}}$catquery3.jsonquery{InventoryItemList{nameidpathuuidtotalcost}}$emergegraphql"$(catquery3.json)"{'InventoryItemList':[{'name':'widget8','id':'widget8','path':'/inventory','uuid':'3fd460a1-07a7-4a3a-8804-87ef0c4cee4f','totalcost':0.0},{'name':'widget3','id':'widget3','path':'/inventory','uuid':'4d7d8d2d-2811-4444-a098-ac1c652073f7','totalcost':0.0},{'name':'widget9','id':'widget9','path':'/inventory','uuid':'8b8c4a11-1e01-4a9f-9070-9ea31e18240d','totalcost':0.0},{'name':'widget4','id':'widget4','path':'/inventory','uuid':'8ba02fc6-5d59-4b03-ae2a-2f4807acb879','totalcost':0.0},{'name':'query1','id':'query1','path':'/queries','uuid':'b2e188fe-e693-4338-a619-71e8d9fa6690','totalcost':None},{'name':'widget1','id':'widget1','path':'/inventory','uuid':'c959c600-9404-4d94-b8d4-135b8e631da1','totalcost':0.0},{'name':'query1','id':'query1','path':'/queries','uuid':'d7ab0d1f-7bf7-4962-9d34-d92bb3ff7069','totalcost':None}Running a Node$emerge--debugnodestart2022-08-3110:48:34,261:rootDEBUG:DebugON2022-08-3110:48:34,322:rootINFO:StartingNodeServeronport:55582022-08-3110:48:34,322:rootDEBUG:[NodeServer]Setup...2022-08-3110:48:34,322:rootDEBUG:[NodeServer]start...2022-08-3110:48:34,323:rootDEBUG:SubscribedtoNODEontcp://0.0.0.0:55572022-08-3110:48:34,323:rootDEBUG:get_messages2022-08-3110:48:34,323:rootDEBUG:Waitingonmessage2022-08-3110:48:34,324:rootINFO:Z0DBFileSystemsetup2022-08-3110:48:34,324:ZODB.BaseStorageDEBUG:createstorageemerge.fs2022-08-3110:48:34,324:txn.139737146840640DEBUG:newtransaction2022-08-3110:48:34,324:txn.139737146840640DEBUG:commit2022-08-3110:48:34,325:rootINFO:Z0DBFileSystemstart2022-08-3110:48:34,325:zerorpc.eventsDEBUG:boundtotcp://0.0.0.0:5558(status=<SocketContext(bind='tcp://0.0.0.0:5558')>)Using Docker Composeemergecomes with a docker compose file to run a broker node and a secondary node.$makeup
$makedown
$makeclean