Package: pipe-framework
Pipe FrameworkData-oriented web microframework allows you to create web services with ETL approach instead of traditional MVC.Not related to HBO's Silicon Valley show.IntroductionYou won't find Models, Controllers, and Views in the Pipe framework, but I'll use them to demonstrate its principles.All functionality of the Pipe framework is built using Steps.Step is a self-sufficient, isolated piece of functionality, which does only one thing at a time
(single responsibility principle).Let me explain this concept in detail.For example, you have a simple task to create an API endpoint with todo tasks listing.In the traditional approach, you'll have to create a Todo model, which will represent a database table.
In controller bound to route, you'll use the model instance toextractdata about todo tasks,
thentransformit to the http response and send it to the user.I've markedextractandtransformso you can link MVC concepts to the concepts used in Pipe framework.According to the paragraph above, we can set an analogy betweenMVC(Model-View-Controller) andETL(Extract-Transform-Load):Model -> ExtractorController -> TransformerView -> LoaderThis analogy isn't 100% correct but it demonstrates how parts of the approach are related to each other.As you can see, I've set the view layer as Loader. I'll explain it a bit later.Your first routeLet's implement the above-mentioned task using the Pipe framework.The first thing you should know is there are three kinds of steps:ExtractorTransformerLoaderHow to decide which step do you require?If you need to get the data from an external source: extractor.If you need to send data outside of the framework: loader.If you need to change data in some way: transformer.That's why View is linked with Loader in the example. You can think of this as of loading data to the user browser.First, we need to decompose this functionality into smaller parts.
For this task, we need to do the following:Extract data from a database;Transform data to the JSON HTTP response;Load this response to the browser.So we need 1 extractor, 1 transformer, and 1 loader.
Thankfully, the Pipe framework provides several generic steps, so we can skip some boring parts,
but anyway we need to write our own extractor to set the database access details for the step.AsStepis an independent part of the application not aware of anything going beyond the step purpose, it is easily transferable between apps.
The downside of this solution is that we cannot have a centralized configuration repository.
All configuration applying to steps should be stored in the exact step properties, but sometimes it means that
we need to write the same thing every time we write steps sharing the same configuration.For this purposes, the Pipe framework provides@configuredecorator. You simply write properties you want to add to the step like here:DATABASES={'default':{'driver':'postgres','host':'localhost','database':'todolist','user':'user','password':'','prefix':''}}DB_STEP_CONFIG={'connection_config':DATABASES}and then apply this to the step as in the example below:@configure(DB_STEP_CONFIG)classEDatabase(EDBReadBase):passAs you can tell, the name of the step begins with a capital letter E.
In the Pipe framework, you always implement Extractors, Transformers and Loaders, but it's really hard to keep names short if you use it like this:classExtractTodoFromDatabase(Step):passAll steps should extendStepclass, but implement orextractortransformorloadmethodThat is why all generic steps follow an agreement to indicate the step type with the first letter of the step name:classETodoFromDatabase(Step):passEis for extractor,Tfor transformer andLfor loader.
But of course you are free to use any names you want.So, let's create the project root folder:pipe-sample/Then createsrcfolder inside of thepipe-samplefolder:pipe-sample/
    src/All database-related steps will be in the db package, so let's create it as well:pipe-sample/
    src/
        db/
            __init__.pyCreate aconfig.pyfile, as there will be the database configuration:pipe-sample/src/db/config.pyDATABASES={'default':{'driver':'postgres','host':'localhost','database':'todolist','user':'user','password':'','prefix':''}}DB_STEP_CONFIG={'connection_config':DATABASES}Then create theextract.pyfile to keep our configured extractor:pipe-sample/src/db/extract.pyfromsrc.db.configimportDB_STEP_CONFIG# our configuration"""Pipe framework includes some generics for database as well,you can check them in theAPI documentation"""frompipe.generics.db.orator_orm.extractimportEDBReadBase@configure(DB_STEP_CONFIG)# applying configuration to the stepclassEDatabase(EDBReadBase):pass# we don't need to write anything inside the class,# all logic is already implemented inside EDBReadBaseCreating a whole folder structure could be an overhead for one small task, but this was done here to show the preferred folder structure for other projects.Pretty easy so far. We don't need to repeat these actions with other steps, because they are not
configuration-dependent.Actually, we're ready to create our first pipe now.Createapp.pyin the project root. Then put this code to the file:pipe-sample/app.pyfrompipe.serverimportHTTPPipe,appfromsrc.db.extractimportEDatabasefrompipe.server.http.loadimportLJsonResponsefrompipe.server.http.transformimportTJsonResponseReady@app.route('/todo/')# this decorator tells to the WSGI app that this pipe serves this routeclassTodoResource(HTTPPipe):"""we extending HTTPPipe class which provides pipe_schemaapplicable to the http requests""""""pipe_schema is a dictionary with sub-pipes for every HTTP method.'in' and 'out' is a direction inside the pipe, when pipe handles request,first this request goes through 'in' and then through 'out' pipe.Here we don't need any processing before the response, so only 'out' is presented"""pipe_schema={'GET':{'out':(EDatabase(table_name='todo-items'),TJsonResponseReady(data_field='todo-items_list'),LJsonResponse())}}"""Pipe framework uses Werkzeug as a WSGI server, so configuration is pretty familiar except'use_inspection' argument. Inspection - is a mode for debugging. In case you set it to Truebefore step execution framework will print data state and step name to the console"""if__name__=='__main__':app.run(host='127.0.0.1',port=8080,use_debugger=True,use_reloader=True,use_inspection=True)Now you can execute$ python app.pyand go tohttp://localhost:8000/todo/.Store validationFirst, we need to find out what is store in Pipe framework.StoreAfter pipe is started, but before the first step pipe is calledbefore_pipehook (you can use this hook to perform some operations on the store before executing). This hook accepts initial store, created from initial data passed to the pipe.classBasePipe:def__init__(self,initial,inspection:bool=False):self.__inspection_mode=inspectionself.store=self.before_pipe(frozendict(initial))As you can see, store is nothing but thefrozendictinstance. Although you can't manipulate data inside the store, you can still create a new instance using thefrozendict().copy()method. You can find more information inreadme file.ValidationEven though steps are independent pieces of functionality, sometimes they could require the specific data in the store to perfom certain operations. For this purposes, there are therequired_fieldsfield in step configuration.The Pipe framework usesValideerfor validation, but it's a candidate for the deprecation during the next iterations.ExampleAll you have to do is to write a dict with the required fields (checkValideerfor more information about the available validators).required_fields={'+some_field':valideer.Type(dict)}# `+` means required fieldDynamic validationSometimes, you can have some dynamic fields in step, showing which store field contains required information.
You can't know how this field is named, but you do know in what step variable this value is available.
If you want to validate these fields as well, you'll have to add curly braces, inside which the name of a class field will be.required_fields={'+{pk_field}':valideer.Type(dict)}# everything else is the sameThe Pipe framework will substitute class field value for this field automatically, and then perform validation.Steps arithmeticsYou can combine two or more steps in case you need some conditional execution.In this example, you can see the first operation available -|(OR)pipe_schema={'GET':{'out':(# In this case, if EDatabase step throws# any exception, then LNotFound step will be executed, with information about exception# in storeEDatabase(table_name='todo-items')|LNotFound(),TJsonResponseReady(data_field='todo-items_item'),LJsonResponse())},There is also the second operator -&(AND)pipe_schema={'GET':{'out':(# In this case, if EDatabase step throws# any exception, or SomethingImportantAsWell throws any exception# then nothing happens and store without a change goes to next stepEDatabase(table_name='todo-items')&SomethingImportantAsWell(),TJsonResponseReady(data_field='todo-items_item'),LJsonResponse())},