Package: lilly
LillyLilly is fast service-oriented and layered Python 3.6+ web framework built on top ofFastAPIIt is enforces a certain way of creating FastApi applications that is much easier to reason about.
Since it is based on FastAPI, it is modern, fast (high performance), and works well with Python type hints.PurposeLilly signifies peaceful beauty.Lillyis thus an opinionated framework that ensures clean beautiful
code structure that scales well for large projects and large teams.It just adds more opinionated structure to the already beautifulFastAPI.It ensures that when someone is building a web application basing on Lilly, they don't need to think about the structure.The developer should just know that it is a service-oriented architecture with each service having a layered architecture
that ensures layers don't know what the other layer is doing.Key FeaturesOn top of thekey features of FastAPIwhich include:Fast. It is based on FastApiIntuitive: Great editor support. Completion everywhere. Less time debugging.Easy: Designed to be easy to use and learn. Less time reading docs.Short: Minimize code duplication. Multiple features from each parameter declaration. Fewer bugs.Robust: Get production-ready code. With automatic interactive documentation.Standards-based: Based on (and fully compatible with) the open standards for APIs: OpenAPI (previously known as
Swagger) and JSON Schema.It also:Enforces a separation of concerns between service to serviceEnforces a separation of concerns within the service between presentation, business, persistence, and data_source
layersQuick StartEnsure you havePython 3.7or +3.7 installedCreate a new folder for your applicationmkdirlilly_sample&&cdlilly_sampleCreate the virtual environment and activate itpython3-mvenvenvsourceenv/bin/activateInstall lillypipinstalllillyCreate your first application based off the frameworkpython-mlillycreate-appThis will create the following folder structure with some fully functional sample code.├──main.py├──settings.py└──services├──__init__.py└──hello├──__init__.py├──actions.py├──datasources.py├──dtos.py├──repositories.py└──routes.pyInstall uvicorn and run the apppipinstalluvicorn
uvicornmain:app--reloadView the OpenAPI docs athttp://127.0.0.1:8000/docsFor you to add another service in the services folder, run the command:python-mlillycreate-service<service-name>e.g.python-mlillycreate-serviceblogFor more information about the commands, just run thehelpcommandspython-mlilly--help
python-mlillycreate-app--help
python-mlillycreate-service--helpHow to Run testsClone the repositorygitclonegit@github.com:sopherapps/lilly.git&&cdlillyCreate a test postgres database if you have not yetsudo-supostgres
createdb<test_db_name>exitCopy the.example.envfile to.envcp.example.env.envUpdate theTEST_DATABASE_URLto the URL of your test postgres database in the.envfileCreate virtual environment for Python 3.7 and above and activate itpython3-mvenvenvsourceenv/bin/activateInstall requirementspipinstall-rrequirements.txtRun the test commandpython-munittestUsageLilly can be used easily in your app.To create a new app, we use the command:python-mlillycreate-app<app-name>To add another service in the service folder, we use the command:python-mlillycreate-service<service-name>These two commands create a starting point with a sample fully-functional web app whose docs can be found athttp://127.0.0.1:8000/docswhen the app is run locally with the command.uvicornmain:app--reloadThe twocreatecommands typically create a service folder with the follwoing structure└──<service-name>├──__init__.py├──actions.py├──datasources.py├──dtos.py├──repositories.py└──routes.pyTheActions can be found in theactions.pymodule. Customize them accordingly following the guidance of the already
existing code.TheDataSources can be found in thedatasources.pymodule. Customize them accordingly following the guidance of the
already existing code.TheRepositorys can be found in therespositories.pymodule. Customize them accordingly following the guidance of
the already existing code.TheRouteSets can be found in theroutes.pymodule. Customize them accordingly following the guidance of the already
existing code.TheDataModelDTOs can be found in thedtos.pymodule. Customize them accordingly following the guidance of the
already existing code.Data SourcesTo create a new data source, one needs to subclass theDataSourceclass and override theconnect(self)method.fromtypingimportContextManagerfromlilly.datasourcesimportDataSourceclassSampleConnectionContextManager:def__init__(self,connection):self.connection=connectiondef__enter__(self):returnself.connectiondef__exit__(self,exc_type,exc_val,exc_tb):self.connection.close()classSampleDataSource(DataSource):defconnect(self)->ContextManager:# do some stuff and return a context manager for a connectionpassTo make life easier for the developer, we have created a few DataSources that can be used or overridden. They include:1. SQLAlchemyDataSourceThis connects to any relational database e.g. MySQL, PostgreSQL, Sqlite etc.
usingSQLAlchemyIt can be used in a repository as in this example:fromtypingimportAnyfromsqlalchemy.ormimportdeclarative_basefromsqlalchemyimportColumn,Integer,Stringfromlilly.repositoriesimportRepositoryfromlilly.datasourcesimportSQLAlchemyDataSource,DataSourcefromlilly.confimportsettingsBase=declarative_base()classUserModel(Base):"""The database model for users"""__tablename__="users"id=Column(Integer,primary_key=True)name=Column(String)email=Column(String)classUsersRepository(Repository):"""Repository for saving and retrieving users"""_users_db=SQLAlchemyDataSource(db_uri=settings.DATABASE_URL,declarative_meta=Base)# -- other important methods need to be overridden also. I have excluded them for brevity.@propertydef_datasource(self)->DataSource:returnself._users_dbRepositoriesTo create a new repository, one needs to subclass theRepositoryclass and override all the following methods:_get_one(self, datasource_connection: Any, record_id: Any, **kwargs) -> Anymethod to get one record of
idrecord_id_get_many(self, datasource_connection: Any, skip: int, limit: int, filters: Dict[Any, Any], **kwargs) -> List[Any]method to get many records that fulfil thefilters_create_one(self, datasource_connection: Any, record: BaseModel, **kwargs) -> Anymethod to create one record_create_many(self, datasource_connection: Any, record: List[BaseModel], **kwargs) -> List[Any]method to create many
records_update_one(self, datasource_connection: Any, record_id: Any, new_record: BaseModel, **kwargs) -> Anymethod to
update one record of idrecord_id_update_many(self, datasource_connection: Any, new_record: BaseModel, filters: Dict[Any, Any], **kwargs) -> Anymethod to update many records that fulfil thefilters_remove_one(self, datasource_connection: Any, record_id: Any, **kwargs) -> Anymethod to remove one record of
idrecord_id_remove_many(self, datasource_connection: Any, filters: Dict[Any, Any], **kwargs) -> Anymethod to remove many
records that fulfil thefilters_datasource(self) -> DataSourcean @property-decorated method to return the DataSource whoseconnect()method is
to be called in any of the other methods to get its instance._to_output_dto(self, record: Any) -> BaseModelmethod which converts any record from the data source raw to DTO
for the public methodsA good example is how we implemented theSQLAlchemyRepository. Feel free to look at it.To make life easier for the developer, we have created a few off-the-shelfRepositorysubclasses with most of those methods implemented.
They just need to be inherited and a few abstract methods filled with one-liners (or slightly more than one-liners if you wish).These include:1. SQLAlchemyRepositoryThis connects to any relational database e.g. MySQL, PostgreSQL, Sqlite etc. usingSQLAlchemyvia theSQLAlchemyDataSourcedata source class.Here is a sample of its usage:fromtypingimportTypefrompydanticimportBaseModelfromsqlalchemyimportColumn,Integer,Stringfromsqlalchemy.ormimportDeclarativeMeta,declarative_basefromlilly.repositoriesimportSQLAlchemyRepositoryfromlilly.datasourcesimportSQLAlchemyDataSourcefromlilly.confimportsettingsfrom.dtosimportNameRecordDTO# a subclass of pydantic.BaseModel that is a Data Transfer Object for Name typesBase=declarative_base()className(Base):__tablename__="names"id=Column(Integer,primary_key=True)title=Column(String,nullable=False)classNamesRepository(SQLAlchemyRepository):"""Repository for saving and retrieving random names"""_names_db=SQLAlchemyDataSource(declarative_meta=Base,db_uri=settings.DATABASE_URL)@propertydef_model_cls(self)->Type[DeclarativeMeta]:returnName@propertydef_dto_cls(self)->Type[BaseModel]:returnNameRecordDTO@propertydef_datasource(self)->SQLAlchemyDataSource:returnself._names_db# The NamesRepository can then be instantiated in the `Actions` subclassesActionsTo create a new action, one needs to subclass theActionclass and override therun()method.For instance:importrandomimportstringfrompydanticimportBaseModelfromlilly.actionsimportActionfrom.repositoriesimportNamesRepository# A Repository for namesfrom.dtoimportNameCreationRequestDTO# The Data Transfer Object to used when creating a nameclassGenerateRandomName(Action):"""Generates a random string and persists it in the data source"""_vowels="aeiou"_consonants="".join(set(string.ascii_lowercase)-set("aeiou"))_name_repository=NamesRepository()def__init__(self,length:int=7):self._length=lengthdefrun(self)->BaseModel:"""Actual method that is run"""name=self._generate_random_word()returnself._name_repository.create_one(NameCreationRequestDTO(title=name))def_generate_random_word(self):"""Generates a random word"""word=""foriinrange(self._length):ifi%2==0:word+=random.choice(self._consonants)else:word+=random.choice(self._vowels)returnword# The GenerateRandomName action is then used in a route as self._do(GenerateRandomName, length=9)To make life easier for the developer, we have developed a few Actions that can be inherited and used easily. They
include:1. CreateOneActionThis is a CRUD action that creates a single item in the repository. Here is a sample of how it is used.fromlilly.actionsimportCreateOneActionfromlilly.repositoriesimportRepository# inherit the CreateOneAction and implement its _repository @property methodclassCreateOneName(CreateOneAction):"""Create a single Name record in the repository"""@propertydef_repository(self)->Repository:return# your repository# then use it in your routes like self._do(CreateOneName, data_dto)2. CreateManyActionThis is a CRUD action that creates multiple items in the repository at one go. Here is a sample of how it is used.fromlilly.actionsimportCreateManyActionfromlilly.repositoriesimportRepository# inherit the CreateManyAction and implement its _repository @property methodclassCreateManyNames(CreateManyAction):@propertydef_repository(self)->Repository:return# your repository# then use it in your routes like self._do(CreateManyNames, data_dtos)3. ReadOneActionThis is a CRUD action that reads a single item from the repository. Here is a sample of how it is used.fromlilly.actionsimportReadOneActionfromlilly.repositoriesimportRepository# inherit the ReadOneAction and implement its _repository @property methodclassReadOneName(ReadOneAction):@propertydef_repository(self)->Repository:return# your repository# then use it in your routes like self._do(ReadOneName, record_id)4. ReadManyActionThis is a CRUD action that reads multiple items in the repository at one go basing on a number of filters and pagination
controls. Here is a sample of how it is used.fromlilly.actionsimportReadManyActionfromlilly.repositoriesimportRepository# inherit the ReadManyAction and implement its _repository @property methodclassReadManyNames(ReadManyAction):@propertydef_repository(self)->Repository:return# your repository# then use it in your routes like self._do(ReadManyNames, "id > 8 AND title LIKE "%doe", skip=1, limit=10, address="Kampala")# To read all names that:#  - have an id greater than 8#  - and title ending with 'doe'#  - as well having the address for that name equal to "Kampala"#  - but skipping the first item in that collection#  - and returning not more than ten records5. UpdateOneActionThis is a CRUD action that updates a single item in the repository. Here is a sample of how it is used.fromlilly.actionsimportUpdateOneActionfromlilly.repositoriesimportRepository# inherit the UpdateOneAction and implement its _repository @property methodclassUpdateOneName(UpdateOneAction):@propertydef_repository(self)->Repository:return# your repository# then use it in your routes like self._do(UpdateOneName, record_id, new_data_dto)6. UpdateManyActionThis is a CRUD action that updates multiple items in the repository at one go basing on a number of filters supplied.
Here is a sample of how it is used.fromlilly.actionsimportUpdateManyActionfromlilly.repositoriesimportRepository# inherit the UpdateManyAction and implement its _repository @property methodclassUpdateManyNames(UpdateManyAction):@propertydef_repository(self)->Repository:return# your repository# then use it in your routes like self._do(UpdateManyNames, new_data_dto, "id > 8 AND title LIKE "%doe", address="Kampala")# To update all names to resemble new_data_dto for all names that:#  - have an id greater than 8#  - and title ending with 'doe'#  - as well having the address for that name equal to "Kampala"7. DeleteOneActionThis is a CRUD action that deletes a single item in the repository. Here is a sample of how it is used.fromlilly.actionsimportDeleteOneActionfromlilly.repositoriesimportRepository# inherit the DeleteOneAction and implement its _repository @property methodclassDeleteOneName(DeleteOneAction):@propertydef_repository(self)->Repository:return# your repository# then use it in your routes like self._do(DeleteOneName, record_id)6. UpdateManyActionThis is a CRUD action that deletes multiple items in the repository at one go basing on a number of filters supplied.
Here is a sample of how it is used.fromlilly.actionsimportDeleteManyActionfromlilly.repositoriesimportRepository# inherit the DeleteManyAction and implement its _repository @property methodclassDeleteManyNames(DeleteManyAction):@propertydef_repository(self)->Repository:return# your repository# then use it in your routes like self._do(DeleteManyNames, "id > 8 AND title LIKE "%doe", address="Kampala")# To delete all names that:#  - have an id greater than 8#  - and title ending with 'doe'#  - as well having the address for that name equal to "Kampala"Route SetsTo create a new route set, one needs to subclass theRouteSetclass and decorate it withroutesetand decorate each
method that is to be an endpoint with appropriate the method (HTTP or websocket) decorators likeget,postetc.For instance:fromlilly.routingimportrouteset,RouteSet,get,postfrom.dtoimportMessageDTO# the Data Transfer Object to pass data around the app@routesetclassNormalRouteSet(RouteSet):"""A basic Class based route that can have any method as an endpoint and can have common variables in the initattached to self"""def__init__(self):self.name="Lilly"@get("/",response_model=MessageDTO)defhome(self):"""Home"""return{"message":f"Welcome to{self.name}"}@get("/login",response_model=MessageDTO)deflogin(self):"""Login"""return{"message":f"{self.name}invites you to login"}To make the life of the developer easier, there are someRouteSetsubclasses that one can inherit from and easily have
a set of endpoints that fulfill a particular purpose.They include:1. CRUDRouteSetFor CRUD (Create-Read-Update-Delete) actions, a RouteSet can be created be by subclassingCRUDRouteSetand overriding theget_settings()class method on it to return the appropriateCRUDRouteSetSettingsfor the given
route set.For example:fromlilly.routingimportrouteset,CRUDRouteSet,CRUDRouteSetSettings,get,postfrom.dtoimport(NameRecordDTO,NameCreationRequestDTO,MessageDTO,RandomNameCreationRequestDTO,)# The Data Transfer Objects to be used as responses or requestsfrom.actionsimport(CreateOneName,CreateManyNames,ReadOneName,ReadManyNames,UpdateOneName,UpdateManyNames,DeleteOneName,DeleteManyNames,GenerateRandomName,)# The Actions for CRUD@routesetclassHelloWorld(CRUDRouteSet):"""Class Based Route set that handles CRUD functionality out of the box"""@classmethoddefget_settings(cls)->CRUDRouteSetSettings:# When an action is not defined, the dependant routes will not be shownreturnCRUDRouteSetSettings(id_type=int,base_path="/names",base_path_for_multiple_items="/admin/names",response_model=NameRecordDTO,creation_request_model=NameCreationRequestDTO,create_one_action=CreateOneName,create_many_action=CreateManyNames,read_one_action=ReadOneName,read_many_action=ReadManyNames,update_one_action=UpdateOneName,update_many_action=UpdateManyNames,delete_one_action=DeleteOneName,delete_many_action=DeleteManyNames,string_searchable_fields=["title"],)# You can add even more routes on the CRUD routeset@get("/hello/{name}",response_model=MessageDTO)defsay_hello(self,name:str):return{"message":f"Hi{name}"}@post("/random-names/",response_model=NameRecordDTO)defcreate_random_name(self,request:RandomNameCreationRequestDTO):returnself._do(GenerateRandomName,length=request.length)DesignRequirementsThe following features are required.ConfigurationAll services are put in theservicesfolder whose import path is passed as a parameter to theLillyinstance
during initialization.
(Default: folder calledserviceson root of project)All settings are put as constants in thesettingspython module whose import path is passed toLillyinstance at
initialization.
(Default:settings.pyon the root of project)Base StructuresAll services must have the following modules or packages:routes(if a package is used, allRouteSetsubclasses must be imported into theroutes.__init__module)actionsrepositoriesdatasourcesdtosJust likeFastAPI Class-based views (CBV)routes, Lilly routes (which are technically methods of the Service subclass) should have thepost,get,put,patch...decorators. The format is exactly as it is in FastAPI. In addition, dependencies can be shared across multiple
endpoints of the same service thanks toFastApi CBV.RouteSetis the base class of all Routes. It should have the following methods overridden:_do(self, actionCls: Type[Action], *args, **kwargs)which internally initializes the actionCls and callsrun()on itActionsubclasses should have an overriddenrun(self) -> AnymethodTherun(self)method should be able to access any repositories by directly importing any it needsRepositorysubclasses should have public:get_one(self, record_id: Any, **kwargs) -> Anymethod to get one record of idrecord_idget_many(self, skip: int, limit: int, filters: Dict[Any, Any], **kwargs) -> List[Any]method to get many records
that fulfil thefilterscreate_one(self, record: BaseModel, **kwargs) -> Anymethod to create one recordcreate_many(self, records: List[BaseModel], **kwargs) -> List[Any]method to create many recordsupdate_one(self, record_id: Any, new_record: Any, **kwargs) -> Anymethod to update one record of idrecord_idupdate_many(self, new_record: BaseModel, filters: Dict[Any, Any], **kwargs) -> Anymethod to update many records
that fulfil thefiltersremove_one(self, record_id: Any, **kwargs) -> Anymethod to remove one record of idrecord_idremove_many(self, filters: Dict[Any, Any], **kwargs) -> Anymethod to remove many records that fulfil
thefiltersRepositorysubclasses should also have the following methods overridden:_get_one(self, datasource_connection: Any, record_id: Any, **kwargs) -> Anymethod to get one record of
idrecord_id_get_many(self, datasource_connection: Any, skip: int, limit: int, filters: Dict[Any, Any], **kwargs) -> List[Any]method to get many records that fulfil thefilters_create_one(self, datasource_connection: Any, record: BaseModel, **kwargs) -> Anymethod to create one record_create_many(self, datasource_connection: Any, record: List[BaseModel], **kwargs) -> List[Any]method to create
many records_update_one(self, datasource_connection: Any, record_id: Any, new_record: BaseModel, **kwargs) -> Anymethod to
update one record of idrecord_id_update_many(self, datasource_connection: Any, new_record: BaseModel, filters: Dict[Any, Any], **kwargs) -> Anymethod to update many records that fulfil thefilters_remove_one(self, datasource_connection: Any, record_id: Any, **kwargs) -> Anymethod to remove one record of
idrecord_id_remove_many(self, datasource_connection: Any, filters: Dict[Any, Any], **kwargs) -> Anymethod to remove many
records that fulfil thefilters_datasource(self) -> DataSourcean @property-decorated method to return the DataSource whoseconnect()method is
to be called in any of the other methods to get its instance._to_output_dto(self, record: Any) -> BaseModelmethod which converts any record from the data source raw to DTO
for the public methodsDataSourcesubclasses should have an overriddenconnect(self)methoddtos(Data Transfer Object classes) are subclasses of thepydantic.BaseModelwhich are to be used to move data
across the layersAny setting added to the gazetted settings file can be accessed vialilly.conf.settings.<setting_name>e.g.lilly.conf.settings.APP_SETTINGRunningTheLillyinstance should be run the same way as FastAPI instances are run e.g.uvicornmain:app# for app defined in the main.py moduleImplementation IdeasThe application is an instance of theLillyclass which is a subclass of theFastAPIclass.To create aLillyinstance, we need to pass in the following parameters:services_path (an import path as string, default is "services")settings_path (an import path as string, default is "settings")DuringLillyinitialization, all routes are automatically imported usingimportlib.import_moduleby concatenating
the<services_path>.<service_name>.routese.g.services.hello.routes.In order to make route definition solely dependent on folder structure, we change@app.getdecorators to@getapp.get,app.postetc. should throwNotImplementedErrorerrorsThe whole app has one instance of therouter: APIRouter. It is defined in theroutingmodule.In that sameroutingmodule,router.get,router.post,router.delete,router.put,router.patch,router.head,router.optionsare all aliased by their post-periodsuffixese.g.get,postetc.When initializing ininitof Lilly, we fetch the routes in all services then callself.include_router(router).app.mountshould throw anNotImplementedErrorerror because it complicates the app structure if used to mount
other applications, considering the fact that all routes share onerouterinstance.In order to have a protected method_do()to call an action within the routers, we use class-based views
fromfastapi-utils CBV.All these class based views will be subclasses ofRouteSetwhich has an overridable protected
method_do(self, action_cls: Action, *args, **kwargs)to make a call to any actionAll these class based views will have a decorator@routesetwhich is an alias of@cbv(router)whererouteris
the router common to all routesAll the routes in the app have one router so their endpoints need to be different and explicit since no mounting will
be allowedTheconnect()method of theDataSourceclass should return aContextManagerwrapped around the connection itself
so as to allow for any clean up tasks to be done in the__exit__()method of that ContextManager after each
connection is ready to be dropped. The__enter__method of the ContextManager needs to return the actual connection
object.ToDoSet up the abstract methods structureSet up the CLI to generate an appSet up the CLI to generate a serviceMake repository publicPackage it and publish itAdd some out-of-the-box base data sources e.g.SqlAlchemyRedisMemcachedRESTAPIGraphQLRabbitMQActiveMQWebsocketsKafkaMongodbCouchbaseDiskCacheAdd some out-of-the-box base repositories e.g.SqlAlchemyRepository (RDBM e.g. PostgreSQL, MySQL etc.)SQLAlchemyRepository hangs when postgres is used (try running tests)RedisRepositoryMemcachedRepositoryRESTAPIRepositoryGraphQLRepositoryRabbitMQRepositoryActiveMQRepositoryWebsocketsRepositoryKafkaRepositoryMongodbRepositoryCouchbaseRepositoryDiskCacheRepositoryAdd some out-of-the-box base actions e.g.CreateOneActionCreateManyActionUpdateOneActionUpdateManyActionReadOneActionReadManyActionDeleteOneActionDeleteManyActionAdd some out-of-the-box base route setsCRUDRouteSetWebsocketRouteSetGraphQLRouteAdd example code in examples folderTodolist (CRUDRouteSet, SqlAlchemyRepo)RandomQuotes (WebsocketRouteSet, MongodbRepo) (quotes got from the Bible)Clock (WebsocketRouteSet, WebsocketsRepo)Set up automatic documentationSet up CI via Github actionsSet up CD via Github actionsWrite about it in hashnode or Medium or bothInspirationThe idea to createlillycame after looking at what theLoopbackteam did
withLoopback4ChangeLogFor the changes across versions, look at theCHANGELOG.mdLicenseCopyright (c) 2022Martin AhinduraLicensed under theMIT License