Package: pyesorm
ESORM - Python ElasticSearch ORM based on PydanticSome ideas come fromPydasticlibrary, which is similar,
but not as advanced (yet).☰ Table of Contents💾 Installation🚀 FeaturesSupported ElasticSearch versionsSupported Python versions📖 UsageDefine a modelPython basic typesESORM field typesNested documentsId fieldModel SettingsDescribe fieldsESModelTimestampConnecting to ElasticSearchClientCreate index templatesCreate indices and mappingsCRUD: CreateCRUD: ReadCRUD: UpdateCRUD: DeleteBulk operationsSearchGeneral searchSearch with field value terms (dictioanry search)AggregationsPagination and sorting🔬 Advanced usageLazy propertiesShard routingWatchersFastAPI integration🧪 Testing🛡 License📃 Citation💾 Installationpipinstallpyesorm🚀 FeaturesPydantic model representation of ElasticSearch documentsAutomatic mapping and index creationCRUD operationsFull async support (no sync version at all)Mapping to and from ElasticSearch typesSupport for nested documentsCustom id fieldContext for bulk operationsSupported IDE autocompletion and type checking (PyCharm tested)Everything in the source code is documented and annotatedTypedDicts for ElasticSearch queries and aggregationsDocstring support for fieldsShard routing supportLazy propertiesSupport >= Python 3.8 (tested with 3.8 through 3.12)Support for ElasticSearch 8.x and 7.xWatcher support (You may need ElasticSearch subscription license for this)Pagination and sortingFastAPI integrationNot all ElasticSearch features are supported yet, pull requests are welcome.Supported ElasticSearch versionsIt is tested with ElasticSearch 7.x and 8.x.Supported Python versionsTested with Python 3.8 through 3.12.📖 UsageDefine a modelYou can use allPydanticmodel features, becauseESModelis a subclass ofpydantic.BaseModel.Python basic typesfromesormimportESModelclassUser(ESModel):name:strage:intThis is how the python types are converted to ES types:Python typeES typestrtextintlongfloatdoubleboolbooleandatetime.datetimedatedatetime.datedatedatetime.timedateESORM field typesYou can specify ElasticSearch special fields usingesorm.fieldsmodule.fromesormimportESModelfromesorm.fieldsimportkeyword,text,byte,geo_pointclassUser(ESModel):name:textemail:keywordage:bytelocation:geo_point...The supported fields are:Field nameES typekeywordkeywordtexttextbinarybinarybytebyteshortshortintegerorint32integerlongorint64longfloat16orhalf_floathalf_floatfloat32floatdoubledoublebooleanbooleangeo_pointgeo_pointNested documentsfromesormimportESModelfromesorm.fieldsimportkeyword,text,byteclassUser(ESModel):name:textemail:keywordage:byte=18classPost(ESModel):title:textcontent:textwriter:User# User is a nested documentId fieldYou can specify id field inmodel settings:fromesormimportESModelfromesorm.fieldsimportkeyword,text,byteclassUser(ESModel):classESConfig:id_field='email'name:textemail:keywordage:byte=18This way the field specified inid_fieldwill be removed from the document and used as the document_idin the
index.If you specify a field namedidin your model, it will be used as the document_idin the index
(it will automatically override theid_fieldsetting):fromesormimportESModelclassUser(ESModel):id:int# This will be used as the document _id in the indexname:strYou can also create an__id__property in your model to return a custom id:fromesormimportESModelfromesorm.fieldsimportkeyword,text,byteclassUser(ESModel):name:textemail:keywordage:byte=18@propertydef__id__(self)->str:returnself.emailNOTE: annotation of__id__method is important, and it must be declared as a property.Model SettingsYou can specify model settings usingESConfigchild class.fromtypingimportOptional,List,Dict,AnyfromesormimportESModelclassUser(ESModel):classESConfig:""" ESModel Config """# The index nameindex:Optional[str]=None# The name of the 'id' fieldid_field:Optional[str]=None# Default sortdefault_sort:Optional[List[Dict[str,Dict[str,str]]]]=None# ElasticSearch index settings (https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html)settings:Optional[Dict[str,Any]]=None# Maximum recursion depth of lazy propertieslazy_property_max_recursion_depth:int=1ESModelTimestampYou can useESModelTimestampclass to addcreated_atandupdated_atfields to your model:fromesormimportESModelTimestampclassUser(ESModelTimestamp):name:strage:intThese fields will be automatically updated to the actualdatetimewhen you create or update a document.
Thecreated_atfield will be set only when you create a document. Theupdated_atfield will be set
when you create or update a document.Describe fieldsYou can use the usualPydanticfield description, but you can also use docstrings like this:fromesormimportESModelfromesorm.fieldsimportTextFieldclassUser(ESModel):name:str='John Doe'""" The name of the user """age:int=18""" The age of the user """# This is the usual Pydantic way, but I think docstrings are more intuitive and readableaddress:str=TextField(description="The address of the user")The documentation is usseful if you create an API and you want to generate documentation from the model.
It can be used inFastAPIfor example.Connecting to ElasticSearchYou can connect with a simple connection string:fromesormimportconnectasyncdefes_init():awaitconnect('localhost:9200')Also you can connect to multiple hosts if you have a cluster:fromesormimportconnectasyncdefes_init():awaitconnect(['localhost:9200','localhost:9201'])You can wait for node or cluster to be ready (recommended):fromesormimportconnectasyncdefes_init():awaitconnect('localhost:9200',wait=True)This will ping the node in 2 seconds intervals until it is ready. It can be a long time.You can pass any arguments thatAsyncElasticsearchsupports:fromesormimportconnectasyncdefes_init():awaitconnect('localhost:9200',wait=True,sniff_on_start=True,sniff_on_connection_fail=True)ClientTheconnectfunction is a wrapper for theAsyncElasticsearchconstructor. It creates and stores
a global instance of a proxy to anAsyncElasticsearchinstance. The model operations will use this
instance to communicate with ElasticSearch. You can retrieve the proxy client instance and you can
use the same way asAsyncElasticsearchinstance:fromesormimportesasyncdefes_init():awaites.ping()Create index templatesYou can create index templates easily:fromesormimportmodelasesorm_model# Create index templateasyncdefprepare_es():awaitesorm_model.create_index_template('default_template',prefix_name='esorm_',shards=3,auto_expand_replicas='1-5')Here this will be applied allesorm_prefixed (default) indices.All indices created by ESORM have a prefix, which you can modify globally if you want:fromesorm.modelimportset_default_index_prefixset_default_index_prefix('custom_prefix_')The default prefix isesorm_.Create indices and mappingsYou can create indices and mappings automatically from your models:fromesormimportsetup_mappings# Create indices and mappingsasyncdefprepare_es():importmodels# Import your models# Here models argument is not needed, but you can pass it to prevent unused import warningawaitsetup_mappings(models)First you must create (import) all model classes. Model classes will be registered into a global registry.
Then you can callsetup_mappingsfunction to create indices and mappings for all registered models.IMPORTANT:This method will ignore mapping errors if you already have an index with the same name. It can update the
indices
by new fields, but cannot modify or delete fields! For that you need to reindex your ES database. It is an ElasticSearch
limitation.CRUD: CreatefromesormimportESModel# Here the model have automatically generated idclassUser(ESModel):name:strage:intasyncdefcreate_user():# Create a new useruser=User(name='John Doe',age=25)# Save the user to ElasticSearchnew_user_id=awaituser.save()print(new_user_id)CRUD: ReadfromesormimportESModel# Here the model have automatically generated idclassUser(ESModel):name:strage:intasyncdefget_user(user_id:str):user=awaitUser.get(user_id)print(user.name)CRUD: UpdatefromesormimportESModel# Here the model have automatically generated idclassUser(ESModel):name:strage:intasyncdefupdate_user(user_id:str):user=awaitUser.get(user_id)user.name='Jane Doe'awaituser.save()CRUD: DeletefromesormimportESModel# Here the model have automatically generated idclassUser(ESModel):name:strage:intasyncdefdelete_user(user_id:str):user=awaitUser.get(user_id)awaituser.delete()Bulk operationsBulk operations could be much faster than single operations, if you have lot of documents to
create, update or delete.You can use context for bulk operations:fromtypingimportListfromesormimportESModel,ESBulk# Here the model have automatically generated idclassUser(ESModel):name:strage:intasyncdefbulk_create_users():asyncwithESBulk()asbulk:# Creating or modifiying modelsforiinrange(10):user=User(name=f'User{i}',age=i)awaitbulk.save(user)asyncdefbulk_delete_users(users:List[User]):asyncwithESBulk(wait_for=True)asbulk:# Here we wait for the bulk operation to finish# Deleting modelsforuserinusers:awaitbulk.delete(user)Thewait_forargument is optional. If it isTrue, the context will wait for the bulk operation to finish.SearchGeneral searchYou can search for documents usingsearchmethod, where an ES query can be specified as a dictionary.
You can useres_dict=Trueargument to get the result as a dictionary instead of a list. The key will be theidof the document:await User.search(query, res_dict=True).If you only need one result, you can usesearch_onemethod.fromesormimportESModel# Here the model have automatically generated idclassUser(ESModel):name:strage:intasyncdefsearch_users():# Search for users at least 18 years oldusers=awaitUser.search(query={'bool':{'must':[{'range':{'age':{'gte':18}}}]}})foruserinusers:print(user.name)asyncdefsearch_one_user():# Search a user named John Doeuser=awaitUser.search_one(query={'bool':{'must':[{'match':{'name':{'query':'John Doe'}}}]}})print(user.name)Queries are type checked, because they are annotated asTypedDicts. You can use IDE autocompletion and type checking.Search with field value terms (dictionary search)You can search for documents usingsearch_by_fieldsmethod, where you can specify a field and a value.
It also has ares_dictargument andsearch_one_by_fieldsvariant.fromesormimportESModel# Here the model have automatically generated idclassUser(ESModel):name:strage:intasyncdefsearch_users():# Search users age is 18users=awaitUser.search_by_fields({'age':18})foruserinusers:print(user.name)AggregationsYou can useaggregatemethod to get aggregations.
You can specify an ES aggregation query as a dictionary. It also accepts normal ES queries,
to be able to fiter which documents you want to aggregate.
Both the aggs parameter and the query parameter are type checked, because they are annotated asTypedDicts.
You can use IDE autocompletion and type checking.fromesormimportESModel# Here the model have automatically generated idclassUser(ESModel):name:strage:intcountry:strasyncdefaggregate_avg():# Get average age of usersaggs_def={'avg_age':{'avg':{'field':'age'}}}aggs=awaitUser.aggregate(aggs_def)print(aggs['avg_age']['value'])asyncdefaggregate_avg_by_country(country='Hungary'):# Get average age of users by countryaggs_def={'avg_age':{'avg':{'field':'age'}}}query={'bool':{'must':[{'match':{'country':{'query':country}}}]}}aggs=awaitUser.aggregate(aggs_def,query)print(aggs['avg_age']['value'])asyncdefaggregate_terms():# Get number of users by countryaggs_def={'countries':{'terms':{'field':'country'}}}aggs=awaitUser.aggregate(aggs_def)forbucketinaggs['countries']['buckets']:print(bucket['key'],bucket['doc_count'])Pagination and sortingYou can usePaginationandSortclasses to decorate your models. They simply wrap your models
and add pagination and sorting functionality to them.Paginationfromesorm.modelimportESModel,PaginationclassUser(ESModel):id:int# This will be used as the document _id in the indexname:strage:intdefget_users(page=1,page_size=10):# 1st create the decorator itselfpagination=Pagination(page=page,page_size=page_size)# Then decorate your modelres=pagination(User).search_by_fields(age=18)# Here the result has maximum 10 itemsreturnresSortingIt is similar to pagination:fromesorm.modelimportESModel,SortclassUser(ESModel):id:int# This will be used as the document _id in the indexname:strage:intdefget_users():# 1st create the decorator itselfsort=Sort(sort=[{'age':{'order':'desc'}},{'name':{'order':'asc'}}])# Then decorate your modelres=sort(User).search_by_fields(age=18)# Here the result is sorted by age ascendingreturnresdefget_user_sorted_by_name():# You can also use this simplified syntaxsort=Sort(sort='name')# Then decorate your modelres=sort(User).all()# Here the result is sorted by age descendingreturnres🧪 TestingFor testing you can use thetest.shin the root directory. It is a script to running
tests on multiple python interpreters in virtual environments. At the top of the file you can specify
which python interpreters you want to test. The ES versions are specified intests/docker-compose.ymlfile.If you already have a virtual environment, simply usepytestto run the tests.🛡 LicenseThis project is licensed under the terms of theMozilla Public License 2.0(
MPL 2.0) license.📃 CitationIf you use this project in your research, please cite it using the following BibTeX entry:@misc{esorm,author={Adam Wallner},title={ESORM: ElasticSearch Object Relational Mapper},year={2023},publisher={GitHub},journal={GitHub repository},howpublished={\url{https://github.com/wallneradam/esorm}},}