Package: pysolaar
PySolaarA highly opinionated, Django-like wrapper round PySolr, for when you want to ridiculously de-normalise some complex data at runtime, and then query it with a pretty interface.FeaturesManaging your Solr dataCreate document types with a Django-like class-based approachDefine how a document instance should get its data. It's just a function you define, so how you get the data is up to you: from a database, by mushing together data from different Django classes, by HTTP request...Define a function for PySolaar to call in order to generate the documents...Declaratively define the structure of your documents, if you want to do any complex embedding or reuse documents.Press go!PySolaar automatically encodes, embeds, etc. all the data and pushes it to Solr.Fields are "namespaced" to document-type (like Python's__name_mangling), so no clashes! Theidfield's value is also prefixed with the class name, so it's unique for a specific class.Querying your Solr dataA nice Django QuerySet-like approach (wraps a modified version ofSolrQ, which you can also use for more complex queries)Automatically prefixes all the queries to deal with the name-mangling.Declaratively define the document you want returned, including a set of transformations for unpacking data, turning dates back into Pythondatetimeobjects, or whatever you want.Lazy query evaluationBasicsCreating PySolaar DocumentsPySolaar allows documents to be defined using Django-like classes, which represent entity types:Create an entity type, by subclassingPySolaar.Define abuild_documentmethod:This is "how you get the data to index".This method defines a single document or set of documents that correspond to a single identifier.It should take an identifier as an argument, and return aself.Documentobject or an iterable ofself.Documentobjects.Pass your data as key words toself.Documentinitialiser or unpack a dict.Define abuild_document_setmethod that iterates through a series of identifiers and returns a call toself.build_documentfor each identifier.frompysolaarimportPySolaar# Create a thing that inherits from PySolaarclassPerson(PySolaar):# Write a `build_document` method -- this gets the data corresponding# to a particular value of `identifier`defbuild_document(self,identifier):# Return an instance of self.Document containing the datareturnself.Document(id="person-{}".format(identifier),name="Claudius the{}".format(identifier),height=100*identifier,moustache=["No","Yes"][bool(identifier%2==0)]# It's odd not to have a moustache!)# Write `build_document_set` that produces an iterator of Person.build_document callsdefbuild_document_set(self):foridentifierin[1,2,3,4,5]:yieldself.build_document(identifier)# Configure PySolaar by setting up the underlying PySolr instancePySolaar.configure_pysolr("http://your-solr-instance")# Then run PySolaar.update() to push data to SolrPySolaar.update()WhenPySolaar.update()is called, PySolaar goes through all its subclasses'build_document_setfunctions, in order to generate the documents, and then pushes them to Solr. Obviously, there's a reasonable amount of magic.Querying the dataPySolaar provides a Django-like interface for querying data. Or just pass aSolrQobject to.filter()fromPySolaarimportQfrom__above__importPerson# Get all the personspersons=Person.all()# Filter by anything ...claudiuses=Person.filter(name="Claudius")# ... and chain QuerySets as in Djangotall_with_moustache=claudiuses.filter(height__gt=250,moustache="Yes")# ... or use a Q objecteither_tall_or_moustache=claudiuses.filter(Q(height__gt=250)|Q(moustache="Yes"))# ... and paginatefirst_page=tall_with_moustache.paginate(page_size=2,page_number=0)# Results aren't evaluated until you need them:tall_with_moustache.count()# -> 2forcinfirst_page:print(c["id"])# And a few other features — see the Advanced section.Restrictions when defining documentsMost of the restrictions here stem from the limitations of Solr and the PySolr library.A single field can be contain:A value (string, int, datetime, etc.)Alistof values (probably asetas well)Adict, whose values are either more dicts or values or lists of values. (Dicts are collapsed down to single value fields using double underscores, i.e.field={"one": "something", "two": "something else"}becomesfield__one="something"andfield__two="something else"— to an arbitrary depth!)NOTa list of dicts. To index a list of associated values (e.g. lists of dicts), instead use Child Documents.Advanced featuresTheMetaclass (as borrowed from Django)Each class can define aMetaclass, which can be used to declaratively define a number of aspects regarding how data is stored in Solr. (The Meta class is passed around in the background to apply settings where appropriate.)Using theMetaclass, you can:Independently from the data-definition method(build_document), declare astore_document_fieldsstructure, defining which fields should be pushed to Solr and in which format. This allows thebuild_documentmethod to be a 'generic' method for getting whatever data is required and allows easy embedding and reuse.Define areturn_document_fieldsstructure to limit the fields that are returned from Solr (so you can have fields that are just there for querying, but you're not interested in)Or (older version, probably will be deprecated), independently define lists of fields such asfields_as_child_docs.Child documentsThe Solr "child field" feature is used to allow one document type to be nested inside another.To associate a nested document with a particular field, first define afields_as_child_docslist in theMetaclass and add the field name. Then set the value of the parent field toChild.items([identifiers]). See the example below.frompysolaarimportPySolaarclassPerson(PySolaar):classMeta:# Define a Meta class with `fields_as_child_docs`# in order to declare a field is a child docfields_as_child_docs=["pets"]defbuild_document(self,identifier):returnself.Document(id="thing-{}".format(identifier),name="Claudius",# Embed another document type by calling Class.items([identifiers])pets=Pets.items([1,2,3]))classPets(PySolaar):defbuild_document(self,identifier):returnself.Document(id="pet-{}".format(identifier),name="Gordon")In the background,PySolaar.itemscallsbuild_documentwith the listed identifiers, and embeds the document as a child (using PySolr's_dockeyword).Nesting searchable documents works up to one level of embedding. After this, documents can be stored as JSON strings and recovered automatically, but not queried. After three levels of embedding a particular type, this embedding will stop (preventing infinite recursion).store_document_fieldsIn general, thebuild_documentfunction should defineallthe fields required for every use-case. The results of calling this function with a particular identifier are cached, so the data can be re-used if embedded as a child document elsewhere (n.b. the cache is cleared after an update!)DocumentFieldsclasses provides a convenient way to describe how the document should be stored in Solr.Set thestore_document_fieldsvalue in your classMetato aDocumentFieldsinstance where you list the fields you want to include (set them toTrue).Also use this structure to declare child fields usingChildDocument(in place of using thefields_as_child_docs, as above)andto control which fields in theChildDocumentare included.This is useful (as in the example below) as we can havePersondocuments with their pets' names embedded as searchable child documents, but also have more detailedPetsdocuments (in turn with  certainownerfields).importdatetimefrompysolaarimportPySolaar,DocumentFields,ChildDocumentclassPerson(PySolaar):classMeta:store_document_fields=(DocumentFields(name=True,school=True,work=True,has_pets=ChildDocument(# Here we can start selecting fields from the Pets classname=True,weight=True,# date_of_birth=True ... we don't care about, so omit itowner=ChildDocument(name=True,# Here, we embed a Person instance again, this time just selecting the name# n.b. this `owner` field will be converted to JSON for storage (see above)),),),)defbuild_document(self,identifier):returnself.Document(id="thing-{}".format(identifier),name="Claudius",school="St Somethings",work="Bus driver",has_pets=Pets.items([1,2,3]))classPets(PySolaar):defbuild_document(self,identifier):returnself.Document(id="pet-{}".format(identifier),name="Gordon"weight=123,date_of_birth=datetime.datetime(1996,1,3),# Re-embed the Person as the owner, as it might be useful# if we ever want a 'Pets' top-level document.owner=Person.items(f"owner_of_{identifier}"))store_document_fieldallows the following structures:DocumentFields: this is the root wrapper for the whole documentChildDocument: embeds another document (with the specified fields) as a Solr child documentJsonChildDocument: embeds a child document by converting it to a JSON string. It can be returned and unpacked back to Python, but not queried in Solr (except as a hit-and-miss string-matching exercise...)SplattedChildDocument: embeds a child document as a list of searchable fields (it works recursively through the child document as a dict, accumulating all the values in a list). Useful for creating a searchable version of a child document, where you don't care about any field in particular, just matching something. Probably don't return this from Solr — unlikeJsonChildDocumentit cannot be reverted back to anything particularly useful.