Package: delune
Table of ContentsIntroduceInstallationQuick StartConfigure DeLuneIndexing and Searching On Local MachineRun Indexer as ServiceUsing Client APIConnecting to Delune ResourcesCreating New CollectionConfiguring CollectionAdding Dcouments To CollectionAdding Dcouments Without IDDeleting Dcouments From CollectionIndexingSearchingTruncating DocumentsDrop CollectionIndexing and Searching On Remote MachineRunning RESTful APIRun Indexer as ServiceUsing Client APIReplicating Delune ResourcesReplicating Your CollectionLimitationLow Level APILoggerStandard AnalyzerImplemented StemmersBi-Gram IndexTri-Gram IndexCollectionIndexerSet Returning ContentSnippet Generating FieldSearchable FieldsSearcherQuery SyntaxMigrating From Version 0.3xLinksChange LogEarlier Wissen PeriodIntroduceDeLune (former Wissen) is a simple fulltext search engine and Python object (similar with noSQL document concept) storage written in Python for logic thing and C for a core index/search module.I had been studedLuceneearlier version withLupyandCLucene. And I had maden my own search engine for excercise.Its file format, numeric compressing algorithm, indexing process are quiet similar with Lucene earlier version (I don’t know about recent versions at all). But querying and result-fetching parts is built from my imagination. As a result it’s entirely unorthodox and possibly inefficient (I am a typical nerd and work-alone programmer ;-)DeLune is a kind of hybrid of search engine and noSQL document database.DeLune stores python objects with pickle-compresses serializing, then if you use DeLune as python module, you can store and get document derectly.DeLune may be useful when it is allowed a few minutes gap on updating, inserting and deleting requests and operations. For example, it will be good for your legacy contents or generated by your own not by customer.As most fulltext search engines, DeLune do always and only append data, no modification for existing files. So inserting, updating and deleting ops need high disk writing cost. Sometimes one small deletion op may trigger massive disk writing for optimization (even deleting cost itself is very low).Anyway, if you need realtime changes on your data, DO BOT USE DeLune or complement with another type of NoSQL or RDBMS.DeLune supports storing multiple documents for polymorphic use cases like listing and detail views. It is inefficient for storage usage, but helps reading performance.DeLune’s searching mechanism is similar with DNA-RNA-Protein working model can be translated into ‘Index File-Temporary Small Replication Buffer-Query Result’.Every searcher (Cell) has a single index file handlers group (DNA group in nuclear)Thread has multiple small memory buffer (RNA) for replicating index as needed partQuery class (Ribosome) creates query result (Protein) by synthesising buffers’ inforamtion (RNAs) and Each thread has own memory space to create result set (Protein) not shared with other threads.Repeat from 2nd if expected more resultsAnd it provides storing, indexing and searching RESTful API throughSkitai App Engine,Work on multi processes environmentMaster-slave replicationUntested (not yet) sharding, map-reducing, load-balancing using Skitai featuresInstallationDeLune contains C extension, so need C compiler.pipinstalldeluneOn posix, it might be required some packages,apt-getinstallbuild-essentialzlib1g-devQuick StartAll field text type should be str type, otherwise encoding should be specified.Here’s an example indexing only one document.importdelunedln=delune.connect("/home/deune")col=dln.create("mycol",["mycol"],1)withcol.documentsasD:song="violin sonata in c k.301"birth=1756d=D.new(100)# document IDd.content([song,{'composer':'mozart','birth':birth}])d.field("default",song,delune.TEXT)d.field("birth",birth,delune.INT16)d.snippet(song)D.add(d)D.commit()D.index()result=D.query("violin")Result will be like this:{'code':200,'time':0,'total':1'result':[[['violin sonata in c k.301',{"composer":'wofgang amadeus mozart','birth':1756}],# content'<b>violin</b> sonata in c k.301',# auto snippet14,0,0,0# additional info]],'sorted':[None,0],'regex':'violin|violins',}DeLune’s document can be any Python objects picklalbe, delune stored document zipped pickled format. But you want to fetch partial documents by key or index, document skeleton shoud be a list or dictionary, but still inner data type can be any picklable objects. I think if your data need much more reading operations than writngs/updatings, DeLune can be as both simple schemaless data storage and fulltext search engine. DeLune’s RESTful API and replication is end of this document.Configure DeLuneWhen indexing it’s not necessory to configure, but searching should be configured. The reason why DeLune allocates memory per thread for searching and classifying on initializing.delune.configure(numthread,logger,io_buf_size=4096,mem_limit=256)numthread: number of threads which access to DeLune collections and models. if set to 8, you can open multiple collections (or models) and access with 8 threads. If 9th thread try to access to delune, it will raise errorlogger:see next chapterio_buf_size = 4096: Bytes size of flash buffer for repliacting index filesmem_limit = 256: Memory limit per a thread, but it’s not absolute. It can be over during calculation if need, but when calcuation has been finished, would return memory ASAP.Finally when your app is terminated, call shutdown.delune.shutdown()Indexing and Searching On Local MachineAlthough quick start, we user indexer.index method for indxing documents, delune provide indexer as backend service.Run Indexer as Service# one timne indexing in consoledeluneindex-v/home/delune# indexing every 5minutes in consoledeluneindex-v/home/delune-i300# indexing every 5 minutes as daemondeluneindex-dv/home/delune-i300# restart indexing daemon every 5 minutes as daemondeluneindex-v/home/delune-i300restart# stop indexing daemondeluneindexstop# status of indexing daemondeluneindexstatusUsing Client APIConnecting to Delune Resourcesimportdelunedln=delune.connect("/home/delune")As result, delune check anf create directories./home/delune/delune/config/home/delune/delune/collectionsCreating New Collectioncol=dln.create("mycol",["mycol"],1)col.save()As result, collection created like this./home/delune/delune/config/mycol:JSONfilecontainsconfigureoptions/home/delune/delune/collections/mycolIf you use multiple disks for increasing speed or capacity of collection.First of all mount your disks to /home/delune/delune/collections,/home/delune/delune/collections/hdd0/home/delune/delune/collections/hdd1Then create collection.col=dln.create("mycol",["hdd0/mycol","hdd1/mycol"],1)col.save()As a result, collection will be created like this./home/delune/delune/collections/hdd0/mycol/home/delune/delune/collections/hdd1/mycolYour segment filess of collection will be created these directories randomly (with considering free space of disks).Configuring CollectionThere’re 2 way for configuring tour collections.First, use col.config dictionalry.col=dln.create("mycol",["mycol"],version=1)col.config>>{'name':'mycol','data_dir':["mycol"],"version":1,'analyzer':{"max_terms":3000,"stem_level":1,"strip_html":0,"make_lower_case":1,"ngram":1,"biword":0,"stopwords_case_sensitive":1,"ngram_no_space":0,"contains_alpha_only":0,"stopwords":[],"endwords":[],},'indexer':{'optimize':1,'force_merge':0,'max_memory':10000000,'max_segments':10,'lazy_merge':(0.3,0.5),},'searcher':{'max_result':2000,'num_query_cache':1000}}You just change values as you want.Another way is set options when creating collection.col=dln.create("mycol",["mycol"],version=1,max_terms=5000,strip_html=1,force_merge=1,max_result=10000)For more detail for analyzer, indexer and searcher options, seeLow Level APIsection.Adding Dcouments To Collectionwithcol.documentsasD:forcode,titleinmy_codes:d=D.new(code)# code is used as document IDd.content([code,title])d.field("code",code,delune.STRING)d.field("default",title,delune.TEXT)D.add(d)D.commit()It is important to understand, above operation actually dosen’t make any change to your collection. It just saves your documents at:/home/delune/delune/collections/mycol/.que/If you commit multiple time, que files will be created as you commit.Adding Dcouments Without IDd=D.new()Note that in this case you canmoy update/modify your documents.Deleting Dcouments From CollectionIf your document has ID,withcol.documentsasD:forcode,titleinmy_codes:D.delete(code)D.commit()Else,withcol.documentsasD:D.qdelete("milk")D.commit()It will be deleted all documents contain ‘milk’.IndexingIf you run delune indexer, these saved documents will be automatically indexed. Or you can index mannually,deluneindex-v/home/deluneSearchingdln=delune.connect("/home/delune")col=dln.load("mycol")withcol.documentsasD:D.search("violin")search() spec is:D.search(q,offset=0,limit=10,sort="",# INT field namesnippet=30,# number of terms for snippetpartial="",# specify index or key of a contentnthdoc=0,# specify index of contentslang="un",analyze=1,# query terms are already analyzed, set to 0data=1,# whether or not return content partqlimit=1# whether or not apply limitation for number of searched documents by max_result)Truncating Documentscol.documents.truncate("mycol")col.documents.commit()Drop Collectioncol.drop(include_data=True)Indexing and Searching On Remote MachineYou can make remote delune resource.Running RESTful APINew in version 0.12.14You can use RESTful API withSkitai App Enginefor your remote machine.First of all, you need to install skitai by,pip3install-UskitaiThen copy and save below code to app.py.importosimportdeluneimportskitaiif__name__=="__main__":pref=skitai.pref()pref.use_reloader=1pref.debug=1config=pref.configconfig.resource_dir="/home/delune"skitai.trackers('delune:collection')skitai.mount("/",delune,"app",pref)skitai.run(workers=2,threads=4,port=5000)And run,app.pySo you can access tohttp://<your IP address>:5000/v1For more detail about API, seeapp.py.Run Indexer as ServiceAnd like local, you shoud run indexer,deluneindex-dv/home/delune-i300This will index committed documents every 5 minutes.Using Client APIIt is exactly same as local API except connect parameter. parameter should starts with “http://” or “https://” and ends with version string like “v1”dln=delune.connect("http://192.168.0.200:5000/v1")col=dln.create("mycol",["mycol"],1)col.save()...Note that you need not reun indexer background at your local machine any more.Replicating Delune ResourcesYou can run replica server for distributed search or backup.Replicating Your Collection# replicate every 5 minutes from http://192.168.0.200/v1delunereplicate-ohttp://192.168.0.200/v1-i300As a result, all remote delune resources will be replicated with exactly same directory structure.LimitationBefore you test DeLune, you should know some limitation.DeLune search cannot sort by string type field, but can by int/bit/coord types and TFIDF ranking.Low Level APILoggerfromdelune.libimportloggerlogger.screen_logger()# it will create file '/var/log.delune.log', and rotated by daily baselogger.rotate_logger("/var/log","delune","daily")Standard AnalyzerAnalyzer is needed by TEXT, TERM types.Basic Usage is:analyzer=delune.standard_analyzer(max_term=8,numthread=1,ngram=TrueorFalse,stem_level=0,1or2(2isonlyappliedtoEnglishLanguage),make_lower_case=TrueorFalse,stopwords_case_sensitive=TrueorFalse,ngram_no_space=TrueorFalse,strip_html=TrueorFalse,contains_alpha_only=TrueorFalse,stopwords=[word,...])stem_level: 0 and 1, especially ‘en’ language has level 2 for hard stemmingmake_lower_case: make lower case for every textstopwords_case_sensitive: it will work if make_lower_case is Falsengram_no_space: if False, ‘泣斬 馬謖’ will be tokenized to _泣, 泣斬, 斬_, _馬, 馬謖, 謖_. But if True, addtional bi-gram 斬馬 will be created between 斬_ and _馬.strip_htmlcontains_alpha_only: remove term which doesn’t contain alphabet, this option is useful for full-text training in some casesstopwords: DeLune has only English stopwords list, You can use change custom stopwords. Stopwords sould be unicode or utf8 encoded bytesDeLune has some kind of stemmers and n-gram methods for international languages and can use them by this way:analyzer=standard_analyzer(ngram=True,stem_level=1)col=delune.collection("./col",delune.CREATE,analyzer)indexer=col.get_indexer()document.field("default",song,delune.TEXT,lang="en")Implemented StemmersExcept English stemmer, all stemmers can be obtained atIR Multilingual Resources at UniNE.ar: Arabicde: Germanen: Englishes: Spanishfi: Finnishfr: Frenchhu: Hungarianit: Italianpt: Portuguesesv: SwedishBi-Gram IndexIf ngram is set to True, these languages will be indexed with bi-gram.cn: Chineseja: Japaneseko: KoreanAlso note that if word contains only alphabet, will be used English stemmer.Tri-Gram IndexThe other languages will be used English stemmer if all spell is Alphabet. And if ngram is set to True, will be indexed with tri-gram if word has multibytes.Methods Specanalyzer.index (document, lang)analyzer.freq (document, lang)analyzer.stem (document, lang)analyzer.count_stopwords (document, lang)CollectionCollection manages index files, segments and properties.col=delune.collection(indexdir=[dirs],mode=[CREATE|READ|APPEND],analyzer=None,logger=None)indexdir: path or list of path for using multiple disks efficientlymodeanalyzerlogger: # if logger configured by delune.configure, it’s not necessaryCollection has 2 major class: indexer and searcher.IndexerFor searching documents, it’s necessary to indexing text to build Inverted Index for fast term query.indexer=col.get_indexer(max_segments=int,force_merge=TrueorFalse,max_memory=10000000(10Mb),optimize=TrueorFalse)max_segments: maximum number of segments of index, if it’s over, segments will be merged. also note during indexing, segments will be created 3 times of max_segments and when called index.close (), automatically try to merge until segemtns is proper numbersforce_merge: When called index.close (), forcely try to merge to a single segment. But it’s failed if too big index - on 32bit OS > 2GB, 64bit > 10 GBmax_memory: if it’s over, created new segment on indexingoptimize: When called index.close (), segments will be merged by optimal number as possibleFor add docuemtn to indexer, create document object:document=delune.document()DeLune handle 3 objects as completly different objects between no relationshipreturning contentsnippet generating fieldsearcherble fieldsSet Returning ContentDeLune serialize returning contents by pickle, so you can set any objects pickle serializable.document.content({"userid":"hansroh","preference":{"notification":"email",...}})ordocument.content([32768,"This is smaple ..."])For saving multiple contents,document.content({"userid":"hansroh","preference":{"notification":"email",...}})document.content([32768,"This is smaple ..."])You can select one of these by query time using nthdoc=0 or 1 parameter.Snippet Generating FieldThis field should be unicode/utf8 encoded bytes.document.snippet("This is sample...")Searchable Fieldsdocument also recieve searchable fields:document.field(name,value,ftype=delune.TEXT,lang="un",encoding=None)document.field("default","violin sonata in c k.301",delune.TEXT,"en")document.field("composer","wolfgang amadeus mozart",delune.TEXT,"en")document.field("lastname","mozart",delune.STRING)document.field("birth",1756,delune.INT16)document.field("genre","01011111",delune.BIT8)document.field("home","50.665629/8.048906",delune.COORD6)name: if ‘default’, this field will be searched by simple string, or use ‘name:query_text’value: unicode/utf8 encode text, or should give encoding arg.ftype:see belowencoding: give like ‘iso8859-1’ if value is not unicode/utf8lang: language code for standard_analyzer, “un” (unknown) is defaultAvalible Field types are:TEXT: analyzable full-text, result-not-sortableTERM: analyzable full-text but position data will not be indexed as result can’t search phrase, result-not-sortableSTRING: exactly string match like nation codes, result-not-sortableLIST: comma seperated STRING, result-not-sortableFNUM: foramted number, value should be int or float and format parameter required, format is “digit.digit” that number of digit interger part with zero leading, and number of float part length. It make possible to search range efficiently.COORDn, n=4,6,8 decimal precision: comma seperated string ‘latitude,longititude’, latitude and longititude sould be float type range -90 ~ 90, -180 ~ 180. n is precision of coordinates. n=4 is 10m radius precision, 6 is 1m and 8 is 10cm. result-sortableBITn, n=8,16,24,32,40,48,56,64: bitwise operation, bit makred string required by n, result-sortableINTn, n=8,16,24,32,40,48,56,64: range, int required, result-sortableNote1: You make sure COORD, INT and BIT fields are at every documents even they havn’t got a value, because these types are depend on document indexed sequence ID. If they have’t a value, please set value to None NOT omit fields.Note2: FNUM 100.12345 with format=”5.3” is interanlly converted into “00100.123” and negative value will be -00100.123 and MAKE SURE your values are within -99999.999 and 99999.999.Repeat add_document as you need and close indexer.for...:document=delune.document()...indexer.add_document(document)indexer.close()If searchers using this collection runs with another process or thread, searcher automatically reloaded within a few seconds for applying changed index.SearcherFor running searcher, you should delune.configure () first and creat searcher.searcher=col.get_searcher(max_result=2000,num_query_cache=200)max_result: max returned number of searching results. default 2000, if set to 0, unlimited resultsnum_query_cache: default is 200, if over 200, removed by access time oldQuery is simple:searcher.query(qs,offset=0,fetch=10,sort="tfidf",summary=30,lang="un")qs: string (unicode) or utf8 encoded bytes. for detail query syntax, see belowoffset: return start position of result recordsfetch: number of records from offsetsort: “(+-)tfidf” or “(+-)field name”, field name should be int/bit type, and ‘-’ means descending (high score/value first) and default if not specified. if sort is “”, records order is reversed indexing ordersummary: number of terms for snippetlang: default is “un” (unknown)For deleting indexed document:searcher.delete(qs)All documents will be deleted immediatly. And if searchers using this collection run with another process or thread, theses searchers automatically reloaded within a few seconds.Finally, close searcher.searcher.close()Query Syntaxviolin composer:mozart birth:1700~1800search ‘violin’ in default field, ‘mozart’ in composer field and search range between 1700, 1800 in birth fieldviolin allcomposer:wolfgang mozartsearch ‘violin’ in default field and any terms after allcomposer will be searched in composer fieldviolin -sonata birth2:1700~1800birth2 is between ‘1700’ and ‘1800’violin -sonata birth:~1800not contain sonata in default fieldviolin -composer:mozartnot contain mozart in composer fieldviolin or piano genre:00001101/allmatched all 5, 6 and 8th bits are 1. also /any or /none is availableviolin or ((piano composer:mozart) genre:00001101/any)support unlimited priority ‘()’ and ‘or’ operators(violin or ((allcomposer:mozart wolfgang) -amadeus)) sonata (genre:00001101/none home:50.6656,8.0489~10000)search home location coordinate (50.6656, 8.0489) within 10 Km“violin sonata” genre:00001101/none home:50.6656/8.0489~10search exaclt phrase “violin sonata”“violin^3 piano” -composer:”ludwig van beethoven”search loose phrase “violin sonata” within 3 termsMigrating From Version 0.3xUpgdare linrariespip3install-UskitaiquestsdeluneThen restructuring directoriesDELUNE_ROOT="/home/delune"mkdir"$DELUNE_ROOT/delune"mv"$DELUNE_ROOT/models/.config""$DELUNE_ROOT/delune/config"mv"$DELUNE_ROOT/models""$DELUNE_ROOT/delune/collections"Edit your all config, removemodels/fro your data_dir option."data_dir":["models/mycols"]=>"data_dir":["mycols"]If you use RESTful API service, remove index or mirror related code lines at your app app launch script.Finally, run indexer.deluneindex-dv/home/delune-i300LinksGitLab RepositoryBug Report:GitLab issuesChange Log0.4 (June 2, 2018)officially seized developing naivebayes classifier & learnerintegrated local and remote indexing and searching APIsdirectory structure is NOT compatible with version 0.3x0.3 (Sep 15, 2017)fix wildcard & range searchfix snippet thingadd stem APIadd index field aliasing to documentadd string range searching, add new field type: ZFnadd multiple documents storing feature. as a result, DeLune can read only for Wissen collections0.2 (Sep 14, 2017)fix minor bugs0.1 (Sep 13, 2017)change package name from Wissen to DeLuneEarlier Wissen Period0.13fix using lockadd truncate collection APIfix updating documentchange replicating way to use sticky session connection with origin serverfix file creation mode on posixfix using lock with multiple workerschange wissen.document method namesfix index queue file locking0.12add biword arg to standard_analyzerchange export package name from appack to packageadd Skito-Saddle appfix analyzer.count_stopwords return valuechange development status to Alphaadd wissen.assign(alias, searcher/classifier) and query(alias), guess(alias)fix threads count and memory allocationadd example for Skitai App Engine app to mannual0.11fix HTML strip and segment merging etc.add MULTIPATH classifieradd learner.optimize ()make learner.build & learner.train efficient0.10 - change version format, remove all str*_s ()0.9 - support Python 3.x0.8 - change license from BSD to GPL V3