Package: searchkit
SearchkitPython library providing tools to search files in parallel.Search TypesDifferent types of search are supported. Add one or more search definition to aFileSearcherobject, registering them against a file, directory or glob path. Results are collected and returned as aSearchResultsCollectionwhich provides different ways to retrieve results.Simple SearchTheSearchDefclass supports matching one or more patterns against each line in a file. Patterns are executed until the first match is found.When defining a search, you can optionally specify field names so that result values can be retrieved by name rather than index e.g. for the following content:PID TTY          TIME CMD
 111024 pts/4    00:00:00 bash
 111031 pts/4    00:00:00 psYou can define as search as follows:SearchDef(r'.*(\S+)\s+(\S+)\s+(\S+)\s+(\S+)')and retrieve results with:forrinresults:pid=r.get(1)tty=r.get(2)time=r.get(3)cmd=r.get(4)or alternatively:forrinresults:pid,tty,time,cmd=ror you can provide field names and types:fields=ResultFieldInfo({'PID':int,'TTY':str,'TIME':str,'CMD':str})SearchDef(r'.*(\S+)\s+(\S+)\s+(\S+)\s+(\S+)',field_info=fields)and retrieve results with:forrinresults:pid=r.PIDtty=r.TTYtime=r.TIMEcmd=r.CMDSequence SearchTheSequenceSearchDefclass supports matching string sequences ("sections") over multiple lines by matching a start, end and optional body in between. These section components are each defined with their ownSearchDefobject.Search ConstraintsIf searching e.g. a log file where each line starts with a timestamp and you only want results that match after a specific time then you can usesearch.constraints.SearchConstraintSearchSinceand apply to either the whole file or each line in turn. The latter allows constraints to be associated with a SearchDef and therefore only apply within the context of that search.Installationsearchkit is packaged inpypiand can be installed as follows:sudo apt install python3-pippip install searchkitExample UsageAn example simple search is as follows:fromsearchkitimportFileSearcher,SearchDeffname='foo.txt'open(fname,'w').write('the quick brown fox')fs=FileSearcher()fs.add(SearchDef(r'.+ \S+ (\S+) .+'),fname)results=fs.run()forrinresults.find_by_path(fname):print(r.get(1))An example sequence search is as follows:fromsearchkitimportFileSearcher,SequenceSearchDef,SearchDefcontent="""Traceback (most recent call last):File "<stdin>", line 1, in <module>ValueError: invalid literal for int() with base 10: 'foo'"""fname='my.log'open(fname,'w').write(content)start=SearchDef(r'Traceback')body=SearchDef(r'.+')# terminate sequence with start of next or EOF so no end def needed.fs=FileSearcher()fs.add(SequenceSearchDef(start,tag='myseq',body=body),fname)results=fs.run()forseq,resultsinresults.find_sequence_by_tag('myseq').items():forrinresults:if'body'inr.tag:print(r.get(0))An example search with constraints is as follows:fromsearchkitimportFileSearcher,SearchDeffromsearchkit.constraintsimportSearchConstraintSearchSince,TimestampMatcherBaseclassMyDateTimeMatcher(TimestampMatcherBase):@propertydefpatterns(self):return[r'^(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2}) 'r'(?P<hours>\d{2}):(?P<minutes>\d{2}):(?P<seconds>\d{2})']fname='foo.txt'withopen(fname,'w')asfd:fd.write('2023-01-01 12:34:24 feeling cold\n')fd.write('2023-06-01 12:34:24 feeling hot')today='2023-06-02 12:34:24'constraint=SearchConstraintSearchSince(today,None,ts_matcher_cls=MyDateTimeMatcher)fs=FileSearcher(constraint=constraint)fs.add(SearchDef(r'\S+ \S+ \S+ (\S+)'),fname)results=fs.run()forrinresults.find_by_path(fname):print(r.get(1)=='hot')