Package: pyterrier-caching
Caching for PyTerrierpyterrier-cachingprovides several components for caching intermediate results.The right component will depend on your use case.InstallationInstall this package using pip:pipinstallpyterrier-cachingCaching Results when IndexingIndexerCachesaves the sequence of documents encountered in an indexing pipeline.
It allows you to repeat that sequence, without needing to re-execute the computations
up to that point.Example use case:I want to test how different retrieval engines perform overlearned
sparse representations, but I don't want to
re-compute the representations each time.You use anIndexerCachethe same way you would use an indexer: as the last component
of a pipeline. Rather than building an index of the data, theIndexerCachewill save
your results to a file on disk. This file can be re-read by iterating over the cache
object withiter(cache).Example:importpyterrierasptpt.init()frompyterrier_cachingimportIndexerCache# Setupcache=IndexerCache('path/to/cache')dataset=pt.get_dataset('some-dataset')# e.g., 'irds:msmarco-passage'# Use the IndexerCache cache object just as you would an indexercache_pipeline=MyExpensiveTransformer()>>cache# The following line will save the results of MyExpensiveTransformer() to path/to/cachecache_pipeline.index(dataset.get_corpus_iter())# Now you can build multiple indexes over the results of MyExpensiveTransformer without# needing to re-run it each timeindexer1=...# e.g., pt.IterDictIndexer('./path/to/index.terrier')indexer1.index(iter(cache))indexer2=...# e.g., pyterrier_pisa.PisaIndex('./path/to/index.pisa')indexer2.index(iter(cache))👁‍🗨 More DetailsIndexerCachecurrently has one implementation,Lz4PickleIndexerCache, which is
set as the default.Lz4PickleIndexerCachesaves the sequence as a sequence of
LZ4-compressed pickled dicts in the file:data.pkl.lz4. Byte-level offsets for each
document are stored as a numpy-compatible float64 array inoffsets.np. If thedocnocolumn is present, annpidsstructure is also stored, facilitating reverse-lookups
of documents by their docno.Caching Results from a ScorerScorerCachesaves thescorebased onqueryanddocno. When the samequery-docnocombination is encountered again, the value is read from the cache,
avoiding re-computation.Example use case:I want to test a neural relevance model over several first-stage
retrieval models, but they bring back many of the same documents, so I don't want to
re-compute the scores each time.You use aScorerCachein place of the scorer in a pipeline. It holds a reference to
the scorer so that it can compute values that are missing from the cache. You need to
"build" aScorerCachebefore you can use it, which creates an internal mapping between
the stringdocnoand the integer indexes at which the scores values are stored.⚠️ Important Caveats:ScorerCachesaves scores based ononlythe value of thequeryanddocno. All
other information is ignored (e.g., the text of the document).ScorerCacheonly stores the result of thescorecolumn. All other outputs of the
scorer will be discarded. (Rank is also outputed, but is caculated byScorerCache,
not the scorer.)Due to limitations of HDF5, only a single process can have the cache open for writing
at a time.Scores are saved asfloat32values. Other values will attempted to be cast up/down
to float32.The value ofnanis reserved as an indicator that the value is missing from the cache.
Scorers should not return this value.AScorerCacherepresents the cross between a scorer and a corpus. Do not try to use a
single cache across multiple scorers or corpora -- you'll get unexpected/invalid results.Example:importpyterrierasptpt.init()frompyterrier_cachingimportScorerCache# Setupcached_scorer=ScorerCache('path/to/cache',MyExpensiveScorer())dataset=pt.get_dataset('some-dataset')# e.g., 'irds:msmarco-passage'# You need to build your cache before you can use it. There are several# ways to do this:ifnotcached_scorer.built():# Easiest:cached_scorer.build(dataset.get_corpus_iter())# If you already have an "npids" file to map the docnos to indexes, you can use:# >>> cached_scorer.build(docnos_file='path/to/docnos.npids')# This will be faster than iterating over the entire corpus, especially for# large datasets.# Use the IndexerCache cache object just as you would an indexercached_pipeline=MyFirstStage()>>cached_scorercached_pipeline(dataset.get_topics())# Will be faster when you run it a second time, since all values are cachedcached_pipeline(dataset.get_topics())# Will only compute scores for docnos that were not returned by MyFirstStage()another_cached_pipeline=AnotherFirstStage()>>cached_scoreranother_cached_pipeline(dataset.get_topics())👁‍🗨 More DetailsScorerCachecurrently has one implementation,Hdf5ScorerCache, which is
set as the default.Hdf5ScorerCachesaves scores in an HDF5 file.ExtrasThese components are not cachingper se, but can be helpful when constructing a
caching pipeline.Lazy(...)allows you to build a transformer object that is only initialized when
it is first executed. This can help avoid the expensive process of reading and loading
a model that may never be executed due to caching.For example, this example usesLazywith aScorerCacheto avoid loadingMyExpensiveTransformerunless it's actually needed:frompyterrier_cachingimportLazy,ScorerCachelazy_transformer=Lazy(lambda:MyExpensiveTransformer())cache=ScorerCache('path/to/cache',lazy_transformer)