Package: theine
TheineHigh performance in-memory cache inspired byCaffeine.High performanceRust coreHigh hit ratio withW-TinyLFUorClock-Proeviction policyExpired data are removed automatically usinghierarchical timer wheelTTL must be considered in in-memory caching because
it limits the effective (unexpired) working set size. Efficiently removing expired objects from cache needs to be
prioritized over cache eviction. -A large scale analysis of hundreds of in-memory
cache clusters at TwitterSimple APIDjango cache backendTable of ContentsRequirementsInstallationCache Eviction PoliciesAPIDecoratorDjango Cache BackendMetadata Memory OverheadBenchmarkscontinuous benchmark10k requestshit ratiosSupportRequirementsPython 3.7+Installationpip install theineCache Eviction PoliciesTheine provides 3 built in cache eviction policies:LRUDiscards the least recently used items first.W-TinyLFUAn approximate LFU policy in order to boost the effectiveness of caches subject to skewed access distributions.Theine uses an adaptive version of W-TinyLFU to get better hit ratio under different types of workloads.Reference:https://arxiv.org/pdf/1512.00727.pdfClock-PROAn improved CLOCK replacement policy(CLOCK: an approximation of LRU), based onPyClockPro.Reference:https://static.usenix.org/event/usenix05/tech/general/full_papers/jiang/jiang_html/html.htmlAPIKey should be aHashableobject, and value can be anyPython object. If key type is notstr/int, Theine will generate a unique key string automatically, this unique str will use extra space in memory and increase get/set/remove overhead.Each Cache instance will span a thread to evict expired entries proactively, and the overhead of cache instance init is relatively high. Sodon't create instance dynamically in your function. Django adapter will create a global cache instance autmoatically, and when using theMemoizedecorator, please make sure your cache instance is created globally, instead of creating a new one in each run.Please be aware the Cache class isnotthread-safe.fromtheineimportCachefromdatetimeimporttimedelta# tlfu is the eviction policy, Theine provide 3 policies lru/tlfu/clockprocache=Cache("tlfu",10000)# without default, return None on missv=cache.get("key")# with default, return default on misssentinel=object()v=cache.get("key",sentinel)# set with ttlcache.set("key",{"foo":"bar"},timedelta(seconds=100))# delete from cachecache.delete("key")# close cache, stop timing wheel threadcache.close()# clear cachecache.clear()# get current cache stats, please call stats() again if you need updated statsstats=cache.stats()print(stats.request_count,stats.hit_count,stats.hit_rate)# get cache max sizecache.max_size# get cache current sizelen(cache)DecoratorTheine support hashable keys, so to use a decorator, a function to convert input signatures to hashable is necessary.The recommended way is specifying the function explicitly, this is approach 1, Theine also support generating key automatically, this is approach 2. Same as Theine API, if key function return type is notstr/int, Theine will generate a unique key string automatically, this unique str will use extra space in memory and increase get/set/remove overhead.- explicit key functionfromtheineimportCache,Memoizefromdatetimeimporttimedelta@Memoize(Cache("tlfu",10000),timedelta(seconds=100))deffoo(a:int)->int:returna@foo.keydef_(a:int)->str:returnf"a:{a}"foo(1)# asyncio@Memoize(Cache("tlfu",10000),timedelta(seconds=100))asyncdeffoo_a(a:int)->int:returna@foo_a.keydef_(a:int)->str:returnf"a:{a}"awaitfoo_a(1)ProsBoth sync and async support.Explicitly control how key is generated. Most remote cache(redis, memcached...) only allow string keys, return a string in key function make it easier when you want to use remote cache later.Thundering herd protection(multithreading: setlock=TrueinMemoize, asyncio: always enabled).Type checked. Mypy can check key function to make sure it has same input signature as original function and return a hashable.ConsYou have to use 2 functions.Performance. Theine API: around 8ms/10k requests ->> decorator: around 12ms/10k requests.- auto key functionfromtheineimportCache,Memoizefromdatetimeimporttimedelta@Memoize(Cache("tlfu",10000),timedelta(seconds=100),typed=True)deffoo(a:int)->int:returnafoo(1)# asyncio@Memoize(Cache("tlfu",10000),timedelta(seconds=100),typed=True)asyncdeffoo_a(a:int)->int:returnaawaitfoo_a(1)ProsSame as explicit key version.No extra key function.ConsWorse performance: around 18ms/10k requests.Unexpected memory usage. The auto key function use same methods as Python's lru_cache. Take a lookthis issueorthis one.Django Cache BackendCACHES={"default":{"BACKEND":"theine.adapters.django.Cache","TIMEOUT":300,"OPTIONS":{"MAX_ENTRIES":10000,"POLICY":"tlfu"},},}Metadata Memory OverheadAssume your key is 24 bytes long, then each meta key entry in Rust is 92 bytes. For 1 million keys, the total memory overhead is 92 megabytes. Clock-Pro will use2xmeta space, which is 184 megabytes.BenchmarksPython version: 3.11OS: Ubuntu 22.04.2 LTScontinuous benchmarkhttps://github.com/Yiling-J/cacheme-benchmark10k requestsCachetools:https://github.com/tkem/cachetoolsCacheout:https://github.com/dgilland/cacheoutSource Code:https://github.com/Yiling-J/theine/blob/main/benchmarks/benchmark_test.pyWrite and Mix Zipf use 1k max cache size, so you can see the high cost of traditional LFU eviction policy here.ReadWriteMix ZipfTheine(Clock-Pro) API3.07 ms9.86 msTheine(W-TinyLFU) API3.42 ms10.14 msTheine(W-TinyLFU) Auto-Key Decorator7.17 ms18.41 ms13.18 msTheine(W-TinyLFU) Custom-Key Decorator6.45 ms17.67 ms11.50 msCachetools LFU Decorator15.70 ms627.10 ms191.04 msCacheout LFU Decorator50.05 ms704.70 ms250.95 msTheine(LRU) Custom-Key Decorator5.70 ms16.04 ms10.91 msCachetools LRU Decorator14.05 ms61.06 ms36.89 msCacheout LRU Decorator47.90 ms94.94 ms68.25 mshit ratiosAll hit ratio benchmarks use small datasets and finish in seconds/minutes, better to try Theine yourself and focus on whether the cache exceeds your performance needs and has the desired capabilities.Source Code:https://github.com/Yiling-J/theine/blob/main/benchmarks/trace_bench.pyzipfsearchThis trace is described as "disk read accesses initiated by a large commercial search engine in response to various web search requests."databaseThis trace is described as "a database server running at a commercial site running an ERP application on top of a commercial database."Scarabresearch database traceScarabresearch 1 hour database trace from thisissueMeta anonymized traceMeta shared anonymized trace captured from large scale production cache services, fromcachelibSupportOpen an issue, ask question in discussions or join discord channel:https://discord.gg/StrgfPaQqETheine Go version is also available, which focus on concurrency performance, take a look if you are interested:Theine Go.