Package: elasticsearch-llm-cache
Elasticsearch LLM CacheA Python library to utilize Elasticsearch as a caching layer for Generative AI applications. By caching the responses from Language Models (LLMs), this library helps in reducing costs associated with LLM services and improving response speed from the user's perspective.Blog PostThis library is covered in depth in the blog postElasticsearch as a GenAI Caching Layer.Key BenefitsReduce costs for LLM services by reusing previous responses.Improve response speed as seen by end users by serving cached responses.Class:ElasticsearchLLMCacheThis class provides the core functionality for creating, querying, and updating a cache index in Elasticsearch.__init__(self, es_client: Elasticsearch, index_name: Optional[str] = None, es_model_id: Optional[str] = 'sentence-transformers__all-distilroberta-v1', create_index=True)Constructor method to initialize the ElasticsearchLLMCache instance.es_client(Elasticsearch): Elasticsearch client object.index_name(str, optional): Name for the index; defaults to 'llm_cache'.es_model_id(str, optional): Model ID for text embedding; defaults to 'sentence-transformers__all-distilroberta-v1'.create_index(bool, optional): Whether to create a new index; defaults to True.create_index(self, dims: Optional[int] = 768) -> DictMethod to create a new index for caching if it does not already exist.dims(int, optional): The dimensionality of the vector; defaults to 768.Returns:Dictionary containing information about index creation.Mapping:prompt: textresponse: textcreate_date: datelast_hit_date: dateprompt_vector: dense_vectorquery(self, prompt_text: str, similarity_threshold: Optional[float] = 0.5) -> dictMethod to query the index to find similar prompts and update thelast_hit_datefor that document if a hit is found.prompt_text(str): The text of the prompt to find similar entries for.similarity_threshold(float, optional): The similarity threshold for filtering results; defaults to 0.5.Returns:A dictionary containing the hits or an empty dictionary if no hits are found.add(self, prompt: str, response: str, source: Optional[str] = None) -> dictMethod to add a new document to the index when there is no cache hit and a response is fetched from LLM.prompt(str): The user prompt.response(str): The LLM response.source(str, optional): Source identifier for the LLM.Returns:A dictionary indicating the successful caching of the new prompt and response.Example UsagefromelasticsearchimportElasticsearchfromelasticsearch_llm_cache.elasticsearch_llm_cacheimportElasticsearchLLMCache# Initialize Elasticsearch clientes_client=Elasticsearch()# Initialize Elasticsearch LLM Cachellm_cache=ElasticsearchLLMCache(es_client)# Query the cachecache_response=llm_cache.query(prompt_text="Hello, how can I help?")# If no cache hit, add new response to cacheifnotcache_response:llm_response="I'm here to assist you!"# Assume this response is fetched from LLMllm_cache.add(prompt="Hello, how can I help?",response=llm_response)Example FileselasticRAG_with_cache.pySample Streamlit App using RAG with Elasticsearchuses existing ElasticSearch Elastic Cloud Deploymentuses either OpenAI or Azure OpenAI for GenAIcan be modified to work with any GenAI serviceEnvironment VariablesOPENAI_API_TYPEOPENAI_API_BASEOPENAI_API_VERSIONOPENAI_API_ENGINEELASTIC_APM_SERVICE_NAMEELASTIC_CLOUD_IDELASTIC_USERELASTIC_PASSWORDELASTIC_INDEX_DOCStest_elasticsearch_llm_cache.pyUnit tests for ElasticsearchLLMCache class