Repository: run-llama/llama_index
Pull request: remove commonly unpicklable entries
# Description

Tiktoken is famously un-picklable, but it's used all across LlamaIndex.

However, with a sneaky wave of a wand (and relying on `BaseComponent` inheritance), we can fix this by removing unpicklable entries.

This probably won't cover every edge case, but it gets most.

Tested with `pickle.dumps(index)` and `pickle.dumps(index.as_query_engine())`.

Fixes https://github.com/run-llama/llama_index/issues/8835

## Type of Change

- [x] Bug fix (non-breaking change which fixes an issue)



Problem statement:
[Question]: How to Query RAG  systems with Multiple prompt concurently 
### Question Validation

- [X] I have searched both the documentation and discord for an answer.

### Question

I want to query the RAG system using multiple questions/queries concurrently.

I am using multiprocessing to do this but llama-index RAG kept throwing an error  `TypeError: cannot pickle 'builtins.CoreBPE' object` as a result of llama-index implementation of RAG that doesn't support multiprocessing.

Is there a module I can use for this or a solution to this?


Code:

```
from llama_index import VectorStoreIndex
from llama_index import Document
import multiprocessing

class ChatBotManager:
    def __init__(self, rerank):
        self.rerank = rerank
      
    def get_query_engine(self, document):
        #  load document
        # docs = self.get_docs(document)
        # build index
        index = VectorStoreIndex.from_documents(documents=document)
        query_engine = index.as_query_engine(
            similarity_top_k=10, node_postprocessors=[self.rerank]
        )

        return query_engine
    
    # @staticmethod
    def chat_with_data_bot(self, prompt, query_engine):
        """
        Perform a chat interaction with the bot.
        
        Args:
            prompt: User's input.
            qa: Initialized RetrieverQueryEngine 
            
        Returns:
            Bot's response.
        """
        
        response = query_engine.query(
                        prompt,
                    )
        
        return  response
    
    def parallel_process_inputs(self, input_value, query_engine):
        if type(input_value) == str:
            results = self.chat_with_data_bot(input_value, query_engine)
        elif type(input_value) == tuple:
            with multiprocessing.Pool(processes=len(input_value)) as pool:
                results = pool.starmap(self.chat_with_data_bot, [(input1, query_engine) for input1 in input_value])
            results = {'company_profile': results[0], 'company_details': results[1]}
        else: 
            with multiprocessing.Pool(processes=len(input_value)) as pool:
                results = pool.starmap(self.chat_with_data_bot, [(input1, query_engine) for input1 in input_value])
            results = {'requirement': results[0],'financial_requirement':results[1], 'scope': results[2], 'evaluation_criteria': results[3], "expected_rfp_response": results[4],
                       'budget_and_financials': results[5], 'legal_and_contractual': results[6], 'submission_guidelines': results[7], "project_timeline": results[8],
                       'vendor_responsibilities': results[9], 'communication_and_reporting': results[10], 'objective': results[11], "deliverable": results[12],
                       'qualification': results[13], "technical_requirement": results[14], "project_duration_and_key_stages": results[15],
                       'collaboration_methods': results[16], "update_frequency": results[17]}
            
        return results
        
 
if __name__ == "__main__":
    from pathlib import Path
   from llama_index import download_loader

   PDFReader = download_loader("PDFReader")

    loader = PDFReader()
   documents = loader.load_data(file=Path(path))
   chatbot = ChatBotManager(rerank)
   query_engine = chatbot.get_query_engine(documents)
   output = chatbot.parallel_process_inputs(information_list, query_engine)

```

information_list contains a lot of questions.

Hints from issue discussion:
_🤖_
<!-- Greeting -->
Hi @okoliechykwuka! Good to see you again. I'm diving into the issue you've reported about querying the RAG system with multiple prompts concurrently. I'll provide a thorough response shortly.
<!-- Will answer -->



_🤖_
<!-- Answer -->
Based on the information you've provided and the similar issues I found in the LlamaIndex repository, it seems like the problem you're encountering is related to the pickling of certain objects in Python, specifically the 'builtins.CoreBPE' object. This is a known issue with multiprocessing in Python, as not all objects can be pickled (serialized) for inter-process communication.

In the issue [cannot pickle 'builtins.CoreBPE' object](https://github.com/run-llama/llama_index/issues/886), a similar problem was encountered. The solution was to make the `ChatMemoryBuffer` pickleable, as detailed in the pull request [Make ChatMemoryBuffer Pickleable](https://github.com/run-llama/llama_index/pull/7205). 

However, in your case, the problem seems to be with the `VectorStoreIndex` object. Unfortunately, the LlamaIndex does not currently support multiprocessing due to the underlying implementation of certain components. 

As a workaround, you might consider using threading instead of multiprocessing. Python threads can share memory and don't require pickling of objects. However, due to Python's Global Interpreter Lock (GIL), this might not lead to a significant speedup.

Another alternative could be to use a different approach for concurrent processing that doesn't require pickling, such as using `concurrent.futures` or asynchronous programming with `asyncio`.

Please note that these are just suggestions and might require significant changes to your current implementation. If you need further assistance, please provide more details about your use case and requirements.

## Sources
- [cannot pickle 'builtins.CoreBPE' object ](https://github.com/run-llama/llama_index/issues/886)

---

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

Commit message: remove commonly unpicklable entries

Patch to review:
diff --git a/llama_index/schema.py b/llama_index/schema.py
index 7df7059435..dc114a785d 100644
--- a/llama_index/schema.py
+++ b/llama_index/schema.py
@@ -42,6 +42,14 @@ def class_name(cls) -> str:
         name changes.
         """
 
+    def __getstate__(self) -> Dict[str, Any]:
+        state = self.dict()
+        # Remove common unpicklable entries
+        state.pop("tokenizer", None)
+        state.pop("tokenizer_fn", None)
+        state.pop("callback_manager", None)
+        return state
+
     def to_dict(self, **kwargs: Any) -> Dict[str, Any]:
         data = self.dict(**kwargs)
         data["class_name"] = self.class_name()


Code review comments:
diff_hunk: @@ -42,6 +42,14 @@ def class_name(cls) -> str:
         name changes.
         """
 
+    def __getstate__(self) -> Dict[str, Any]:
+        state = self.dict()
+        # Remove common unpicklable entries
+        state.pop("tokenizer", None)
+        state.pop("tokenizer_fn", None)
+        state.pop("callback_manager", None)
path: llama_index/schema.py
text: @user1:
are these easy to retrieve when one wants to load the pickle?

@author:
Callback manager less-so, tokenizer yes (and especially since the tokinzer largely isn't customizable right now, but in v0.9 will be able to change the global default)

@author:
Maybe we can keep the callback manager for now actually

@user1:
cool! yeah +1 for keeping callback manager for now until one can easily reload it

@author:
Ok, kept the callback manager for now, it still works, but some callback managers might cause issues (we can fix that if the issue comes up on the callback manager directly)