ConversationalRetrievalChain using local LLM models and tokenizers
Labels: name: auto:improvement, name: area: models

### System Info

I'm using a locally hosted LLM and want to apply Langchain's ConversationalRetrievalChain or RetrievalQA in an offline setting for chatbot developments, however there is an error as the current configuration do not support local hosted tokenizer. 

Appreciate if you can help advise on the modifications to required codes to use local general tokenizers (not just gpt2 tokenizer but any tokenizer in general) in an offline setting.

```
> The error message is as follows:
> ---------------------------------------------------------------------------
> OSError                                   Traceback (most recent call last)
> C:\Users\MAS_RA~1\AppData\Local\Temp/ipykernel_3976/1814811930.py in
>      18     if query == '':
>      19         continue
> ---> 20     result = llama2_7B_qa(
>      21         {"question": query, "chat_history": chat_history})
>      22     print(f"{blue}Answer: " + result["answer"])
> ~\Documents\Wheels\langchain\chains\base.py in __call__(self, inputs, return_only_outputs, callbacks, tags, metadata, run_name, include_run_info)
>     290         except BaseException as e:
>     291             run_manager.on_chain_error(e)
> --> 292             raise e
>     293         run_manager.on_chain_end(outputs)
>     294         final_outputs: Dict[str, Any] = self.prep_outputs(
> ~\Documents\Wheels\langchain\chains\base.py in __call__(self, inputs, return_only_outputs, callbacks, tags, metadata, run_name, include_run_info)
>     284         try:
>     285             outputs = (
> --> 286                 self._call(inputs, run_manager=run_manager)
>     287                 if new_arg_supported
>     288                 else self._call(inputs)
> ~\Documents\Wheels\langchain\chains\conversational_retrieval\base.py in _call(self, inputs, run_manager)
>     132         )
>     133         if accepts_run_manager:
> --> 134             docs = self._get_docs(new_question, inputs, run_manager=_run_manager)
>     135         else:
>     136             docs = self._get_docs(new_question, inputs)  # type: ignore[call-arg]
> ~\Documents\Wheels\langchain\chains\conversational_retrieval\base.py in _get_docs(self, question, inputs, run_manager)
>     287             question, callbacks=run_manager.get_child()
>     288         )
> --> 289         return self._reduce_tokens_below_limit(docs)
>     290
>     291     async def _aget_docs(
> ~\Documents\Wheels\langchain\chains\conversational_retrieval\base.py in _reduce_tokens_below_limit(self, docs)
>     265             self.combine_docs_chain, StuffDocumentsChain
>     266         ):
> --> 267             tokens = [
>     268                 self.combine_docs_chain.llm_chain.llm.get_num_tokens(doc.page_content)
>     269                 for doc in docs
> ~\Documents\Wheels\langchain\chains\conversational_retrieval\base.py in (.0)
>     266         ):
>     267             tokens = [
> --> 268                 self.combine_docs_chain.llm_chain.llm.get_num_tokens(doc.page_content)
>     269                 for doc in docs
>     270             ] 
> ~\Documents\Wheels\langchain\schema\language_model.py in get_num_tokens(self, text)
>     252             The integer number of tokens in the text.
>     253         """
> --> 254         return len(self.get_token_ids(text))
>     255
>     256     def get_num_tokens_from_messages(self, messages: List[BaseMessage]) -> int:
> ~\Documents\Wheels\langchain\schema\language_model.py in get_token_ids(self, text)
>     239                 in the text.
>     240         """
> --> 241         return _get_token_ids_default_method(text)
>     242
>     243     def get_num_tokens(self, text: str) -> int:
> ~\Documents\Wheels\langchain\schema\language_model.py in _get_token_ids_default_method(text)
>      42     """Encode the text into token IDs."""
>      43     # get the cached tokenizer
> ---> 44     tokenizer = get_tokenizer()
>      45
>      46     # tokenize the text using the GPT-2 tokenizer
> ~\Documents\Wheels\langchain\schema\language_model.py in get_tokenizer()
>      36         )
>      37     # create a GPT-2 tokenizer instance
> ---> 38     return GPT2TokenizerFast.from_pretrained("gpt2")
>      39
>      40
> ~\Documents\Wheels\transformers\tokenization_utils_base.py in from_pretrained(cls, pretrained_model_name_or_path, cache_dir, force_download, local_files_only, token, revision, *init_inputs, **kwargs)
>    1836
>    1837         if all(full_file_name is None for full_file_name in resolved_vocab_files.values()):
> -> 1838             raise EnvironmentError(
>    1839                 f"Can't load tokenizer for '{pretrained_model_name_or_path}'. If you were trying to load it from "
>    1840                 "'https://huggingface.co/models', make sure you don't have a local directory with the same name. "
> OSError: Can't load tokenizer for 'gpt2'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'gpt2' is the correct path to a directory containing all relevant files for a GPT2TokenizerFast tokenizer.
```

### Who can help?

@hwchase17
@agola11

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
# ConversationalRetrievalChain 

from langchain.chains import ConversationalRetrievalChain
from langchain.llms import CTransformers

LocalHostedInteractiveBot = ConversationalRetrievalChain.from_llm(
    CTransformers(model="./models/llama-2-7b-chat.Q5_K_M.gguf", model_type="llama"),
    vectordb.as_retriever(search_kwargs={'k': 6}),
    return_source_documents=True,
    verbose=False,
    max_tokens_limit=1000
)

# Terminal interaction with locally hosted LLM 
chathistory = []

while True:
    query = input(f" Prompt: ")
    if query == "exit":
        print('Bye bye')
        sys.exit()
    if query == '':
        continue
    result = LocalHostedInteractiveBot(
        {"question": query, "chat_history": chathistory})
    print(f" Question: " + query)
    print(f"Answer: " + result["answer"])
    chat_history.append((query, result["answer"]))
```

### Expected behavior

The prompt should obtain a chatbot response from the LLM via the retrieval augmented generation methods (ConversationalRetrievalChain or RetrievalQA) in langchain but failed to do so as the current configuration is unable to support local tokenizer.