Repository: run-llama/llama_index
Pull request: feat(UX): provide `all-MiniLM-L6-v2` as local `embed_model`
# Description

Usage:
```python
service_context = ServiceContext(embed_model="local")
```
<details>

For the choice of `all-MiniLM-L6-v2`, see the cost performance analysis [here](https://medium.com/@nils_reimers/openai-gpt-3-text-embeddings-really-a-new-state-of-the-art-in-dense-text-embeddings-6571fe3ec9d9). Just eyeballing suggested that this embed model running locally on CPU is faster than calling the `text-embedding-ada-002` model (although, perhaps this was due to calling this API sequentially? Still, it was really fast for ~40 nodes). Further benchmarks will be performed in https://github.com/jerryjliu/llama_index/issues/6804

</details>

Partially fixes: https://github.com/jerryjliu/llama_index/issues/6720

<details> 

There are two questions here:
1. Is the API ok?
2. Is this way of instantiating the local model (using langchain) ok, or should we define our own `HuggingFaceEmbedding` class?

</details>
---

Advanced usage:

```python
service_context = ServiceContext(embed_model="local:sentence-transformers/all-mpnet-base-v2")
```

No longer planned:
<details>

Future: (multiprocessing)
```python
service_context = ServiceContext(embed_model="local-fast")
service_context = ServiceContext(embed_model="local-fast:sentence-transformers/all-mpnet-base-v2")
```

</details>

<details>
---

In fact, I am inclined to think this is the better default than `OpenAIEmbedding`. So one should have to explicitly pass in an `OpenAIEmbedding`

If we are OK with moving forward with this, we can get rid of the `embed_model="local"` API. Perhaps we can hold for the benchmark results before making a final decision.

---


</details>

### Type of Change

- [X] New feature (non-breaking change which adds functionality)
- [X] This change requires a documentation update

### How Has This Been Tested?
- [X] I stared at the code and made sure it makes sense

###  Suggested Checklist:

- [X] I have performed a self-review of my own code
- [x] I have made corresponding changes to the documentation
- [X] New and existing unit tests pass locally with my changes


Problem statement:
[Feature Request]: Local-model defaults for non-predictor sensitive experimentation
### Feature Description

It can be costly to run experiments using paid APIs, especially, if code is accidentally re-run.

Hence, it should be as easy as `ServiceContext(local=true)` to download e.g. from huggingface, store in local cache, and utilize local model. Local model should be lightweight but effective (100M-500M params, e.g. BERT/sentence transformer).

To evaluate: is dev loop slower due to high latency from local model (e.g. on CPU-only devices)

### Reason
Configuring a cheap local model each time is cumbersome.

### Value of Feature
Save cost, save hassle

Hints from issue discussion:
Good proposal, in my opinion.
Once https://github.com/jerryjliu/llama_index/issues/6692 is implemented, we should decide if it's better to also use the subprocess service.
One concern is that any model that is that size (100M-500M) parameters will likely not be useful enough as an LLM. LLMs at this size (like GPT-2) struggle to follow instructions. At least, the last time I looked this was the case. Maybe this has changed recently. 

For an embedding model though, its a decent size.
Indeed. However, what I was thinking was to provide the ggml 4bit quantized models, (either 7B or 13B) as the local default, for llm which includes models like vicuna and I think orca and also open license models like OpenLlama and perhaps MPT and (need to check for the full list). 

These models can run with 4GB and 8GB RAM respectively, with 30B requiring about 18GB RAM.

I've worked with ggml models before (llama.cpp) so it shouldn't be too difficult to implement this.

(Actually, I think Langchain LLM already implements ggml, we could just use this. See https://python.langchain.com/docs/modules/model_io/models/llms/integrations/llamacpp)
Btw, switching between models easily like this is one of the motivations for a global service context. Perhaps the API would be

```python
service_context.to_local()
service_context.to_defaults()
```

Not too sure yet.

Commit message: commit

Patch to review:
diff --git a/llama_index/indices/service_context.py b/llama_index/indices/service_context.py
index c192bfdb86..e7921ed938 100644
--- a/llama_index/indices/service_context.py
+++ b/llama_index/indices/service_context.py
@@ -1,6 +1,6 @@
 import logging
 from dataclasses import dataclass
-from typing import Optional
+from typing import Optional, Union
 
 import llama_index
 from llama_index.callbacks.base import CallbackManager
@@ -14,6 +14,7 @@
 from llama_index.logger import LlamaLogger
 from llama_index.node_parser.interface import NodeParser
 from llama_index.node_parser.simple import SimpleNodeParser
+from llama_index.embeddings import LangchainEmbedding
 
 logger = logging.getLogger(__name__)
 
@@ -72,7 +73,7 @@ def from_defaults(
         llm_predictor: Optional[BaseLLMPredictor] = None,
         llm: Optional[LLMType] = None,
         prompt_helper: Optional[PromptHelper] = None,
-        embed_model: Optional[BaseEmbedding] = None,
+        embed_model: Optional[Union[BaseEmbedding, str]] = None,
         node_parser: Optional[NodeParser] = None,
         llama_logger: Optional[LlamaLogger] = None,
         callback_manager: Optional[CallbackManager] = None,
@@ -96,6 +97,7 @@ def from_defaults(
             llm_predictor (Optional[BaseLLMPredictor]): LLMPredictor
             prompt_helper (Optional[PromptHelper]): PromptHelper
             embed_model (Optional[BaseEmbedding]): BaseEmbedding
+                or "local" (use local model)
             node_parser (Optional[NodeParser]): NodeParser
             llama_logger (Optional[LlamaLogger]): LlamaLogger (deprecated)
             chunk_size (Optional[int]): chunk_size
@@ -111,6 +113,21 @@ def from_defaults(
             )
             chunk_size = chunk_size_limit
 
+        if isinstance(embed_model, str):
+            if embed_model != "local":
+                raise ValueError(
+                    "embed_model must be str 'local' or of type BaseEmbedding"
+                )
+            try:
+                from langchain.embeddings import HuggingFaceEmbeddings
+            except ImportError as exc:
+                raise ImportError(
+                    "Could not import sentence_transformers or langchain package. "
+                    "Please install with `pip install sentence_transformers langchain`."
+                ) from exc
+
+            embed_model = LangchainEmbedding(HuggingFaceEmbeddings())
+
         if llama_index.global_service_context is not None:
             return cls.from_service_context(
                 llama_index.global_service_context,
@@ -132,7 +149,6 @@ def from_defaults(
         llm_predictor = llm_predictor or LLMPredictor()
         llm_predictor.callback_manager = callback_manager
 
-        # NOTE: the embed_model isn't used in all indices
         embed_model = embed_model or OpenAIEmbedding()
         embed_model.callback_manager = callback_manager
 


Code review comments:
diff_hunk: @@ -111,6 +113,21 @@ def from_defaults(
             )
             chunk_size = chunk_size_limit
 
+        if isinstance(embed_model, str):
+            if embed_model != "local":
+                raise ValueError(
+                    "embed_model must be str 'local' or of type BaseEmbedding"
+                )
+            try:
+                from langchain.embeddings import HuggingFaceEmbeddings
+            except ImportError as exc:
+                raise ImportError(
+                    "Could not import sentence_transformers or langchain package. "
+                    "Please install with `pip install sentence_transformers langchain`."
+                ) from exc
+
+            embed_model = LangchainEmbedding(HuggingFaceEmbeddings())
path: llama_index/indices/service_context.py
text: @user1:
Can we explicitly put the model name here? Just so that any default changes in langchain don't catch us off-gaurd

@author:
sounds good. We can make it a const `DEFAULT_HUGGINGFACE_EMBEDDING_MODEL`

@author:
Let's use `all-MiniLM-L6-v2 ` as the default due to the higher throughput (more than 5x). See: https://www.sbert.net/docs/pretrained_models.html

diff_hunk: @@ -111,6 +113,21 @@ def from_defaults(
             )
             chunk_size = chunk_size_limit
 
+        if isinstance(embed_model, str):
+            if embed_model != "local":
+                raise ValueError(
+                    "embed_model must be str 'local' or of type BaseEmbedding"
+                )
+            try:
+                from langchain.embeddings import HuggingFaceEmbeddings
+            except ImportError as exc:
+                raise ImportError(
+                    "Could not import sentence_transformers or langchain package. "
+                    "Please install with `pip install sentence_transformers langchain`."
path: llama_index/indices/service_context.py
text: @user1:
I'm assuming sentence_transformers also includes the transformers package? I think torch might also be a dependency here

@author:
Hmm true. Turns out it's quite a long list :cry: 

`Requires: huggingface-hub, nltk, numpy, scikit-learn, scipy, sentencepiece, torch, torchvision, tqdm, transformers`

@user1:
I think most of these are cross depending 🤔 there should be a small-ish subset that ends up installing everything you need

@author:
I think these are the only 2