SelfQueryRetriever creates attributes that do not exist
Labels: name: area: vector store, name: auto:bug, name: area: models

### System Info

langchain==0.0.191
llama-cpp-python==0.1.78
chromadb==0.3.22
python3.10
wizard-vicuna-13B.ggmlv3.q4_0.bin

### Who can help?

@hwchase17 @agol

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I have a slightly modified version of privateGPT running. I am facing a weird issue where the SelfQueryRetriever is creating attributes that do not exist in my ChromaDB. This is crashing the app when running the RetrievalQA with error `chromadb.errors.NoDatapointsException: No datapoints found for the supplied filter`. I have provided a list of the attributes that exists in my DB, but still the SelfQueryRetreiever is creating filters on metadata that does not exist.

To reproduce the problem, use the wizard-vicuna-13B.ggmlv3.q4_0.bin model provided by TheBloke/wizard-vicuna-13B-GGML on HuggingFace and run the below code. I don't think the choice of model has an impact here. The issue I am facing is the creation of metadata filters that do not exist. 

Is there a way to limit the attributes allowed by the SelfQueryRetriever?

```python
import logging

import click
import torch
from auto_gptq import AutoGPTQForCausalLM
from huggingface_hub import hf_hub_download
from langchain.chains import RetrievalQA
from langchain.embeddings import HuggingFaceInstructEmbeddings
from langchain.llms import HuggingFacePipeline, LlamaCpp
from langchain.memory import ConversationBufferMemory
from langchain.prompts import PromptTemplate
from langchain.retrievers.self_query.base import SelfQueryRetriever
from langchain.chains.query_constructor.base import AttributeInfo
import time

# from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain.vectorstores import Chroma
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    GenerationConfig,
    LlamaForCausalLM,
    LlamaTokenizer,
    pipeline,
)

from constants import CHROMA_SETTINGS, EMBEDDING_MODEL_NAME, PERSIST_DIRECTORY, MODEL_ID, MODEL_BASENAME
SEED = 42


def load_model(device_type, model_id, model_basename=None, local_model: bool = False, local_model_path: str = None):
    """
    Select a model for text generation using the HuggingFace library.
    If you are running this for the first time, it will download a model for you.
    subsequent runs will use the model from the disk.

    Args:
        device_type (str): Type of device to use, e.g., "cuda" for GPU or "cpu" for CPU.
        model_id (str): Identifier of the model to load from HuggingFace's model hub.
        model_basename (str, optional): Basename of the model if using quantized models.
            Defaults to None.

    Returns:
        HuggingFacePipeline: A pipeline object for text generation using the loaded model.

    Raises:
        ValueError: If an unsupported model or device type is provided.
    """
    if local_model:
        logging.info(f'Loading local model at {local_model_path}')
    else:
        logging.info(f"Loading Model: {model_id}, on: {device_type}")
    logging.info("This action can take a few minutes!")

    if model_basename is not None:
        # if "ggml" in model_basename:
        if ".ggml" in model_basename:
            logging.info("Using Llamacpp for GGML quantized models")
            if local_model:
                model_path = local_model_path
            else:
                model_path = hf_hub_download(repo_id=model_id, filename=model_basename)
            max_ctx_size = 2048
            kwargs = {
                "model_path": model_path,
                "n_ctx": max_ctx_size,
                "max_tokens": max_ctx_size,
            }
            if device_type.lower() == "mps":
                kwargs["n_gpu_layers"] = 1000
            if device_type.lower() == "cuda":
                kwargs['seed'] = SEED
                kwargs["n_gpu_layers"] = 40

            return LlamaCpp(**kwargs)

        else:
            # The code supports all huggingface models that ends with GPTQ and have some variation
            # of .no-act.order or .safetensors in their HF repo.
            logging.info("Using AutoGPTQForCausalLM for quantized models")

            if ".safetensors" in model_basename:
                # Remove the ".safetensors" ending if present
                model_basename = model_basename.replace(".safetensors", "")

            tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)
            logging.info("Tokenizer loaded")

            model = AutoGPTQForCausalLM.from_quantized(
                model_id,
                model_basename=model_basename,
                use_safetensors=True,
                trust_remote_code=True,
                device="cuda:0",
                use_triton=False,
                quantize_config=None,
            )
    elif (
        device_type.lower() == "cuda"
    ):  # The code supports all huggingface models that ends with -HF or which have a .bin
        # file in their HF repo.
        logging.info("Using AutoModelForCausalLM for full models")
        tokenizer = AutoTokenizer.from_pretrained(model_id)
        logging.info("Tokenizer loaded")

        model = AutoModelForCausalLM.from_pretrained(
            model_id,
            device_map="auto",
            torch_dtype=torch.float16,
            low_cpu_mem_usage=True,
            trust_remote_code=True,
            # max_memory={0: "15GB"} # Uncomment this line with you encounter CUDA out of memory errors
        )
        model.tie_weights()
    else:
        logging.info("Using LlamaTokenizer")
        tokenizer = LlamaTokenizer.from_pretrained(model_id)
        model = LlamaForCausalLM.from_pretrained(model_id)

    # Load configuration from the model to avoid warnings
    generation_config = GenerationConfig.from_pretrained(model_id)
    # see here for details:
    # https://huggingface.co/docs/transformers/
    # main_classes/text_generation#transformers.GenerationConfig.from_pretrained.returns

    # Create a pipeline for text generation
    pipe = pipeline(
        "text-generation",
        model=model,
        tokenizer=tokenizer,
        max_length=2048,
        temperature=0,
        top_p=0.95,
        repetition_penalty=1.15,
        generation_config=generation_config,
    )

    local_llm = HuggingFacePipeline(pipeline=pipe)
    logging.info("Local LLM Loaded")

    return local_llm


# chose device typ to run on as well as to show source documents.
@click.command()
@click.option(
    "--device_type",
    default="cuda" if torch.cuda.is_available() else "cpu",
    type=click.Choice(
        [
            "cpu",
            "cuda",
            "ipu",
            "xpu",
            "mkldnn",
            "opengl",
            "opencl",
            "ideep",
            "hip",
            "ve",
            "fpga",
            "ort",
            "xla",
            "lazy",
            "vulkan",
            "mps",
            "meta",
            "hpu",
            "mtia",
        ],
    ),
    help="Device to run on. (Default is cuda)",
)
@click.option(
    "--show_sources",
    "-s",
    is_flag=True,
    help="Show sources along with answers (Default is False)",
)
@click.option(
    "--local_model",
    "-lm",
    is_flag=True,
    help="Use local model (Default is False)",
)
@click.option(
    "--local_model_path",
    "-lmp",
    default=None,
    help="Path to local model. (Default is None)",
)
def main(device_type, show_sources, local_model: bool = False, local_model_path: str = None):
    """
    This function implements the information retrieval task.


    1. Loads an embedding model, can be HuggingFaceInstructEmbeddings or HuggingFaceEmbeddings
    2. Loads the existing vectorestore that was created by inget.py
    3. Loads the local LLM using load_model function - You can now set different LLMs.
    4. Setup the Question Answer retreival chain.
    5. Question answers.
    """

    logging.info(f"Running on: {device_type}")
    logging.info(f"Display Source Documents set to: {show_sources}")

    embeddings = HuggingFaceInstructEmbeddings(model_name=EMBEDDING_MODEL_NAME, model_kwargs={"device": device_type})

    # uncomment the following line if you used HuggingFaceEmbeddings in the ingest.py
    # embeddings = HuggingFaceEmbeddings(model_name=EMBEDDING_MODEL_NAME)

    # load the vectorstore
    db = Chroma(
        persist_directory=PERSIST_DIRECTORY,
        embedding_function=embeddings,
        client_settings=CHROMA_SETTINGS,
    )

    template = """Use the following pieces of context to answer the question at the end. If you don't know the answer,\
    just say that you don't know, don't try to make up an answer.

    {context}

    {history}
    Question: {question}
    Helpful Answer:"""

    prompt = PromptTemplate(input_variables=["history", "context", "question"], template=template)
    memory = ConversationBufferMemory(input_key="question", memory_key="history")

    llm = load_model(
        device_type, model_id=MODEL_ID, model_basename=MODEL_BASENAME, local_model=local_model,
        local_model_path=local_model_path)

    metadata_field_info = [
        AttributeInfo(
            name='country',
            description='The country name.',
            type='string'
        ),
        AttributeInfo(
            name='source',
            description='Filename and location of the source file.',
            type='string'
        )
    ]

    retriever = SelfQueryRetriever.from_llm(
        llm=llm,
        vectorstore=db,
        document_contents='News, policies, and laws for various countries.',
        metadata_field_info=metadata_field_info,
        verbose=True,
    )

    qa = RetrievalQA.from_chain_type(
        llm=llm,
        chain_type="stuff",
        retriever=retriever,
        return_source_documents=True,
        chain_type_kwargs={"prompt": prompt, "memory": memory},
    )
    # Interactive questions and answers
    while True:
        query = input("\nEnter a query: ")
        if query == "exit":
            break
        # Get the answer from the chain
        start = time.time()
        res = qa(query)
        answer, docs = res["result"], res["source_documents"]

        # Print the result
        print(f'Time: {time.time() - start}')
        print("\n\n> Question:")
        print(query)
        print("\n> Answer:")
        print(answer)

        if show_sources:  # this is a flag that you can set to disable showing answers.
            # # Print the relevant sources used for the answer
            print("----------------------------------SOURCE DOCUMENTS---------------------------")
            for document in docs:
                print("\n> " + document.metadata["source"] + ":")
                print(document.page_content)
            print("----------------------------------SOURCE DOCUMENTS---------------------------")


if __name__ == "__main__":
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(filename)s:%(lineno)s - %(message)s", level=logging.INFO
    )
    main()
```

```bash
2023-08-25 12:53:40,256 - INFO - run_localGPT.py:209 - Running on: cuda
2023-08-25 12:53:40,256 - INFO - run_localGPT.py:210 - Display Source Documents set to: True
2023-08-25 12:53:40,397 - INFO - SentenceTransformer.py:66 - Load pretrained SentenceTransformer: hkunlp/instructor-large
load INSTRUCTOR_Transformer
max_seq_length  512
2023-08-25 12:53:42,924 - INFO - __init__.py:88 - Running Chroma using direct local API.
2023-08-25 12:53:42,929 - WARNING - __init__.py:43 - Using embedded DuckDB with persistence: data will be stored in: /home/waleedalfaris/localGPT/DB
2023-08-25 12:53:42,934 - INFO - ctypes.py:22 - Successfully imported ClickHouse Connect C data optimizations
2023-08-25 12:53:42,937 - INFO - json_impl.py:45 - Using python library for writing JSON byte strings
2023-08-25 12:53:47,543 - INFO - duckdb.py:460 - loaded in 129337 embeddings
2023-08-25 12:53:47,545 - INFO - duckdb.py:472 - loaded in 1 collections
2023-08-25 12:53:47,546 - INFO - duckdb.py:89 - collection with name langchain already exists, returning existing collection
2023-08-25 12:53:47,546 - INFO - run_localGPT.py:50 - Loading local model at /home/waleedalfaris/localGPT/models/wizard-vicuna-13B.ggmlv3.q4_0.bin
2023-08-25 12:53:47,546 - INFO - run_localGPT.py:53 - This action can take a few minutes!
2023-08-25 12:53:47,546 - INFO - run_localGPT.py:58 - Using Llamacpp for GGML quantized models
ggml_init_cublas: found 1 CUDA devices:
  Device 0: Tesla T4, compute capability 7.5
llama.cpp: loading model from /home/waleedalfaris/localGPT/models/wizard-vicuna-13B.ggmlv3.q4_0.bin
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 2048
llama_model_load_internal: n_embd     = 5120
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 40
llama_model_load_internal: n_head_kv  = 40
llama_model_load_internal: n_layer    = 40
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: n_gqa      = 1
llama_model_load_internal: rnorm_eps  = 5.0e-06
llama_model_load_internal: n_ff       = 13824
llama_model_load_internal: freq_base  = 10000.0
llama_model_load_internal: freq_scale = 1
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: model size = 13B
llama_model_load_internal: ggml ctx size =    0.11 MB
llama_model_load_internal: using CUDA for GPU acceleration
llama_model_load_internal: mem required  =  669.91 MB (+ 1600.00 MB per state)
llama_model_load_internal: allocating batch_size x (640 kB + n_ctx x 160 B) = 480 MB VRAM for the scratch buffer
llama_model_load_internal: offloading 40 repeating layers to GPU
llama_model_load_internal: offloaded 40/43 layers to GPU
llama_model_load_internal: total VRAM used: 7288 MB
llama_new_context_with_model: kv self size  = 1600.00 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | 

Enter a query: what is the penalty for cybercrime in the United Arab Emirates?

llama_print_timings:        load time =   786.98 ms
llama_print_timings:      sample time =   158.14 ms /   196 runs   (    0.81 ms per token,  1239.39 tokens per second)
llama_print_timings: prompt eval time = 83050.84 ms /   920 tokens (   90.27 ms per token,    11.08 tokens per second)
llama_print_timings:        eval time = 22099.62 ms /   195 runs   (  113.33 ms per token,     8.82 tokens per second)
llama_print_timings:       total time = 105962.51 ms
query=' ' filter=Operation(operator=<Operator.AND: 'and'>, arguments=[Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='country', value='United Arab Emirates'), Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='penalty', value='cybercrime')]) limit=None
Traceback (most recent call last):
  File "/home/waleedalfaris/localGPT/run_localGPT.py", line 302, in <module>
    
  File "/home/waleedalfaris/localGPT/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/waleedalfaris/localGPT/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/waleedalfaris/localGPT/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/waleedalfaris/localGPT/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/waleedalfaris/localGPT/run_localGPT.py", line 279, in main
    query = input("\nEnter a query: ")
  File "/home/waleedalfaris/localGPT/venv/lib/python3.10/site-packages/langchain/chains/base.py", line 140, in __call__
    raise e
  File "/home/waleedalfaris/localGPT/venv/lib/python3.10/site-packages/langchain/chains/base.py", line 134, in __call__
    self._call(inputs, run_manager=run_manager)
  File "/home/waleedalfaris/localGPT/venv/lib/python3.10/site-packages/langchain/chains/retrieval_qa/base.py", line 119, in _call
    docs = self._get_docs(question)
  File "/home/waleedalfaris/localGPT/venv/lib/python3.10/site-packages/langchain/chains/retrieval_qa/base.py", line 181, in _get_docs
    return self.retriever.get_relevant_documents(question)
  File "/home/waleedalfaris/localGPT/venv/lib/python3.10/site-packages/langchain/retrievers/self_query/base.py", line 90, in get_relevant_documents
    docs = self.vectorstore.search(new_query, self.search_type, **search_kwargs)
  File "/home/waleedalfaris/localGPT/venv/lib/python3.10/site-packages/langchain/vectorstores/base.py", line 81, in search
    return self.similarity_search(query, **kwargs)
  File "/home/waleedalfaris/localGPT/venv/lib/python3.10/site-packages/langchain/vectorstores/chroma.py", line 182, in similarity_search
    docs_and_scores = self.similarity_search_with_score(query, k, filter=filter)
  File "/home/waleedalfaris/localGPT/venv/lib/python3.10/site-packages/langchain/vectorstores/chroma.py", line 230, in similarity_search_with_score
    results = self.__query_collection(
  File "/home/waleedalfaris/localGPT/venv/lib/python3.10/site-packages/langchain/utils.py", line 53, in wrapper
    return func(*args, **kwargs)
  File "/home/waleedalfaris/localGPT/venv/lib/python3.10/site-packages/langchain/vectorstores/chroma.py", line 121, in __query_collection
    return self._collection.query(
  File "/home/waleedalfaris/localGPT/venv/lib/python3.10/site-packages/chromadb/api/models/Collection.py", line 219, in query
    return self._client._query(
  File "/home/waleedalfaris/localGPT/venv/lib/python3.10/site-packages/chromadb/api/local.py", line 408, in _query
    uuids, distances = self._db.get_nearest_neighbors(
  File "/home/waleedalfaris/localGPT/venv/lib/python3.10/site-packages/chromadb/db/clickhouse.py", line 576, in get_nearest_neighbors
    raise NoDatapointsException(
chromadb.errors.NoDatapointsException: No datapoints found for the supplied filter {"$and": [{"country": {"$eq": "United Arab Emirates"}}, {"penalty": {"$eq": "cybercrime"}}]}
2023-08-25 13:05:24,584 - INFO - duckdb.py:414 - Persisting DB to disk, putting it in the save folder: /home/waleedalfaris/localGPT/DB
```

### Expected behavior

Result of SelfQueryRetriever should only contain filters country with a value of United Arab Emirates and query should not be blank. Should have an ouptut similar to `query='cybercrime penalty' filter=Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='country', value='United Arab Emirates') limit=None`