[SelfQueryRetriever] Generated Query Mismatched Timestamp Type
Labels: name: area: vector store, name: auto:bug, name: auto:improvement, name: integration: chroma

### System Info

langchain==0.0.266

### Who can help?

@eyurtsev @hwchase17 

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```python
import datetime
import chainlit
from dotenv import load_dotenv
from langchain.chains.query_constructor.base import AttributeInfo
from langchain.chat_models import ChatOpenAI
from langchain.docstore.document import Document  # noqa
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.retrievers import SelfQueryRetriever
from langchain.vectorstores import Chroma

chainlit.debug = True

load_dotenv()

llm = ChatOpenAI()

docs = [
    Document(
        page_content="A bunch of scientists bring back dinosaurs and mayhem breaks loose",
        metadata={"released_at": 1700190868, "rating": 7.7, "genre": "science fiction"},
    ),
    Document(
        page_content="Leo DiCaprio gets lost in a dream within a dream within a dream within a ...",
        metadata={"released_at": 1700190868, "director": "Christopher Nolan", "rating": 8.2},
    ),
    Document(
        page_content="A psychologist / detective gets lost in a series of dreams within dreams within dreams and Inception reused the idea",
        metadata={"released_at": 1700190868, "director": "Satoshi Kon", "rating": 8.6},
    ),
    Document(
        page_content="A bunch of normal-sized women are supremely wholesome and some men pine after them",
        metadata={"released_at": 1700190868, "director": "Greta Gerwig", "rating": 8.3},
    ),
    Document(
        page_content="Toys come alive and have a blast doing so",
        metadata={"released_at": 1700190868, "genre": "animated"},
    ),
    Document(
        page_content="Three men walk into the Zone, three men walk out of the Zone",
        metadata={"released_at": 1700190868, "director": "Andrei Tarkovsky", "genre": "thriller", "rating": 9.9},
    ),
]
vectorstore = Chroma.from_documents(docs, OpenAIEmbeddings())

metadata_field_info = [
    AttributeInfo(
        name="released_at",
        description="Time the movie was released. It's second timestamp.",
        type="integer",
    ),
]
document_content_description = "Brief summary of a movie"
retriever = SelfQueryRetriever.from_llm(
    llm,
    vectorstore,
    document_content_description,
    metadata_field_info,
)

result = retriever.invoke(
    f"What's a movie in this month that's all about toys, and preferably is animated. Current time is: {datetime.datetime.now().strftime('%m/%d/%Y, %H:%M:%S')}.",
)

print(result)

```

### Expected behavior

I declared the `metadata_field_info` which includes the `released_at` field with the data type `integer`.

## I expected the following:

When my query involves the time/timerange of the release time, the query should compare using `integer` instead of `date time`.

### Why I expected this:

- The data type declared in `metadata_field_info` should be utilized.
- In the implementations of `SelfQueryRetriever` (I tested `qdrant` and `chroma`), the accepted type in comparison operations (gte/lte) must be numeric, not a date.

### Identified Reason

I identified the problem due to the `"SCHEMA[s]"` in [langchain/chains/query_constructor/prompt.py](https://github.com/langchain-ai/langchain/blob/190952fe76d8f7bf1e661cbdaa2ba0a2dc0f5456/libs/langchain/langchain/chains/query_constructor/prompt.py#L117). 

This line in prompt led the result:
```
Make sure that filters only use format `YYYY-MM-DD` when handling date data typed values
```

I guess that it works in some SQL queries such as `Postgresql`, which accepts 'YYY-MM-DD' as date query inputs.

However, we are working with metadata in vector records, which are structured like JSON objects with key-value pairs, it may not work.

### Proof of reason

I tryed modifing the PROMPTs by defining my own Classes and Functions such as `load_query_constructor_chain`, `_get_prompt`, `SelfQueryRetriever`.

After replacing the above line with the following, it worked as expected:

```
Make sure that filters only use timestamp in second (integer) when handling timestamp data typed values.
```

### Proposals
- Review the above problem. If metadata fields do not support querying with the date format 'YYYY-MM-DD' as specified in the prompt, please update it.

- If this prompt is specified for some use cases, please allow overriding the prompts.