RAG crash: TypeError: Type is not JSON serializable: numpy.ndarray
Labels: name: auto:bug, name: integration: chroma, name: area: langserve

### System Info

- Python 3.12.1
- MacOS 14.2.1
- langchain-cli 0.0.20 from pip OR langchain-* from git master branch (commit-ish [26f84b7](https://github.com/langchain-ai/langchain/commit/26f84b74d0f7dc4d2211a1a62d47eec36cb1d726)) -- can reproduce with latest code: langchain 0.0.353, langchain-cli 0.0.20, langchain-community 0.0.7, langchain-core 0.1.4
- pandas 2.1.4
- lancedb 0.4.3
- numpy 1.26.2

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I started with the rag-chroma-private quickstart, then modified it to my needs. I got rid of Chroma due to errors and now use a persistent LanceDB file.

When I try to enter a prompt on the playground, I get a crash (see the bottom of this message).

server.py:
```
from fastapi import FastAPI
from fastapi.responses import RedirectResponse
from langserve import add_routes
from app.chain import chain as rag_private_chain

app = FastAPI()


@app.get("/")
async def redirect_root_to_docs():
    return RedirectResponse("/docs")


add_routes(app, rag_private_chain, path="/rag-lancedb-private")

if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app, host="0.0.0.0", port=8000)

```

chain.py:
```
# Load
from typing import List
from langchain.chat_models import ChatOllama
from langchain.document_loaders import UnstructuredMarkdownLoader
from langchain.embeddings import OllamaEmbeddings
from langchain.prompts import ChatPromptTemplate
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_core.output_parsers import StrOutputParser
from langchain_core.pydantic_v1 import BaseModel
from langchain_core.runnables import RunnableParallel, RunnablePassthrough
from langchain_core.documents.base import Document
import os, sys

#DB
import lancedb
from langchain.vectorstores.lancedb import LanceDB

#Chroma
#from langchain.vectorstores.chroma import Chroma
#from langchain.vectorstores.utils import filter_complex_metadata

files_directory = "/Users/sean/Downloads/posts"

all_splits = None
my_emb = OllamaEmbeddings(model="llama2:70b-chat")
table = None
db = lancedb.connect("./lance.db")
vectorstore = None

#Generating the vectors only has to be done when the data changes
def do_loader():
    #List each file in files_directory and loop through them creating an UnstructuredMarkdownLoader for each file
    all_splits = []
    for filename in os.listdir(files_directory):
        full_path = os.path.join(files_directory, filename)
        loader = UnstructuredMarkdownLoader(full_path, mode="elements", strategy="hi_res")
        data: List[Document] = loader.load()
        all_splits.extend(data)
        continue

    print(f"Got {len(all_splits)} documents")

    table = db.create_table(
        "rag",
        data=[
            {
                "vector": my_emb.embed_query("Hello World"),
                "text": "Hello World",
                "id": "1",
            }
        ],
        mode="overwrite",
    )

    #New 
    vectorstore = LanceDB.from_documents(all_splits, my_emb, connection=table)
    print("Added docs to vector store")

#Not calling do_loader() because I have already populated LanceDB with the vectors.
table = db.open_table("rag")
vectorstore = LanceDB(connection=table, embedding=my_emb)
retriever = vectorstore.as_retriever()

print("Loaded DB")
 
# Prompt
# Optionally, pull from the Hub
# from langchain import hub
# prompt = hub.pull("rlm/rag-prompt")
# Or, define your own:
template = """Answer the question based only on the following context:
{context}

Question: {question}
"""
prompt = ChatPromptTemplate.from_template(template)

# LLM
ollama_llm = "llama2:70b-chat"
model = ChatOllama(model=ollama_llm)

print("Created Ollama model")

# RAG chain
chain = (
    RunnableParallel({"context": retriever, "question": RunnablePassthrough()})
    | prompt
    | model
    | StrOutputParser()
)

# Add typing for input
class Question(BaseModel):
    __root__: str

chain = chain.with_types(input_type=Question)

print("Done with chain.py")
```

Error:

```

INFO:     Application startup complete.
INFO:     127.0.0.1:64594 - "GET /rag-chroma-private/playground/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:64594 - "GET /rag-chroma-private/playground/assets/index-52e8ab2f.css HTTP/1.1" 200 OK
INFO:     127.0.0.1:64595 - "GET /rag-chroma-private/playground/assets/index-6a0f524c.js HTTP/1.1" 200 OK
INFO:     127.0.0.1:64595 - "GET /rag-chroma-private/playground/favicon.ico HTTP/1.1" 200 OK
INFO:     127.0.0.1:64594 - "POST /rag-chroma-private/stream_log HTTP/1.1" 200 OK
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/sse_starlette/sse.py", line 269, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/sse_starlette/sse.py", line 258, in wrap
    await func()
  File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/sse_starlette/sse.py", line 215, in listen_for_disconnect
    message = await receive()
              ^^^^^^^^^^^^^^^
  File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 538, in receive
    await self.message_event.wait()
  File "/opt/homebrew/Cellar/python@3.12/3.12.1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/locks.py", line 212, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 15bc21f40

During handling of the above exception, another exception occurred:

  + Exception Group Traceback (most recent call last):
  |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
  |     result = await app(  # type: ignore[func-returns-value]
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
  |     return await self.app(scope, receive, send)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
  |     await super().__call__(scope, receive, send)
  |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/starlette/applications.py", line 116, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
  |     raise exc
  |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
  |     await self.app(scope, receive, _send)
  |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
  |     raise exc
  |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/starlette/routing.py", line 754, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/starlette/routing.py", line 774, in app
  |     await route.handle(scope, receive, send)
  |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/starlette/routing.py", line 296, in handle
  |     await self.app(scope, receive, send)
  |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/starlette/routing.py", line 75, in app
  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
  |     raise exc
  |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
  |     await response(scope, receive, send)
  |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/sse_starlette/sse.py", line 255, in __call__
  |     async with anyio.create_task_group() as task_group:
  |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 678, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/langserve/serialization.py", line 90, in default
    |     return super().default(obj)
    |            ^^^^^^^
    | RuntimeError: super(): __class__ cell not found
    |
    | The above exception was the direct cause of the following exception:
    |
    | Traceback (most recent call last):
    |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/sse_starlette/sse.py", line 258, in wrap
    |     await func()
    |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/sse_starlette/sse.py", line 245, in stream_response
    |     async for data in self.body_iterator:
    |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/langserve/api_handler.py", line 1049, in _stream_log
    |     "data": self._serializer.dumps(data).decode("utf-8"),
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/sean/dev/cc/cc/ccvenv/lib/python3.12/site-packages/langserve/serialization.py", line 168, in dumps
    |     return orjson.dumps(obj, default=default)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    | TypeError: Type is not JSON serializable: numpy.ndarray
    +------------------------------------
```

### Expected behavior

It's not clear to me which part of the stack is failing, but the expected behavior is that the LLM should generate some output rather than crashing.