ConversationalRetrievalChain having trouble with vector store retriever 'as_retriever()' 
Labels: name: area: memory, name: area: embeddings, name: area: vector store, name: auto:bug, name: area: models

### System Info

Code snapshot: 
![image](https://github.com/langchain-ai/langchain/assets/32274469/bb5df33f-9854-48bf-b97c-85253d8d01be)

Data in Azure Cognitive Search Vector Store: 
data ={
 	"id": " ",
 	"content": "",
 	"content_vector": [],
 	"metadata": "{}",
 	"a": "",
 	"b": "",
 	"c": 20.4,
 	"d": ""
 }



Issue:  with langchain==0.0.261, but it is working fine with langchain==0.0.242
Issue Description : SerializationError: (', DeserializationError: (", AttributeError: \'float\' object has no attribute \'lower\'", \'Unable to deserialize to object: type\', AttributeError("\'float\' object has no attribute \'lower\'"))', 'Unable to build a model: (", AttributeError: \'float\' object has no attribute \'lower\'", \'Unable to deserialize to object: type\', AttributeError("\'float\' object has no attribute \'lower\'"))', DeserializationError(", AttributeError: 'float' object has no attribute 'lower'", 'Unable to deserialize to object: type', AttributeError("'float' object has no attribute 'lower'")))

Note: In the upgraded version the response time also increases. 

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [X] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [X] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [X] Callbacks/Tracing
- [ ] Async

### Reproduction

def __llm_client(self, streaming: bool =False):
        # LLM connection
        if streaming is False:
            return AzureOpenAI(
                deployment_name=AZURE_OPENAI_LLM_MODEL_DEPLOYMENT_NAME,
                model=AZURE_OPENAI_LLM_MODEL_NAME,
                openai_api_type=self.__azure_openAI_type,
                openai_api_version=self.__azure_openAI_version,
                openai_api_base=self.__azure_openAI_base,
                openai_api_key=self.__azure_openAI_key,
                temperature=0.0,
                max_tokens=2000
            )
        return AzureOpenAI(
                deployment_name=AZURE_OPENAI_LLM_MODEL_DEPLOYMENT_NAME,
                model=AZURE_OPENAI_LLM_MODEL_NAME,
                openai_api_type=self.__azure_openAI_type,
                openai_api_version=self.__azure_openAI_version,
                openai_api_base=self.__azure_openAI_base,
                openai_api_key=self.__azure_openAI_key,
                temperature=0.0,
                max_tokens=2000,
                streaming=True, 
                callbacks=[StreamingStdOutCallbackHandler()]
            )
    
    # LLM Embeddings Client
    def __embeddings_client(self):
        return OpenAIEmbeddings(
            model=AZURE_OPENAI_LLM_EMBEDDING_NAME,
            deployment=AZURE_OPENAI_LLM_EMBEDDING_DEPLOYMENT_NAME,
            openai_api_type=self.__azure_openAI_type,
            openai_api_version=self.__azure_openAI_version,
            openai_api_base=self.__azure_openAI_base,
            openai_api_key=self.__azure_openAI_key,
            chunk_size=1536
        )
    
    # Embedding vector store client
    def __vector_store_client(self):
        acs_vector_store: AzureSearch = AzureSearch(
            azure_search_endpoint=self.__acs_endpoint,
            azure_search_key=self.__acs_key,
            index_name=self.__acs_index_name,
            embedding_function=self.__embeddings_client().embed_query,
        )
        return acs_vector_store
    
    # Langchain Chain Client
    def __chain_client(self):
        chain_type = "stuff"
        return load_qa_chain(
            llm=self.__llm_client(streaming=True), 
            chain_type=chain_type,
            document_variable_name='context',
            prompt=QA_PROMPT
            # verbose=True
            )
    
    def __conversational_retrieval_chain_client(self):
        # print(self.__vector_store_client().as_retriever())
        return ConversationalRetrievalChain(
            retriever=self.__vector_store_client().as_retriever(),
            question_generator=self.__question_generator(),
            combine_docs_chain=self.__chain_client(),
            memory=ConversationBufferMemory(
                memory_key="chat_history",
                return_messages=False
            )
        )

    def __question_generator(self):
        return LLMChain(
            llm=self.__llm_client(),
            prompt=CONDENSE_QUESTION_PROMPT
        )
    
    # Main function
    def smart_chat_bot(self, query: str="*", conversation: list=[]):
        self.user_input = query
        print(f"Human Input: {self.user_input}", end="\n")
        result = self.__conversational_retrieval_chain_client()({"question": self.user_input, "chat_history": conversation})
        return result

### Expected behavior

It should return the response on the bases of user input and the documents that came from the vector store. And the latency will be lesser.