delayedkarma/langchain-issues huggingface dataset:
generate_queries results in list with empty inbetween generated questions. 
Labels: name: area: vector store, name: auto:bug

### Checked other resources

- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a similar question and didn't find it.

### Example Code

multiquery_llm_retriever = MultiQueryRetriever.from_llm(
            retriever = vectordb.as_retriever(),
            llm=llm,
            include_original = top_k_params["mq_include_original"]
        )
        docs = multiquery_llm_retriever.get_relevant_documents(query=user_input)

Response of the underlying generate_queries function:
{'question': 'The actual question', 'text': LineList(lines=['Alternative Question 1: First generated question?', '', 'Alternative Question 2: Second generated question?', '', 'Alternative Question 3: Third generated question?'])}

Notice that in between are empty strings: ''
They later cause an error in similarity search since empty strings cannot be embedded. Consequently, the retrieve_documents function fails.

### Description

I am trying to use the MultiQueryRetriever. However, the generate queries function returns a list that contains empty strings in between the 3 (as in the default prompt) generated answers. 
This will later cause the retrieve_documents to fail since the similarity search throws an exception. 


### System Info

langchain==0.0.348
langchain-community==0.0.2
langchain-core==0.1.0
Python 3.11.6

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async
community: fix the "page" mode in the AzureAIDocumentIntelligenceParser (bug)
Labels: name: lgtm, name: area: doc loader, name: auto:bug, name: size:XS

**Description**: the "page" mode in the AzureAIDocumentIntelligenceParser is not accessible due to a wrong membership test. The mode argument can only be a string (also see the assertion in the `__init__`:  `assert self.mode in ["single", "page", "object", "markdown"]`, so the check `elif self.mode == ["page"]:` always fails. 
As a result, effectively the "object" mode is used when selecting the "page" mode, which may lead to errors.

The docstring of the `AzureAIDocumentIntelligenceLoader` also ommitted the `mode` parameter alltogether, so I added it.

**Issue**: I could not find a related issue (this class is only 3 weeks old anyways)

**Dependencies**: this PR does not introduce or affect dependencies.

The current demo notebook and examples are not affected because they all use the default markdown mode.
Input or output validation errors are not reported to run manager
Labels: name: auto:bug

### Checked other resources

- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a similar question and didn't find it.

### Example Code

```python
prompt_template = ("Human: Say hello to the user using the given username."
                   "Username: {username}")

chain = LLMChain(
            llm=llm,
            prompt=PromptTemplate(template=prompt_template, input_variables=[]),
            verbose=True,
        )

def execute(args:dict):
    with callbacks.collect_runs() as cb:
        try:
            output = chain.invoke(args)
            print(f"output: {output}")
        finally:
            print(f"traced_runs: {cb.traced_runs}")


# executes successfully
execute({"username": "John"})

# failed with validation error, but traced runs is empty, while it should have one run with exception details
execute({"user_name": "John"})

``

### Description

Callback manager doesn't get invoked if there's an input or output validation error. 
For logging purposes the callback manager should be invoked and any chain related error, even if it's a validation error.
The related logic exists in file `langchain/chains/base.py` in `acall` and `__call__` functions, as the `callback_manager` & `run_manager` gets initialized after validating input. To fix this, `run_manager` should be initialized first, then wrap preparing input, output and executing in try/except and update the `run_manager` accordingly

### System Info

langchain==0.0.354
langchain-community==0.0.11
langchain-core==0.1.9

Python 3.10.13

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async
Add documentation for AstraDBStore
Labels: name: auto:documentation, name: size:XS

<!-- Thank you for contributing to LangChain!

Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified.

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes if applicable,
  - **Dependencies:** any dependencies required for this change,
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/docs/integrations` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Preview: https://langchain-git-fork-cbornet-astradb-store-doc-langchain.vercel.app/docs/integrations/stores/astradb
OpenAIAssistantRunnable raise exception when pass client
Labels: name: area: agent, name: auto:bug

### Checked other resources

- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a similar question and didn't find it.

### Example Code

I want to create an OpenAI assistant using an existing ID, but I'm encountering an error.

1. assistant = OpenAIAssistantRunnable(assistant_id='assistant_id', as_agent=True)
`openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable`
2. assistant = OpenAIAssistantRunnable(assistant_id='assistant_id', as_agent=True, client=openai.OpenAI(api_key=self.model.openai_api_key))
`pydantic.v1.errors.ConfigError: field "client" not yet prepared, so the type is still a ForwardRef. You might need to call OpenAIAssistantRunnable.update_forward_refs()`

In both cases, an error is encountered. The only successful scenario is when the openai_api_key is set in the environment, but my requirement is for the user to provide it. Has anyone else faced a similar issue? Any assistance would be appreciated.

### Description

OpenAIAssistantRunnable raise exception when pass client. There should be a way to pass the key from outside

### System Info

langchain==0.1.0

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async
is Retreival QA returning True result when I am implementing RAG
Labels: name: area: memory, name: auto:bug, name: integration: chroma

qa = RetrievalQA.from_chain_type(llm=llm,
                                       chain_type='stuff',
                                       retriever=retriever,
                                       return_source_documents=True,
                                       chain_type_kwargs={'prompt': prompt},
                                       memory=memory,
                                       )
 print(qa)
below is my result when I am printing qa , I am implementing RAG
memory=ConversationBufferMemory(output_key='result', input_key='query', return_messages=True, memory_key='chat_history') combine_documents_chain=StuffDocumentsChain(llm_chain=LLMChain(prompt=PromptTemplate(input_variables=['context', 'question'], template="Use the following pieces of information to answer the user's question.\n    If you don't know the answer, just say that you don't know, don't try to make up an answer.\n    Context: {context}\n    Question: {question}\n    Only return the helpful answer below and nothing else.\n    Helpful answer:\n    "), llm=ChatOpenAI(client=<class 'openai.api_resources.chat_completion.ChatCompletion'>, model_name='gpt-3.5-turbo-16k', temperature=0.1, openai_api_key='sk-??', openai_proxy='')), document_variable_name='context') return_source_documents=True retriever=VectorStoreRetriever(tags=['Chroma', 'OpenAIEmbeddings'], vectorstore=<langchain_community.vectorstores.chroma.Chroma object at 0x7fbc6a8bfd60>, search_kwargs={'k': 2})
docs: Update Robocorp Action Server installation instructions
Labels: name: lgtm, name: auto:documentation, name: size:M

**Description:**

Remove section on how to install Action Server and direct the users t o the instructions on Robocorp repository.

**Reason:**

Robocorp Action Server has moved from a pip installation to a standalone cli application and is due for changes. Because of that, leaving only LangChain integration relevant part in the documentation.
Community: Adding asynchronous function implementation for Doctran
Labels: name: lgtm, name: area: doc loader, name: auto:bug, name: size:S

## Description 
In this update, I addressed the missing implementation for atransform_document, which is the asynchronous counterpart of transform_document in Doctran.

### Usage Example:
```py
# Instantiate DoctranPropertyExtractor with specified properties
property_extractor = DoctranPropertyExtractor(properties=properties)

# Asynchronously extract properties from a list of documents
extracted_document = await property_extractor.atransform_documents(
    documents, properties=properties
)

# Display metadata of the first extracted document
print(json.dumps(extracted_document[0].metadata, indent=2))

```

## Issue
- Pull request #14525 has caused a break in the aforementioned code. Instead of removing an asynchronous implementation of a function, consider implementing a synchronous version alongside it.
getting error while implementing memory
Labels: name: area: memory, name: auto:bug, name: integration: chroma

<code>
def set_custom_prompt():
    prompt_template = """Use the following pieces of information to answer the user's question.
    If you don't know the answer, just say that you don't know, don't try to make up an answer.
    Context: {context}
    Question: {question}
    Only return the helpful answer below and nothing else.
    Helpful answer:
    """
    prompt = PromptTemplate(template=prompt_template, input_variables=["context", "question"])
    return prompt
def retreival_qa_chain(chroma_db_path):
    prompt = set_custom_prompt() 
    embedding = OpenAIEmbeddings()
    vectordb = Chroma(persist_directory=chroma_db_path, embedding_function=embedding)
    llm = ChatOpenAI(model="gpt-3.5-turbo-16k",temperature=0.1)
    retriever = vectordb.as_retriever(search_kwargs={"k": 2})
    memory = ConversationTokenBufferMemory(llm=llm, memory_key="chat_history", return_messages=True, input_key='question', output_key='answer')
    qa = RetrievalQA.from_chain_type(llm=llm,
                                       chain_type='stuff',
                                       retriever=retriever,
                                       return_source_documents=True,
                                       chain_type_kwargs={'prompt': prompt},
                                       memory=memory,
                                       )
    # qa = RetrievalQA.from_chain_type(llm=llm, chain_type="stuff", retriever=retriever,return_source_documents=True)
    return qa
def create_global_qa_chain():
    chroma_db_path = "chroma-databases"
    folders = os.listdir(chroma_db_path)
    qa_chains = {}
    for index, folder in enumerate(folders):
        folder_path = f"{chroma_db_path}/{folder}"
        project = retreival_qa_chain(folder_path)
        qa_chains[folder] = project
    return qa_chains
def chat_langchain(new_project_qa, query):
    bot_response = new_project_qa({"query": query})
    result = bot_response['result']
   return result
qa_chains = create_global_qa_chain()
</code>
I am getting below error
    self.memory.save_context(inputs, outputs)
  File "/home/hs/env/lib/python3.8/site-packages/langchain/memory/token_buffer.py", line 51, in save_context
    super().save_context(inputs, outputs)
  File "/home/hs/env/lib/python3.8/site-packages/langchain/memory/chat_memory.py", line 37, in save_context
    input_str, output_str = self._get_input_output(inputs, outputs)
  File "/home/hs/env/lib/python3.8/site-packages/langchain/memory/chat_memory.py", line 33, in _get_input_output
    return inputs[prompt_input_key], outputs[output_key]
KeyError: 'question'
jaguar vector store reports error when adding integer metadata field
Labels: name: area: vector store, name: auto:bug

### Checked other resources

- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a similar question and didn't find it.

### Example Code

In langchain_community/vectorstores/jaguar.py line 196:
 values_comma = "'" + "','".join(vvec) + "'"

If any element in vvec is not string, runtime error is thrown:
TypeError: sequence item 0: expected str instance, int found
The vvec is a list of values in metadata dictionary.


### Description

* Some document loaders, for instance, PyPDFDirectoryLoader, add an integer page number to the metadata. In such cases, the list of values contain integer elements.  The statement "','".join(vvec) will report an error. 

### System Info

langchain==0.1.0
langchain-community==0.0.11
langchain-core==0.1.10


### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async
community: bug fix - replace image uris with b64_texts
Labels: name: area: vector store, name: auto:bug, name: integration: chroma, name: size:XS

  - **Description:** Fix an issue where the image path (uris) were inserted instead of the b64 texts when image metadata is present. In response to https://github.com/langchain-ai/langchain/issues/15935
  - **Issue:** Returns b64 text instead of image uris when doing a similarity search with multimodal embeddings
b64 strings not inserted for Chroma 'add_images' method when metadata is present
Labels: name: area: vector store, name: auto:bug, name: integration: chroma

### Checked other resources

- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a similar question and didn't find it.

### Example Code

```            if non_empty_ids:
                metadatas = [metadatas[idx] for idx in non_empty_ids]
                images_with_metadatas = [uris[idx] for idx in non_empty_ids]
                embeddings_with_metadatas = (
                    [embeddings[idx] for idx in non_empty_ids] if embeddings else None
                )
                ids_with_metadata = [ids[idx] for idx in non_empty_ids]
                try:
                    self._collection.upsert(
                        metadatas=metadatas,
                        embeddings=embeddings_with_metadatas,
                        documents=images_with_metadatas,
                        ids=ids_with_metadata,
```

should be

```            if non_empty_ids:
                metadatas = [metadatas[idx] for idx in non_empty_ids]
                images_with_metadatas = [b64_texts[idx] for idx in non_empty_ids]
                embeddings_with_metadatas = (
                    [embeddings[idx] for idx in non_empty_ids] if embeddings else None
                )
                ids_with_metadata = [ids[idx] for idx in non_empty_ids]
                try:
                    self._collection.upsert(
                        metadatas=metadatas,
                        embeddings=embeddings_with_metadatas,
                        documents=images_with_metadatas,
                        ids=ids_with_metadata,
```

### Description

When inserting images with metadata (Chroma), the 'add_images' function does not insert the b64 strings.

### System Info

Python = 3.10
Langchain = 0.1.0

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async
langchain_google_vertexai: Add support for SystemMessage for Gemini chat model
Labels: name: auto:improvement, name: area: models, name: size:L, name: partner

- **Description:** In Google Vertex AI, Gemini Chat models currently doesn't have a support for SystemMessage. This PR adds support for it only if a user provides additional convert_system_message_to_human flag during model initialization (in this case, SystemMessage would be prepended to the first HumanMessage). **NOTE:** The implementation is similar to #14824 


- **Twitter handle:** rajesh_thallam
Community: Basic Logging and Human input to ShellTool
Labels: name: area: agent, name: lgtm, name: needs test, name: auto:improvement, name: size:M

  - **Description:** As Shell tool is very versatile, while integrating it into applications as openai functions, developers have no clue about what command is being executed using the ShellTool. All one can see is:
![image](https://github.com/langchain-ai/langchain/assets/60742358/540e274a-debc-4564-9027-046b91424df3)

Summarising my feature request:
1. There's no visibility about what command was executed.
2. There's no mechanism to prevent a command to be executed using ShellTool, like a y/n human input which can be accepted from user to proceed with executing the command., 
  - **Issue:** the issue #15931 it fixes if applicable,
  - **Dependencies:** There isn't any dependancy,
  - **Twitter handle:** @krishnashed
Adding more capabilities to Langchain Shell Tool
Labels: name: auto:improvement, name: area: models

### Feature request

The tool I'm talking about is:
from langchain.tools import ShellTool

As Shell tool is very versatile, while integrating it into applications as openai functions, developers have no clue about what command is being executed using the ShellTool. All one can see is:
![image](https://github.com/langchain-ai/langchain/assets/60742358/540e274a-debc-4564-9027-046b91424df3)

Which shouldn't be the case right ? Developers need to know what command is going to be executed by the ShellTool, and whether its the right command to execute, or you just prompt better for the LLM to generate a new command and let that execute. 

Summarising my feature request:
1. There's no visibility about what command was executed.
2. There's no mechanism to prevent a command to be executed using ShellTool, like a y/n human input which can be accepted from user to proceed with executing the command.

I would like to implement this in a PR if the project maintainer feels this feature adds value.

### Motivation

I was using ShellTool as a OpenAI Function in Autogen, which is a multi-agent conversation framework. Where in one of my AI Agent is a programmer, and it has access to the ShellTool. When its executing any command  using ShellTool, we have no clue about what command is being executed, and are thrown to mercy of the LLM that the generated command is what we need. 

### Your contribution

I have raised a [PR](https://github.com/langchain-ai/langchain/pull/15932), would like you to review and provide feedback on it. Or if there's some other way around to implement the feature I'm proposing, please let me know.
i am working on SQL Langchain to ask natural english questions and get answers from postgress sql but not getting any answers
Labels: name: area: agent, name: auto:question

### Discussed in https://github.com/langchain-ai/langchain/discussions/15928

<div type='discussions-op-text'>

<sup>Originally posted by **BalajiPolisetty2207** January 11, 2024</sup>
i am using this langchain to query my postgres sql with natural language.

with sql server i am successful in asking and getting answers . 

I am using azure open ai key and endpoint and gpt 4 model and deployment .

Can some one help what is missing here to query postgressql and how i can set the schema before querying.
Below is the code 

# Importing necessary libraries
from sqlalchemy import create_engine, text
from langchain.chat_models import AzureChatOpenAI
from langchain.prompts.chat import ChatPromptTemplate
from langchain.agents import AgentType, create_sql_agent
from langchain.sql_database import SQLDatabase
from langchain.agents.agent_toolkits.sql.toolkit import SQLDatabaseToolkit
import os


db_url = 'postgresql://asdas:e?Yasdad@asdasd-qwewqe.qwewqe.database.azure.com:5432/asdad'  # PostgreSQL connection
openai_api_key = 'asdsadad'  # Azure OpenAI key
openai_model = 'eqwewqe-4'  # OpenAI model, like GPT-4
openai_deployment = 'eqwewqe'  # OpenAI deployment name
os.environ["OPENAI_API_KEY"]  =  'xxxx'
os.environ["OPENAI_API_BASE"]="https://asdasd.qweqw.azure.com/"
os.environ["OPENAI_API_TYPE"]="azure"
os.environ["OPENAI_API_VERSION"]="2023-07-01-preview"
# Creating the database engine
engine = create_engine(db_url)

# Setting up the Azure OpenAI for LangChain
llm = AzureChatOpenAI(
    model=openai_model, 
    deployment_name=openai_deployment, 
    temperature=0
)

# Setting up the prompt template
final_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", 
         """
          You are an AI assistant expert in querying the PostgreSQL database. 
          you are going to respond the question on the students schema.
         """
         ),
        ("user", "{question}\n ai: "),
    ]
)

# Creating the SQL Database object for LangChain
db = SQLDatabase(engine)

# Setting up the SQL Toolkit for LangChain
sql_toolkit = SQLDatabaseToolkit(db=db, llm=llm)
sql_toolkit.get_tools()

# Creating the SQL Agent for LangChain
sqldb_agent = create_sql_agent(
    llm=llm,
    toolkit=sql_toolkit,
    agent_type=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    verbose=True,
    handle_parsing_errors=True,
)

# Example usage of the agent to answer a question
question = "get me list of tables in the students schema ?"
final_prompt.messages[-1] = ("user", f"{question}\n ai: ")
response = sqldb_agent.run(final_prompt)
print(response)
</div>