delayedkarma/langchain-issues huggingface dataset:
Issue: LLMChain error. response_format json error with messages. Messages is array of array
Labels: name: auto:bug, name: area: models

### Issue you'd like to raise.

Environment
```
Edition	Windows 11 Home
Version	22H2
Installed on	‎4/‎30/‎2023
OS build	22621.2861
Experience	Windows Feature Experience Pack 1000.22681.1000.0

langchain package version: "0.0.212"
zod package version: "3.22.4"
typescript package version: "5.1.6"
```

Prompt
```
My prompt data with keys: {chat_history}, {currentPoint}, {language}, {topic} and Last AI message: {lastAiMessage} and User response: {message}| format: json
```

Creating model code
```
// LLM constructor
constructor(args: any[]) {
  this.llm = new ChatOpenAI({
    openAIApiKey: process.env.OPENAI_API_KEY,
    modelName: 'gpt-3.5-turbo-1106',
    modelKwargs: {
      response_format: {
        type: 'json_object',
      },
    },
  });

  this.answerScheme = LLMChain.getAnswerScheme();
  this.formatInstructions = createParserFromSchema(this.answerScheme).getFormatInstructions();

  const prompt = ChatPromptTemplate.fromMessages([new SystemMessage(args.prompt, { "json": true })]);

  const memory = new ConversationSummaryMemory({
    llm: this.llm,
    memoryKey: 'chat_history',
    inputKey: 'message',
  });

  this.chain = new ConversationChain({
    llm: this.llm,
    prompt,
    memory,
    verbose: true,
  })
}

private static getAnswerScheme() {
  return z.object({
    answer: z.string(),
    action: z.enum(['none', 'next']),
  });
}
```

Send message code
```
async sendMessage(chainValues: ChainValues) {
  chainValues['currentPoint'] = this.currentPoint;
  chainValues['lastAiMessage'] = this.lastAiMessage ?? '';

  try {
    const modelKwargs = {
      response_format: {
        type: 'json_object',
      },
    };

    const rawResponse = await this.chain.call({ ...chainValues, ...modelKwargs, format_instructions: this.formatInstructions });
    const { answer, action } = this.answerScheme.parse(rawResponse);

    this.lastAiMessage = answer;
    this.__parseActionKeyword(action);

    return answer;
  }
  catch (error) {
    console.error('LLMChain ERROR:', error);
    return "Something goes wrong.\n\n" + error;
  }
}
```

LLM run with input
```
[llm/start] [1:chain:ConversationChain > 2:llm:ChatOpenAI] Entering LLM run with input: {
  "messages": [
    [
      {
        "lc": 1,
        "type": "constructor",
        "id": [
          "langchain_core",
          "messages",
          "SystemMessage"
        ],
        "kwargs": {
          "content": "My prompt data with keys: {chat_history}, {currentPoint}, {language}, {topic} and Last AI message: {lastAiMessage} and User response: {message}| format: json",
          "additional_kwargs": {
            "json": true
          }
        }
      }
    ]
  ]
}
```

Error: `400 'messages' must contain the word 'json' in some form, to use 'response_format' of type 'json_object'.`

### Suggestion:

_No response_
If my agent tool requires user to pass 2 parameters, and if these 2 parameters are not included in the user's question, how can I remind him to enter the parametersIssue: <Please write a comprehensive title after the 'Issue: ' prefix>
Labels: name: area: agent, name: auto:question

### Issue you'd like to raise.

If my agent tool requires user to pass 2 parameters, and if these 2 parameters are not included in the user's question, how can I remind him to enter the parameters

### Suggestion:

_No response_
Chroma.from_documents exclude metadata in embedding? [Question]
Labels: name: area: vector store, name: auto:question, name: integration: chroma

### Issue you'd like to raise.

I suspect a potential issue where Chroma.from_documents might not be embedding and storing vectors for metadata in documents.

I have loaded five tabular documents using DataFrameLoader. However, when attempting to retrieve content based on similarity from the vector store, it appears that sentences in the metadata are not being utilized for matching. I don't see the documentation have clarify if this is the expected behavior or if I might be overlooking a specific argument or setting?

To illustrate, suppose I have a table with three fields: customer_question, agent_answer, and manager_note. If I query using the exact string from one of a manager_note, it surprisingly doesn't return the corresponding document at the top of the results. 
**Is this a normal outcome? 
Should I modify my table structure to include all relevant content in the page_content_column when setting up the DataFrameLoader?**

Here is the process

```
loader = DataFrameLoader(customer_q_a_001, page_content_column='customer_question')
docs = loader2.load()

text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=15)
docs = text_splitter.split_documents(docs )


# db = Chroma.from_documents(all_docs, embeddings, persist_directory="./chroma_db")
# db.persist()

db.similarity_search_with_score((query)) 
```

Library version:

langchain: 0.0.352
I would appreciate any insights or suggestions regarding this question.

### Suggestion:

_No response_
refactored `text_splitter`
Labels: name: area: doc loader, name: auto:improvement, name: auto:refactor, name: integration: elasticsearch, name: size:XXL

All text splitters are in a single file. It is hard to extract a base classes to `langchain_core`.
- split code to several files
- added interface classes `BaseTextSplitter`, `BaseTextToDocumentsSplitter` and moved them into `core`
add support for embedding models with adapters
Labels: name: area: embeddings, name: auto:enhancement

### Feature request

It would be great to have adapters support in huggingface embedding class 

### Motivation

Many really good embedding models have special adapters for retrieval, for example specter2 which is a leading embedding for scientific texts have many adapters, for example https://huggingface.co/allenai/specter2_aug2023refresh
and current huggingface embedding implementations does not allow using them

### Your contribution

so far I am just implementing it in ugly way in my projects, not sure if/when I will have time for proper PR
community:Lazy load wikipedia dump file
Labels: name: area: doc loader, name: auto:refactor, name: size:M

**Description:** the MWDumpLoader implementation currently does not support the lazy_load method, and the files are usually very large. We are proposing refactoring the load function, extracting two private functions with the functionality of loading the dump file and parsing a single page, to reuse the code in the lazy_load implementation.
community: Add streaming support for together AI LLM integration
Labels: name: auto:enhancement, name: area: models, name: size:L

<!-- Thank you for contributing to LangChain!

Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified.

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes if applicable,
  - **Dependencies:** any dependencies required for this change,
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/docs/integrations` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
- **Description:**  Adding Streaming Support for Together AI LLM integration with the universal `streaming= True` flag 
with an updated notebook example to show the user that the model supports streaming and how it's done.

- **Issue:** implementing  #15109 
- **Dependencies:** No addtional dependencies
-  **Twitter handle:** @[YossefDawoad](https://twitter.com/YossefDawoad)
[improvement] Add Streaming Support for Together AI
Labels: name: auto:improvement, name: area: models

### Feature request

Add streaming support for Together AI Endpoints in Langchain. The official endpoint supports streaming with `stream_tokens` keyword, which should be not that hard to implement `_stream` method and add streaming support with the `streaming = True` flag
this is what the endpoint output when `stream_token` is set to `true`
```
data: {
    "choices": [{"text": " the"}],
    "request_id": "83a18448f8c030ab-SEA",
    "token": {"engine": "", "id": 253, "logprob": 0, "special": false},
    "id": "671a9e090c3fe06af8ab9445a46684298b6f5e5b458c4ff8a145bee456eb77cf",
}

data: {
    "choices": [{"text": " French"}],
    "request_id": "83a18448f8c030ab-SEA",
    "token": {"engine": "", "id": 5112, "logprob": -0.8027344, "special": false},
    "id": "671a9e090c3fe06af8ab9445a46684298b6f5e5b458c4ff8a145bee456eb77cf",
}

...

data: [DONE]
```
 supports

### Motivation

together LLM integration does not support streaming although its endpoints are supported officially, adding streaming adds a huge benefit to user experience and quickly shows the model output generation

### Your contribution

implementing `_stream` method and processing the output of the API response, this can be done like this:
```python
payload = {
    ...,
    "stream_tokens": True
}
response = requests.post(..., payload, stream=True)
for line in response.iter_lines():
    ....
    yield  GenerationChunk(
           text = line["choices"][0]["text"], 
           ...
    )
refactored `CacheBackedEmbeddings`, `langchain.storage`
Labels: name: area: embeddings, name: auto:refactor, name: size:XXL

Refactored `CacheBackedEmbeddings`. Moved it from `langchain` to `langchain_core` (by @eyurtsev recommendation). As a result, [`langchain.embeddings` namespace completely removed](https://api.python.langchain.com/en/latest/langchain_api_reference.html#module-langchain.embeddings).
There was a dependency on `BaseStore`. So, I have also moved `langchain.storage` core artifacts to `langchain_core.storage`. [`langchain.storage` also](https://api.python.langchain.com/en/latest/langchain_api_reference.html#module-langchain.storage) completely removed.
langchain/libs/partners/google-genai/langchain_google_genai: safety_settings added to ChatGoogleGenerativeAI model in the langchain_google_genai library
Labels: name: auto:improvement, name: area: models, name: size:S, name: partner

- **Description** -> This PR **_adds safety_settings parameter_** to the **_ChatGoogleGenerativeAI_** class. The safety_settings parameter is originally there in the google.generativeai library's GenerativeModel class
- **Issue** -> https://github.com/langchain-ai/langchain/issues/15095
- **Dependencies** -> None added
docs: updated document for 'Return Source Documents' Functionality
Labels: name: area: doc loader, name: auto:documentation, name: size:S

<!-- Thank you for contributing to LangChain!

Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified.

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes if applicable,
  - **Dependencies:** any dependencies required for this change,
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/docs/integrations` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->

  - **Description:** updated the outdated code in the document that was generating the error, 
  - **Issue:** #15086 ,
  - **Dependencies:** N/A,
  - **Twitter handle:** [@vardhaman722](https://twitter.com/vardhaman722)
community: Integration of New Chat Model Based on ChatGLM3 via ZhipuAI API
Labels: name: lgtm, name: auto:enhancement, name: area: models, name: size:XL

  - **Description:** 
    - This PR introduces a significant enhancement to the LangChain project by integrating a new chat model powered by the third-generation base large model, ChatGLM3, via the zhipuai API. 
    - This advanced model supports functionalities like function calls, code interpretation, and intelligent Agent capabilities. 
    - The additions include the chat model itself, comprehensive documentation in the form of Python notebook docs, and thorough testing with both unit and integrated tests.
  - **Dependencies:** This update relies on the ZhipuAI package as a key dependency.
  - **Twitter handle:** If this PR receives spotlight attention, we would be honored to receive a mention for our integration of the advanced ChatGLM3 model via the ZhipuAI API. Kindly tag us at @kaiwu.

To ensure quality and standards, we have performed extensive linting and testing. Commands such as make format, make lint, and make test have been run from the root of the modified package to ensure compliance with LangChain's coding standards.

TO DO: Continue refining and enhancing both the unit tests and integrated tests.
community: added support for Guardrails for Amazon Bedrock
Labels: name: auto:enhancement, name: area: models, name: integration: aws, name: size:L

Added support for optionally supplying 'Guardrails for Amazon Bedrock' on both types of model invocations (batch/regular and streaming) and for all models supported by the Amazon Bedrock service.

@baskaryan 

```python 
llm = Bedrock(model_id="<model_id>", client=bedrock,
                  model_kwargs={},
                  guardrails={"id": " <guardrail_id>",
                              "version": "<guardrail_version>",
                               "trace": True}, callbacks=[BedrockAsyncCallbackHandler()])

class BedrockAsyncCallbackHandler(AsyncCallbackHandler):
    """Async callback handler that can be used to handle callbacks from langchain."""

    async def on_llm_error(
            self,
            error: BaseException,
            **kwargs: Any,
    ) -> Any:
        reason = kwargs.get("reason")
        if reason == "GUARDRAIL_INTERVENED":
           # kwargs contains additional trace information sent by 'Guardrails for Bedrock' service.
            print(f"""Guardrails: {kwargs}""")


# streaming 
llm = Bedrock(model_id="<model_id>", client=bedrock,
                  model_kwargs={},
                  streaming=True,
                  guardrails={"id": "<guardrail_id>",
                              "version": "<guardrail_version>"})
```
add: QuipLoader in langchain.document_loaders _init_.py
Labels: name: area: doc loader, name: auto:enhancement, name: size:XS

<!-- Thank you for contributing to LangChain!

Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified.

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes if applicable,
  - **Dependencies:** any dependencies required for this change,
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/docs/integrations` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Return similarity score ConversationalRetrievalChain
Labels: name: area: vector store, name: auto:improvement, name: integration: redis

### Feature request

I am using langchain.vectorstores.redis and langchain.chains.ConversationalRetrievalChain.from_llm

I would like to get the scores of the matching documents with my query. 


I know you can filter with the `search_kwargs={"score_threshold": 0.8}`

But still I want to get the similarity scores in the output.

### Motivation

To be able to play with the similarity scores on my end and allow flexibility to the user

### Your contribution

The output should be a list (like now) of tupples (Doc, score). In fact this already exists in the similarity_search_with_relevance_scores in lanchain.schema.vectorstore so the implementation should be quite straightforward

Thanks!
Foreign key associated with column 'langchain_pg_embedding.collection_id' could not find table
Labels: name: area: vector store, name: auto:bug

### Issue you'd like to raise.

I'm trying to initialize an existing collection via:

store = PGVector(
    collection_name=COLLECTION_NAME,
    connection_string=CONNECTION_STRING,
    embedding_function=embeddings,
)

I keep getting:

Exception has occurred: NoReferencedTableError
Foreign key associated with column 'langchain_pg_embedding.collection_id' could not find table 'langchain_pg_collection' with which to generate a foreign key to target column 'uuid'

I've used a docker installation for PGVector and I can confirm that the table langchain_pg_collection does exist with the key.

This is the view from PgAdmin

langchain_pg_collection
![image](https://github.com/pgvector/pgvector/assets/48178546/64b7d9ed-4046-4d14-a07b-7a7f23b38d19)

and

langchain_pg_embedding
![image](https://github.com/pgvector/pgvector/assets/48178546/dcd6ff9c-5ac8-4789-8e36-6aa63973c5eb)

So I'm not sure why its throwing the exception or how to resolve it


If its relevant I had to make this change inside pgvector

```
from sqlalchemy import MetaData

class CollectionStore(BaseModel):
    """Collection store."""

    metadata = MetaData()

    if not metadata.tables.get('langchain_pg_collection'):
        __tablename__ = "langchain_pg_collection"

        name = sqlalchemy.Column(sqlalchemy.String)
        cmetadata = sqlalchemy.Column(JSON)

        embeddings = relationship(
            "EmbeddingStore",
            back_populates="collection",
            passive_deletes=True,
        )
```

to resolve [aTable 'langchain_pg_collection' is already defined for this MetaData instance](https://github.com/langchain-ai/langchain/issues/14699)





### Suggestion:

_No response_
Feature: No safety settings when using langchain_google_genai's ChatGoogleGenerativeAI
Labels: name: auto:improvement, name: area: models

### Feature request

The safety settings are there in the **google_generativeai** library are are **not** there in the **langchain_google_genai** library
The safety settings is an basically array of dictionaries passed when sending the prompt

### Motivation

The problem with not having this is that when we use the ChatGoogleGenerativeAI model, if there is some kind of prompt which violate the basic safety settings then the model won't return with your answer 

If we can change the safety settings and send it with the prompt to the model we could fix this issue 

### Your contribution

I am currently reading the code of the library and will raise a PR if i could fix the issue
Agent how to call remote tool (exposed by langserve)
Labels: name: area: agent, name: auto:question

### System Info

langchain 0.0.352

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [X] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I use langserve develop a chain , and expose as remote tool. my friend wants to call my chain in his agent, how to do it? 

**Joke chain:** 

```
#!/usr/bin/env python
from fastapi import FastAPI
from langchain.prompts import ChatPromptTemplate
from langchain.chat_models import ChatAnthropic, ChatOpenAI
from langserve import add_routes
llm = ChatOpenAI(
    openai_api_base=f"http://192.168.1.201:18001/v1",
    openai_api_key="EMPTY",
    model="gpt-3.5-turbo",
    temperature=0.5,
    top_p="0.3",
    default_headers={"x-heliumos-appId": "general-inference"},
    tiktoken_model_name="gpt-3.5-turbo",
    verbose=True,
)

app = FastAPI(
  title="LangChain Server",
  version="1.0",
  description="A simple api server using Langchain's Runnable interfaces",
)


prompt = ChatPromptTemplate.from_template("tell me a joke about {topic}")
add_routes(
    app,
    prompt | llm,
    path="/joke",
)

if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app, host="0.0.0.0", port=8000)
```

**Agent:**

```
from langchain.agents import initialize_agent, AgentType

from langchain_community.chat_models import ChatOpenAI
from langserve import RemoteRunnable

from langchain.tools import Tool

llm = ChatOpenAI(
    openai_api_base=f"http://xxxx:xxx/v1",
    openai_api_key="EMPTY",
    model="gpt-3.5-turbo",
    temperature=0.5,
    top_p="0.3",
    tiktoken_model_name="gpt-3.5-turbo",
    verbose=True,
)

remote_tool = RemoteRunnable("http://xxx:xxx/joke/")
tools = [
    Tool.from_function(
        func=remote_tool.invoke,
        name="joke",
        description="用户要求讲笑话的时候使用该工具",
        # coroutine= ... <- you can specify an async method if desired as well
    ),
]
agent = initialize_agent(
    tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True
)

result = agent.run("讲一个关于坐出租车的笑话")
print(result)

```

agent always got error: because not there is not valid input to remote tool.




### Expected behavior

no error