delayedkarma/langchain-issues huggingface dataset:
Feature/fix azureopenai model mappings
Labels: name: auto:improvement

This pull request aims to ensure that the `OpenAICallbackHandler` can properly calculate the total cost for Azure OpenAI chat models. The following changes have resolved this issue:

- The `model_name` has been added to the ChatResult llm_output. Without this, the default values of `gpt-35-turbo` were applied. This was causing the total cost for Azure OpenAI's GPT-4 to be significantly inaccurate.
- A new parameter `model_version` has been added to `AzureChatOpenAI`. Azure does not include the model version in the response. With the addition of `model_name`, this is not a significant issue for GPT-4 models, but it's an issue for GPT-3.5-Turbo. Version 0301 (default) of GPT-3.5-Turbo on Azure has a flat rate of 0.002 per 1k tokens for both prompt and completion. However, version 0613 introduced a split in pricing for prompt and completion tokens.
- The `OpenAICallbackHandler` implementation has been updated with the proper model names, versions, and cost per 1k tokens.

Unit tests have been added to ensure the functionality works as expected; the Azure ChatOpenAI notebook has been updated with examples.

Maintainers: @hwchase17, @baskaryan

Twitter handle: @jjczopek
MarkdownHeaderTextSplitter is not a TextSplitter
Labels: name: auto:bug

### System Info

The class `MarkdownHeaderTextSplitter` is not a `TextSplitter`, and must implements all the corresponding methods.

```
class MarkdownHeaderTextSplitter:
  ...
```
@hwchase17 @eyurtsev

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

headers_to_split_on = [
                            ("#", "Header 1"),
                            ("##", "Header 2"),
                            ("###", "Header 3"),
                        ]
splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=headers_to_split_on
)
splitter.split_documents(docs)

### Expected behavior

Accept the call
Langchain Support for Onnx Llama 
Labels: name: auto:enhancement

### Feature request

Recently Microsoft have announced their [first iteration of running Llama using Onnx format](https://github.com/microsoft/Llama-2-Onnx/tree/main). Hence it will be awesome if LangChain comes up with an early support for Onnx runtime models.  

### Motivation

There are two reasons for this.

1. Onnx has always been the standard for running inference in CPU / GPU (Onnx GPU), so this idea of providing LLMs supported for Onnx runtime format will move forward fast
2. Current implementations of running the same is an overhead, LangChain can provide the abstraction easily.

### Your contribution

I can try starting out to experiment with this of whether we can implement or not by using the existing LLM interface. However the bottleneck here becomes the .onnx format weights which are to be requested to Microsoft. I filled out the application, waiting for approval. Let me know if we can work on this issue.
AgentType.OPENAI_FUNCTIONS did not work
Labels: name: area: agent, name: auto:bug

### System Info

langchain  0.0.249
Python  3.11.2

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I follow the egs of openai_functions_agent at

https://python.langchain.com/docs/modules/agents/
https://python.langchain.com/docs/modules/agents/agent_types/openai_functions_agent
https://python.langchain.com/docs/modules/agents/how_to/custom-functions-with-openai-functions-agent

```
from langchain.chat_models import ChatOpenAI
from langchain.schema import SystemMessage
from langchain.agents import OpenAIFunctionsAgent
from langchain.agents import tool
from langchain.agents import AgentExecutor

llm = ChatOpenAI(temperature=0)

@tool
def get_word_length(word: str) -> int:
    """Returns the length of a word."""
    import pdb   # never go in here
    pdb.set_trace()
    return len(word)

tools = [get_word_length]

system_message = SystemMessage(content="You are very powerful assistant, but bad at calculating lengths of words")
prompt = OpenAIFunctionsAgent.create_prompt(system_message=system_message)

agent = OpenAIFunctionsAgent(llm=llm, tools=tools, prompt=prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

agent_executor.run("how many letters in the word educa?")
```

But the openai_functions_agent do not invoke the tool as expected。 

```
> Entering new AgentExecutor chain...
[{'role': 'system', 'content': 'You are very powerful assistant, but bad at calculating lengths of words'}, {'role': 'user', 'content': 'how many letters in the word educa?'}]
There are 5 letters in the word "educa".

> Finished chain.
```

I tried using other tools but they were not used . It seems that OPENAI_FUNCTIONS has bugs。



### Expected behavior

AgentType.OPENAI_FUNCTIONS works as docs show
Enhance GraphQL Tool to accept Endpoint, Headers and Query Variables at the time of Query Execution
Labels: name: auto:improvement

 - Description: The current GraphQL tool requires graphql endpoint and request headers at the time of the tool instantiation which makes the tool less flexible as it could be used only to run queries against a single endpoint and using the same headers. This pull-request contains changes to allow using a different graphql endpoint and request headers at the time of the query execution making the tool dynamic. Secondly, this pull-request further enhances the GraphQL tool by accepting query variables besides the GraphQL query. 

 - Issue: https://github.com/hwchase17/langchainjs/issues/1713
 
  - Dependencies: None
  
  - Tag maintainer: @hinthornw
  
  - Twitter handle: None
improve the performance of base.py
Labels: name: area: agent, name: lgtm, name: auto:improvement

This removes the use of the intermediate df list and directly concatenates the dataframes if path is a list of strings. The pd.concat function combines the dataframes efficiently, making it faster and more memory-efficient compared to appending dataframes to a list.

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
[Opensearch] : Fix the service validation in http_auth
Labels: name: area: vector store, name: auto:bug

### Description
OpenSearch supports validation using both Master Credentials (Username and password) and IAM. For Master Credentials users will not pass the argument `service` in `http_auth` and the existing code will break. To fix this, I have updated the condition to check if service attribute is present in http_auth before accessing it. 

### Maintainers
@baskaryan @navneet1v
fix issue #8556 by restructure the metadata links
Labels: name: area: doc loader, name: auto:bug

  - Description: restructure the metadata links from {str, list}, and flatten it to {str, str} which can be supported by Chroma vector store. The former format for links loaded from markdown file is not supported by Chroma. 
  - Issue: the issue #8556
  - Tag maintainer: @rlancemartin
Add firestore_client param to FirestoreChatMessageHistory if caller already has one; also lets them specify GCP project, etc.
Labels: name: area: memory, name: lgtm, name: auto:improvement

Existing implementation requires that you install `firebase-admin` package, and prevents you from using an existing Firestore client instance if available.

This adds optional `firestore_client` param to `FirestoreChatMessageHistory`, so users can just use their existing client/settings. If not passed, existing logic executes to initialize a `firestore_client`.
LlamaCpp, add n_gqa flag to support Llama2
Labels: name: auto:improvement

To load and run llama 2 70b models `n_gqa` must be set to 8. For other models it can be unset (and defaults to 1). Tested with both llama 2 70b (set) and llama 2 13b (unset) and both work.

Maintainer guess @hwchase17 

Fixes https://github.com/langchain-ai/langchain/issues/8486
Fixed source key name for docugami loader
Labels: name: area: doc loader, name: auto:bug

The Docugami loader was not returning the source metadata key. This was triggering this exception when used with retrievers, per https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/schema/prompt_template.py#L193C1-L195C41

The fix is simple and just updates the metadata key name for the document each chunk is sourced from, from "name" to "source" as expected.

I tested by running the python notebook that has an end to end scenario in it.

Tagging DataLoader maintainers @rlancemartin @eyurtsev
docs: update `unstructured` install instructions
Labels: name: auto:documentation

### Summary

Updates the `unstructured` install instructions. For `unstructured>=0.9.0`, dependencies are broken out by document type and the base `unstructured` package includes fewer dependencies. `pip install "unstructured[local-inference]"` has been replace by `pip install "unstructured[all-docs]"`, though the `local-inference` extra is still supported for the time being.

### Reviewers

- @rlancemartin
- @eyurtsev
- @hwchase17
Support for `Segment` analytics as a utility tool
Labels: name: auto:enhancement

Replace this comment with:
  - Description: The changes will introduce a tool `Segment` for LLM based app's analytics.
  - Dependencies: `segment-analytics-python`
  - Tag maintainer: @hinthornw 
  - Twitter handle: @yashpokar

The PR is passing linting and testing before submitting. Ran `make format`, `make lint` and `make test` to check this locally.

If you're adding a new integration, please include:
- [x]  a test for the integration, preferably unit tests that do not rely on network access,
- [x] an example notebook showing its use.

<!--
If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->
updated to use new llm_util query
Labels: name: lgtm, name: auto:enhancement

  - Description: added memgraph_graph.py which defines the MemgraphGraph class, subclassing off the existing Neo4jGraph class. This lets you query the Memgraph graph database using natural language. It leverages the Neo4j drivers and the bolt protocol. 
  - Dependencies: since it is a subclass off of Neo4jGraph, it is dependent on it and the GraphCypherQA Chain implementations. It is dependent on the Neo4j drivers being present. It is dependent on having a running Memgraph instance to connect to. 
  - Tag maintainer: @baskaryan
  - Twitter handle: @villageideate
  - example usage can be seen in this repo https://github.com/brettdbrewer/MemgraphGraph/
Missing some input keys: {'input'} when using Combined memory and ConversationalRetrievalChain
Labels: name: area: memory, name: auto:bug

### Issue you'd like to raise.

Hello there, 

I am currently attempting to add combined memory to ConversationalRetrievalChain (CRC). However, I am unsure if combined memory is compatible with CRC. I have made a few modifications, but unfortunately, I have encountered errors preventing it from running properly. I was hoping that someone could assist me in resolving this issue. Thank you.

```

from langchain.prompts import PromptTemplate
from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferWindowMemory, CombinedMemory, ConversationSummaryMemory

chat_history = []

conv_memory = ConversationBufferWindowMemory(
    memory_key="chat_history_lines",
    input_key="input",
    output_key='answer',
    k=1
)

summary_memory = ConversationSummaryMemory(llm=turbo_llm, input_key="input", output_key='answer')

# Combined
memory = CombinedMemory(memories=[conv_memory, summary_memory])

_DEFAULT_TEMPLATE = """The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.
{context}
Summary of conversation:
{history}
Current conversation:
{chat_history_lines}
Human: {input}
AI:"""

PROMPT_= PromptTemplate(
    input_variables=["history", "input", "context", "chat_history_lines"],
    template=_DEFAULT_TEMPLATE
)

qa = ConversationalRetrievalChain.from_llm(
                        llm=turbo_llm,
                        chain_type="stuff",
                        memory=memory,
                        retriever=retriever,
                        return_source_documents=True,
                        return_generated_question=True,
                        combine_docs_chain_kwargs={'prompt': PROMPT_})


chat_history = []
query = "What did the president say about Ketanji Brown Jackson"
result = qa({"question": query, "chat_history": chat_history})`
```
Error:
`ValueError: Missing some input keys: {'input'}`


when I replace "input" with "question" in _DEFAULT_TEMPLATE and PROMPT_
then  the error is :
`ValueError: One output key expected, got dict_keys(['answer', 'source_documents', 'generated_question'])
`
### Suggestion:

_No response_
Implement a router for openai functions
Labels: name: auto:enhancement

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Handling potential issue in file.py
Labels: name: auto:bug

Potential Resource Leak: The FileCallbackHandler class opens a file in its constructor but does not handle the case where the file opening fails. If there's an exception while opening the file, the object is not closed, which may lead to resource leaks. It's recommended to handle the file opening more gracefully using a try-except block or a context manager (with statement) to ensure the file is closed correctly, even if an exception occurs during the opening process.

@baskaryan , @hwchase17
Callback handler for Amazon SageMaker Experiments
Labels: name: lgtm, name: auto:enhancement

## Description

This PR implements a callback handler for SageMaker Experiments which is similar to that of mlflow.
* When creating the callback handler, it takes the experiment's run object as an argument. All the callback outputs are then logged to the run object.
* The output of each callback action (e.g., `on_llm_start`) is saved to S3 bucket as json file.
* Optionally, you can also log additional information such as the LLM hyper-parameters to the same run object.
* Once the callback object is no more needed, you will need to call the `flush_tracker()` method. This makes sure that any intermediate files are deleted.
* A separate notebook example is provided to show how the callback is used.

Thank you [Mohamad Al Jazaery](https://github.com/mohjaz), [Shinan Zhang](https://github.com/sz640) for your contribution and review!

@3coins  @agola11
Airbyte based loaders
Labels: name: area: doc loader, name: auto:enhancement

This PR adds 8 new loaders:
* `AirbyteCDKLoader` This reader can wrap and run all python-based Airbyte source connectors.
* Separate loaders for the most commonly used APIs:
  * `AirbyteGongLoader`
  * `AirbyteHubspotLoader`
  * `AirbyteSalesforceLoader`
  * `AirbyteShopifyLoader`
  * `AirbyteStripeLoader`
  * `AirbyteTypeformLoader`
  * `AirbyteZendeskSupportLoader`

## Documentation and getting started
I added the basic shape of the config to the notebooks. This increases the maintenance effort a bit, but I think it's worth it to make sure people can get started quickly with these important connectors. This is also why I linked the spec and the documentation page in the readme as these two contain all the information to configure a source correctly (e.g. it won't suggest using oauth if that's avoidable even if the connector supports it).

## Document generation
The "documents" produced by these loaders won't have a text part (instead, all the record fields are put into the metadata). If a text is required by the use case, the caller needs to do custom transformation suitable for their use case.

## Incremental sync
All loaders support incremental syncs if the underlying streams support it. By storing the `last_state` from the reader instance away and passing it in when loading, it will only load updated records.
GPTCache implementation isnt working with ChatOpenAI
Labels: name: auto:bug

### System Info

langchain = "^0.0.248"



### Who can help?

@hwchase17 @agola11 

### Information

- [x] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [x] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```python
import os
import hashlib
import langchain

from gptcache import Cache  # type: ignore
from gptcache.manager.factory import manager_factory  # type: ignore
from gptcache.processor.pre import get_prompt  # type: ignore

from langchain.cache import GPTCache
from langchain.llms import OpenAI
from langchain.chat_models import ChatOpenAI

os.environ["OPENAI_API_KEY"] = "sk-..."


llm = ChatOpenAI(temperature=0) # type: ignore


def get_hashed_name(name):
    return hashlib.sha256(name.encode()).hexdigest()


def init_gptcache(cache_obj: Cache, llm: str):
    hashed_llm = get_hashed_name(llm)
    cache_obj.init(
        pre_embedding_func=get_prompt,
        data_manager=manager_factory(manager="map", data_dir=f"map_cache_{hashed_llm}"),
    )


langchain.llm_cache = GPTCache(init_gptcache)

# The first time, it is not yet in cache, so it should take longer
response = llm("Tell me a joke")
print(f"Response 1: {response}")

# The first time, it is not yet in cache, so it should take longer
response = llm("Tell me a joke")
print(f"Response 2: {response}")
```

### Expected behavior

Response 1: 

Why did the chicken cross the road?

To get to the other side.
Response 2: 

Why did the chicken cross the road?

To get to the other side.
SQL block potential harmful commands
Labels: name: auto:improvement

  Description: 
This PR aims to give a bit more control on the commands executed by the `SQLDatabase`. At the moment, if you ask "Drop the table X in database Y" with `SQLDatabaseChain`, it will generate code to drop the table that will be run by `SQLDatabase`. Today no checks are made on the commands in `SQLDatabase`, it's free real estate.

Ideally, rights on user must be set to perform or not this kind of action on the database. In real life, lot of people use full rights. To prevent this I added a feature that could, if provided a list of restricted actions, stop the execution of the SQL query.

It will raise a `PermissionError` when restricted keywords are in the command for `SQLDatabase`

Key Changes:
- Add a new argument to `SQLDatabase` class named `restricted_keywords`
- Add a new method to detect restricted keywords in the SQL command
- Raise `PermissionError` if keywords are found in the sql command
- Execute sql command like before if not keyworkds were found
- Add a test to check if the raise is working while adding one restricted keyword

 `make format`, `make lint` and `make test` were tested locally

I think this could help for [CVE-2023-36189](https://github.com/advisories/GHSA-7q94-qpjr-xpgm) Where the developer could block any keywords that could be generated by `SQLDatabaseChain`. It can block any string, so he could also block the usage of wildcards, name of tables, databases. I think it can be powerfull, it gives control to the developer and protect the user doing unwanted actions or protect his database.

@baskaryan