delayedkarma/langchain-issues huggingface dataset:
Getting Import Error Basemodel

### System Info

Hello,

I am getting an error in import



Here are my import statements:
from llama_index import SimpleDirectoryReader, GPTListIndex, readers, GPTSimpleVectorIndex, LLMPredictor, PromptHelper
from langchain import OpenAI
import sys
import os
from IPython.display import Markdown, display

Here is the error:
from langchain.schema import BaseLanguageModel
ImportError: cannot import name 'BaseLanguageModel' from 'langchain.schema' (C:\Users\ali\PycharmProjects\GeoAnalyticsFeatures\venv\lib\site-packages\langchain\schema.py)

What I have tried:
upgrade PIP
upgrade lama

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Hello,

I am getting an error in import



Here are my import statements:
from llama_index import SimpleDirectoryReader, GPTListIndex, readers, GPTSimpleVectorIndex, LLMPredictor, PromptHelper
from langchain import OpenAI
import sys
import os
from IPython.display import Markdown, display

Here is the error:
from langchain.schema import BaseLanguageModel
ImportError: cannot import name 'BaseLanguageModel' from 'langchain.schema' (C:\Users\ali\PycharmProjects\GeoAnalyticsFeatures\venv\lib\site-packages\langchain\schema.py)

What I have tried:
upgrade PIP
upgrade lama

### Expected behavior

Hello,

I have followed all the instructions in the comments above but still coming the same issue:

Here are my import statements:
from llama_index import SimpleDirectoryReader, GPTListIndex, readers, GPTSimpleVectorIndex, LLMPredictor, PromptHelper
from langchain import OpenAI
import sys
import os
from IPython.display import Markdown, display

Here is the error:
from langchain.schema import BaseLanguageModel
ImportError: cannot import name 'BaseLanguageModel' from 'langchain.schema' (C:\Users\ali\PycharmProjects\GeoAnalyticsFeatures\venv\lib\site-packages\langchain\schema.py)

What I have tried:
upgrade PIP
upgrade lama
Fix exporting GCP Vertex Matching Engine from vectorstores
Labels: name: lgtm

The Vertex Matching Engine docs include [the line](https://github.com/hwchase17/langchain/blob/b177a29d3f942eeccf85814f0f628c32509b9b6a/docs/modules/indexes/vectorstores/examples/matchingengine.ipynb?short_path=54ebfde#L32) `from langchain.vectorstores import MatchingEngine` which doesn't work as it wasn't added to the vectorestores module exports.



  - @dev2049
Enable the use of Unstructured loaders with Google drive loader

### Feature request

The goal of this issue is to enable the use of Unstructured loaders in conjunction with the Google drive loader.

### Motivation

This would enable the use of the GoogleDriveLoader with document types other than the standard Google Drive documents/spreadsheets/presentations.

### Your contribution

I can create a PR for this issue as soon as I have bandwidth. If another community member picks this up first, I'd be happy to review.
added support for different types in ResponseSchema class

I added support for specifing different types with ResponseSchema objects:

## before
`
extracted_info = ResponseSchema(name="extracted_info", description="List of extracted information")
`
generate the following doc: ```json\n{\n\t\"extracted_info\": string // List of extracted information}```
This brings GPT to create a JSON with only one string in the specified field even if you requested a List in the description.

## now
`extracted_info = ResponseSchema(name="extracted_info", type="List[string]", description="List of extracted information")
`
generate the following doc: ```json\n{\n\t\"extracted_info\": List[string] // List of extracted information}```
This way the model responds better to the prompt generating an array of strings.

Tag maintainers/contributors who might be interested:
  Agents / Tools / Toolkits
  @vowelparrot

Don't know who can be interested, I suppose this is a tool, so I tagged you vowelparrot,
anyway, it's a minor change, and shouldn't impact any other part of the framework.
Not only print HumanInputRun tool query to console, but also export query as a string variable

### Feature request

Now when the `HumanInputRun` is run, it goes through three steps in order:

1. print out the query that is to be clarified to console
2. call input function to get user input
3. return the user input to the downstream

Hopefully the query in 1st step can be exported as a string **variable** like the LLM response. The flow like this:

Without tool:

```
user_input <- user input

response <- llm.run(user_input)
```

With tool:

```
user_input <- user input

response <- humantool(user_input)

additional_info <- user input

response <- llm.run(additional_info)
```

### Motivation

Because the HumanInputRun tool only prints the query to the console, looks like it can only be used with a console. It would be more flexible if the query can be exported as a variable and pass it to the user, then the user inputs the additional info as step 2 does.

### Your contribution

Currently, I still don't have a solution to it. Any suggestions?
Capture agent observations for further processing

### Feature request

I would like a way to capture observations in the same way you can capture new llm tokens using the callback's on_llm_new_token.

### Motivation

I've written a basic self-improving agent. I get it to "self-improve" by feeding the entire previous chain output to a meta agent that then refines the base agent's prompt. I accomplish this by appending all the tokens using the callback on_llm_new_token. However I realized it is not capturing observations as they are not llm generated tokens. How can I capture the observation as well?

### Your contribution

I've written an article detailing the process I've created for the self-improving agent along with the code [here](https://medium.com/p/d31204e1c375). Once I figure out how to also capture the agent observations I will add the solution to the documentation under a header such as "Agent observability - capture the entire agent chain"
Issue: How to do the recursivley summarization with the summarize_chain?

### Issue you'd like to raise.

For the code of class`MapReduceDocumentsChain`, there is a comment in `combine_docs` saying that
        
        """Combine documents in a map reduce manner.

        Combine by mapping first chain over all documents, then reducing the results.
        This reducing can be done recursively if needed (if there are many documents).
        """

My questioin is how to build the recursively manner? I didn't see any specific API that allow me to implement it? Do I need to build this functionality by myself? Suppose I have a book-length document to summarize.

### Suggestion:

_No response_
[Feature] Implementation of OpenAI's Edit Models

### Feature request

Regarding issue #4999, the `code-davinci-002` model has been deprecated, so we can no longer use it with langchain.llm.OpenAI. Additionally, OpenAI also released the API called [edit models](https://platform.openai.com/docs/api-reference/edits), which includes `text-davinc-edit-001` and `code-davinci-edit-001`.

I'm curious to know if there are any plans to integrate these edit models into the existing LLMs and Chat Models. If this integration has already been implemented, kindly let me know, as I might have missed the update.

### Motivation

This feature allows for convenient utilization of OpenAI's edit models, such as `text-davinc-edit-001` and `code-davinci-edit-001`.

### Your contribution

If this feature is valuable and requires implementation, I'm willing to assist or collaborate with the community in its implementation. However, please note that I may not have the ability to implement it independently.
fix: OpenAI and AzureChatOpenAI variables conflict

<!--
Thank you for contributing to LangChain! Your PR will appear in our release under the title you set. Please make sure it highlights your valuable contribution.

After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost.

Finally, we'd love to show appreciation for your contribution - if you'd like us to shout you out on Twitter, please also include your handle!
-->

Fixes #5422.
Issue arises when using ChatOpenAI/OpenAI (llm) and AzureChatOpenAI one after another, as the latter modifies some default `openai` vars such as `openai.api_base` which end up raising an error when calling openai services.

Proposed solution is to read the default values and store them. As the `openai` module can easily be imported and changed before `langchain` come into context, my proposal is to run (once) the import in a fresh sub-process, get the default variables we need, and then caching the result. 
Alternatively one can hard-code these values like so 
```python
  openai.api_type = "open_ai"
  openai.api_base = "https://api.openai.com/v1"
  openai.api_version = None
```
But they might change if OpenAI decides so.

#### Before submitting

<!-- If you're adding a new integration, please include:

1. a test for the integration - favor unit tests that does not rely on network access.
2. an example notebook showing its use


See contribution guidelines for more information on how to write tests, lint
etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->
Not sure how to add tests as it requires making the request and hence having keys and endpoint setup for azure. Is there any dry-run code infrastructure I can leverage to do so..? 

#### Who can review?

@vowelparrot (apologies if the tag is misplaced, I am not sure who to add for this). 

<!-- For a quicker response, figure out the right person to tag with @

  @hwchase17 - project lead

  Tracing / Callbacks
  - @agola11

  Async
  - @agola11

  DataLoaders
  - @eyurtsev

  Models
  - @hwchase17
  - @agola11

  Agents / Tools / Toolkits
  - @vowelparrot

  VectorStores / Retrievers / Memory
  - @dev2049

 -->
fix intermediate step over length
Labels: name: area: agent, name: auto:bug

<!--
fix the problem of agent_scratchpad over length
-->
fix the problem of agent_scratchpad over length, during using 'CONVERSATIONAL_REACT_DESCRIPTION' agent. 
This problem is caused by the simple concatenation of intermediate_steps in Function '_construct_scratchpad', so I changed this to ignore the earliest intermediate_steps.

Note: This fix add two parameters in Agent class:
 - scratchpad_max_size: the max size of constructed agent_scratchpad.
 - length_function: the length function used to count the length of agent_scratchpad.


#### Who can review?

Tag maintainers/contributors who might be interested:

<!-- For a quicker response, figure out the right person to tag with @

   @hwchase17 - project lead

  Agents / Tools / Toolkits
  - @vowelparrot

 -->
Dynamic loading of tools and custom tools

### Feature request

Maybe I didn't get the idea: Some tools can be loaded like 'google-search' other tools can't be loaded like duckduckgo-search by the load_tools: langchain/agents/load_tools.py
E.g. specified in _EXTRA_OPTIONAL_TOOLS.
As for custom tools they can't be loaded at all it seems in that way. It would be great to register custom tools so that they can be loaded by load_tools as well.

I've done something like this for the moment, so I have Class-Names for my own tools and the lower-case would be standard:

```
        standard_tools = []
        for toolStr in tool_names:
            if toolStr[0].islower():
                standard_tools.append(toolStr)
        self.tools = load_tools(standard_tools, **tool_kwargs)
        for toolStr in tool_names:
            if toolStr[0].isupper():
                toolClass = globals()[toolStr]
                toolObject = toolClass()            
                self.tools.append(copy.deepcopy(toolObject))
```

### Motivation

When assigning agents a set of tools it's very handy to do it through a list.

### Your contribution

I'm not familiar enough with the architecture of langchain so I'd better not. But I'm happy to test it.
YoutubeAudioLoader and updates to OpenAIWhisperParser

This introduces the `YoutubeAudioLoader`, which will load blobs from a YouTube url and write them. Blobs are then parsed by `OpenAIWhisperParser()`, as show in this [PR](https://github.com/hwchase17/langchain/pull/5580), but we extend the parser to split audio such that each chuck meets the 25MB OpenAI size limit. As shown in the notebook, this enables a very simple UX:

```
# Transcribe the video to text
loader = GenericLoader(YoutubeAudioLoader([url],save_dir),OpenAIWhisperParser())
docs = loader.load()
``` 

Tested on full set of Karpathy lecture videos:

```
# Karpathy lecture videos
urls = ["https://youtu.be/VMj-3S1tku0"
        "https://youtu.be/PaCmpygFfXo",
        "https://youtu.be/TCH_1BHY58I",
        "https://youtu.be/P6sfmUTpUmc",
        "https://youtu.be/q8SA3rM6ckI",
        "https://youtu.be/t3YJ5hKiMQ0",
        "https://youtu.be/kCc8FmEb1nY"]

# Directory to save audio files 
save_dir = "~/Downloads/YouTube"
 
# Transcribe the videos to text
loader = GenericLoader(YoutubeAudioLoader(urls,save_dir),OpenAIWhisperParser())
docs = loader.load()
```
ML flow call backs fails at missing keys

### System Info

Langchain version 0.0.191 , python version 3.9.15 


### Who can help?

@agola

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [X] Callbacks/Tracing
- [ ] Async

### Reproduction

following the notebook given at  https://python.langchain.com/en/latest/integrations/mlflow_tracking.html?highlight=mlflow

I just changed the chain from LLM chain to retrieval chain and OpenAI to Azure Open AI

i continuously faced the error KeyError: "None of [Index(['step', 'prompt', 'name'], dtype='object')] are in the [columns]"

But some data is getting logged into ML_flow experiments. like chain_1_start and chain_2_start and their ends respectively.

### Expected behavior

To have the necessary data in the ML flow experiment logs and should work without throwing any error.
Fix IndexError when input_list is Empty in prep_prompts
Labels: name: area: memory, name: lgtm, name: auto:bug

This MR corrects the IndexError arising in prep_prompts method when no documents are returned from a similarity search.

Fixes #1733 

#### Who can review?

Tag maintainers/contributors who might be interested:

  @hwchase17 - project lead

  VectorStores / Retrievers / Memory
  - @dev2049
Observability Feature Request: Metrics, Data Monitoring, and Graph Visualization

### Feature request

I would like to propose the addition of an **observability feature** to LangChain, enabling developers to monitor and analyze their applications more effectively. This feature aims to provide metrics, data tracking, and graph visualization capabilities to enhance observability.

Key aspects of the proposed feature include:

- **Metrics Tracking**: Capture time taken by LLM model to handle request, errors, number of tokens, and costing indication for the particular LLM.
- **Data Tracking**: Log and store prompt, request, and response data for each LangChain interaction.
- **Graph Visualization**: Generate basic graphs over time, depicting metrics such as request duration, error occurrences, token count, and cost.

This feature request aims to improve the development and debugging experience by providing developers with better insights into the performance, behavior, and cost implications of their LangChain applications.



### Motivation

Last week, while I was creating an application using OpenAI's GPT 3.5 APIs to develop a chat-based solution for answering SEO-related queries, I encountered a significant challenge. During the launch, I realized that I had no means to monitor the responses suggested by the language model (GPT, in my case). Additionally, I had no insights into its performance, such as speed or token usage. To check the token count, I had to repeatedly log in to the OpenAI dashboard, which was time-consuming and cumbersome.

It was frustrating not having a clear picture of user interactions and the effectiveness of the system's responses. I realised that I needed a way to understand the system's performance in handling different types of queries and identify areas that required improvement.

I strongly believe that incorporating an observability feature in LangChain would be immensely valuable. It would empower developers like me to track user interactions, analyze the quality of responses, and measure the performance of the underlying LLM requests. Having these capabilities would not only provide insights into user behavior but also enable us to continuously improve the system's accuracy, response time, and overall user experience.


### Your contribution

Yes, I am planning to raise a PR along with a couple of my friends to add an observability feature to Langchain.

I would be more than happy to take suggestions from the community, on what we could add to make it more usable!
feature: add `split_wrap_text` method to Recursive/CharacterTextSplitter

This method allows adding a `prefix` and a `suffix` to each split text.
This is useful for wrapping the text with special characters, tokens or formatting tags.

- This method implemented in `CharacterTextSplitter` and `RecursiveCharacterTextSplitter`
- The method `split_wrap_text` calls the existing `split_text` method internally and then adds the prefix and suffix
 to each element of the output list.
- The method has an optional parameters for the prefix and suffix, which default to empty strings.

<!--
Thank you for contributing to LangChain! Your PR will appear in our release under the title you set. Please make sure it highlights your valuable contribution.

Replace this with a description of the change, the issue it fixes (if applicable), and relevant context. List any dependencies required for this change.

After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost.

Finally, we'd love to show appreciation for your contribution - if you'd like us to shout you out on Twitter, please also include your handle!
-->

<!-- Remove if not applicable -->


<!-- If you're adding a new integration, please include:

1. a test for the integration - favor unit tests that does not rely on network access.
2. an example notebook showing its use


See contribution guidelines for more information on how to write tests, lint
etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->

#### Who can review?

Tag maintainers/contributors who might be interested: @eyurtsev, @vowelparrot

<!-- For a quicker response, figure out the right person to tag with @

  @hwchase17 - project lead

  Tracing / Callbacks
  - @agola11

  Async
  - @agola11

  DataLoaders
  - @eyurtsev

  Models
  - @hwchase17
  - @agola11

  Agents / Tools / Toolkits
  - @vowelparrot

  VectorStores / Retrievers / Memory
  - @dev2049

 -->
Wheynelau/json double quote
Labels: name: needs test, name: auto:bug

In the "langchain/tools/json/tool.py" file, fixed the _parse_input function to properly handle single quotes. I've added a second .replace() to replace the single quotes. This is a workaround I can suggest.

Resolves issue #5759 

Test:
1. Added different orderings of quotes in test_json.py
2. Was unable to run poetry but tested in notebook


  - @vowelparrot
Improving Custom Agent Speed?

### Discussed in https://github.com/hwchase17/langchain/discussions/5730

<div type='discussions-op-text'>

<sup>Originally posted by **Radvian** June  5, 2023</sup>

Hello, not sure if I should ask this in Discussion or Issue, so apologies beforehand.

I'm doing a passion project in which I try to create a knowledge base for a website, and create a Conversational Retrieval QA Chain so we can ask questions about the knowledge base. 

This is very much akin to this: https://python.langchain.com/en/latest/modules/chains/index_examples/chat_vector_db.html 

And I have no problem to implement it. 

Now, I want to go to the next step. Instead of creating a chain, I want to create an agent.

I want to do something like this:
- First, always ask the user for their name.
- If the user asks a particular question that is not in the knowledge base, don't reply with "I don't know", but search Wikipedia about it
- If the user asks about X, send an image link (from google search images) of 'X' item
- and few other custom tools

And, I also don't have any problem in creating tools and combining them into a unified agent.

However, my problem is this (and this is expected): the agent performs slower, much slower than if I just use the chain.

If I only use Retrieval QA Chain, and enables _streaming_, it can start typing the output in less than 5 seconds.

If I use the complete agent, with a few custom tools, it starts to type the answer / output within 10-20 seconds.

I know this is to be expected, since the agent needs to think about which tools to use. But any ideas on how to make this faster? In the end, I want to deploy this whole project in a streamlit and wants this to perform like a 'functional' chatbot, with almost-instant reply, and not having to wait 10-20 seconds before the 'agent' starts typing.

Any ideas are welcome, apologies if this is a stupid question. Thank you!</div>
fix(langchain): fix _parse_input function

In the "langchain/tools/json/tool.py" file, fixed the _parse_input function to properly handle single quotes.

Resolves issue #5759.


#### Before submitting

<!-- If you're adding a new integration, please include:

1. I've added some tests but I couldn't get poetry to work, still troubleshooting


#### Who can review?

Tag maintainers/contributors who might be interested:

  Agents / Tools / Toolkits
  - @vowelparrot
Modify ElasticKnnSearch to return Document object

### Feature request

When performing knnSearch or hybridKnnSearch return `Document` object

### Motivation

Currently the return a `dict` object. It would be more useful to return [langchain.schema.Document](https://github.com/hwchase17/langchain/blob/d5b160821641df77df447e6dfce21b58fbb13d75/langchain/schema.py#LL266C10-L266C10)

### Your contribution

I will make the canges