delayedkarma/langchain-issues huggingface dataset:
[Enhancement] Timestamp supported Message and/or Memory
Labels: name: area: memory, name: auto:enhancement

### Feature request

Make `Message` and/or `Memory` to support `Timestamp`

### Motivation

To provide context and clarity regarding the timing of conversation. This can be helpful for reference and coordination, especially when discussing time-sensitive topics.

I noticed that [one agent in opengpts](https://opengpts-example-vz4y4ooboq-uc.a.run.app/) has supported this feature.

### Your contribution

I have not made a clear outline of adding the `Timestamps` feature.

The following is some possible ways to support it for discussion:

Proposal 1: Add `Timestamps` to `Message Schema`. This way every `Memory Entity` should support `Timestamps`.

Proposal 2: Create a new `TimestampedMemory`. This way has a better backward compatibility.
DOC: Clarify how to handle runs and linked calls with run_managers
Labels: name: area: doc loader, name: auto:documentation

### Issue with current documentation:

I'm trying to work through the streaming parameters around run_manager and callbacks.

Here's a minimal setup of what I'm trying to establish

```
class MyTool(BaseTool)
    name: "my_extra_tool"

    async def _arun(self, query: str, run_manager: Optional[AsyncCallbackManagerForToolRun] = None) -> str:
        """Use the tool asynchronously."""
        nested_manager = run_manager.get_child() # run_manager doesn't have an `on_llm_start` method, only supports `on_tool_end` / `on_tool_error` and `on_text` (which has no callback in LangchainTracer
        llm_run_manager = await nested_manager.on_llm_start({"llm": self.llm, "name": self.name+"_substep"}, prompts=[query]) # need to give
        # do_stuff, results in main_response
        main_response = "<gets created in the tool, might be part of streaming output in the future>"
        await llm_run_manager[0].on_llm_new_token(main_response) # can't use llm_run_manager directly as it's a list
        await llm_run_manager[0].on_llm_end(response=main_response)
```

I'm seeing that on_llm_new_token callback is being called in my custom callback handler, but I don't see the response in Langsmith.

The docs aren't fully clear on how to make sure these run_ids should be propagated.


### Idea or request for content:

It would be fantastic to have a detailed example of how to correctly nest runs with arbitrary tools.
Feature: Add iFlyTek Spark LLM chat model support
Labels: name: auto:enhancement, name: area: models, name: size:XL

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->

  - **Description:** This PR enables LangChain to access the iFlyTek's Spark LLM via the chat_models wrapper.
  - **Dependencies:** websocket-client ^1.6.1
  - **Tag maintainer:** @baskaryan 

### SparkLLM chat model usage

Get SparkLLM's app_id, api_key and api_secret from [iFlyTek SparkLLM API Console](https://console.xfyun.cn/services/bm3) (for more info, see [iFlyTek SparkLLM Intro](https://xinghuo.xfyun.cn/sparkapi) ), then  set environment variables `IFLYTEK_SPARK_APP_ID`, `IFLYTEK_SPARK_API_KEY` and `IFLYTEK_SPARK_API_SECRET` or pass parameters when using it like the demo below:

```python3
from langchain.chat_models.sparkllm import ChatSparkLLM

client = ChatSparkLLM(
    spark_app_id="<app_id>",
    spark_api_key="<api_key>",
    spark_api_secret="<api_secret>"
)
```
MySQL : SQL Query generated contains enclosing single quotes leading to SQL Alchemy giving Programming Error, 1064
Labels: name: auto:bug, name: area: models

### System Info

Langchain version: 0.0.321
Python: 3.10

### Who can help?

@hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I am using Langchain to generate and execute SQL queries for MySql database.
The SQL Query generated is enclosed in single quotes

Generate SQL Query: **'**"SELECT * FROM EMPLOYEE WHERE ID = 123"**'**
Expected SQL Query: "SELECT * FROM EMPLOYEE WHERE ID = 123"

Hence though the query is correct, sql alchemy is unable to execute query and gives Programming error
(pymysql.err.ProgrammingError) (1064, 'You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use 

### Expected behavior

Expected SQL Query should be without enclosing single quotes.

I did some debugging as looks like we get single quotes while invoking predict method of LLM - https://github.com/langchain-ai/langchain/blob/master/libs/experimental/langchain_experimental/sql/base.py#L156
Multi-modal RAG template
Labels: name: auto:enhancement, name: area: models

Template that follow recipe for multi-modal RAG using multi-vector retriever as shown in recent [cookbook](https://github.com/langchain-ai/langchain/blob/master/cookbook/Multi_modal_RAG.ipynb).

Applies RAG on [this blog post](https://cloudedjudgement.substack.com/p/clouded-judgement-111023).
Issue: <error faq() takes 1 positional argument but 2 were given>
Labels: name: area: agent, name: auto:bug

### Issue you'd like to raise.

```
openai.proxy = {
            "http": "http://127.0.0.1:7890",
            "https": "http://127.0.0.1:7890"
        }

callback = AsyncIteratorCallbackHandler()

llm = OpenAI(
    openai_api_key= os.environ["OPENAI_API_KEY"],
    temperature=0, 
    streaming=True, 
    callbacks=[callback]
)
embeddings = OpenAIEmbeddings()

# faq
loader = TextLoader("static/faq/ecommerce_faq.txt")
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
texts = text_splitter.split_documents(documents)

docsearch = Chroma.from_documents(texts, embeddings)

faq_chain = RetrievalQA.from_chain_type(
    llm, 
    chain_type="stuff", 
    retriever=docsearch.as_retriever(),
)

@tool("FAQ")
def faq(input) -> str:
    """"useful for when you need to answer questions about shopping policies, like return policy, shipping policy, etc."""
    print('faq input', input)
    return faq_chain.acall(input)

tools = [faq]

memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)

conversation_agent = initialize_agent(
    tools, 
    llm, 
    agent="conversational-react-description", 
    memory=memory, 
    verbose=True,
)
    
 async def wait_done(fn, event):
    try:
        await fn
    except Exception as e:
        print('error', e)
        # event.set()
    finally:
        event.set()


async def call_openai(question):

    # chain = faq(question)
    chain = conversation_agent.acall(question)

    coroutine = wait_done(chain, callback.done)

    task = asyncio.create_task(coroutine)

    async for token in callback.aiter():
        # print('token', token)
        yield f"{token}"

    await task

app = FastAPI()

@app.get("/")
async def homepage():
    return FileResponse('static/index.html')

@app.post("/ask")
def ask(body: dict):
    return StreamingResponse(call_openai(body['question']), media_type="text/event-stream")

if __name__ == "__main__":
   uvicorn.run(host="127.0.0.1", port=8888, app=app)
```

RuntimeWarning: Enable tracemalloc to get the object allocation traceback

Thought: Do I need to use a tool? No
AI: 你好！很高兴认识你！

> Finished chain.
INFO:     127.0.0.1:65193 - "POST /ask HTTP/1.1" 200 OK
conversation_agent <coroutine object Chain.acall at 0x1326b78a0>
coroutine <async_generator object AsyncIteratorCallbackHandler.aiter at 0x133b49340>


> Entering new AgentExecutor chain...

Thought: Do I need to use a tool? Yes
Action: FAQ
Action Input: 如何更改帐户信息error faq() takes 1 positional argument but 2 were given

### Suggestion:

_No response_
Custom Tool Output
Labels: name: auto:question, name: area: models

### Issue you'd like to raise.

Hi guys,

Here is the situation, I have 3 tools to use one by one. But, the outputs of which are not in the same types. Say:

Tool 1: output normally, and can be further thought by GPT in the chain,
Tool 2: output specially, which means the output should not be thought further by GPT again, because in that way the output format and data will be not correctly,
Tool 3: output normally, like Tool 1.

In this way, I found it hard to get the correct answer for myself. If all the tools are set to return_direct=False, the answer to Tool 2 will not be right, and if I set return_dircect=True, the chain of Tools 1->2->3 would be lost...

What should I do?

Any help will be highly appreciated.
Best

### Suggestion:

_No response_
Minor redis improvements
Labels: name: area: vector store, name: auto:improvement, name: integration: redis

  - **Description:**
      - Fixes a `key_prefix` bug where passing it in on `Redis.from_existing(...)` did not work properly. Updates doc strings accordingly.
      - Updates Redis filter classes logic with best practices on typing, string formatting, and handling "empty" filters.
      - Fixes a bug that would prevent multiple tag filters from being applied together in some scenarios.
      - Added a whole new filter unit testing module. Also updated code formatting for a number of modules that were failing the `make` commands.
  - **Issue:** N/A
  - **Dependencies:** N/A
  - **Tag maintainer:** @baskaryan 
  - **Twitter handle:** @tchutch94
docs: install nit
Labels: name: auto:documentation

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Set openai_api_type to be passed as values to leverage azure openai's default
Labels: name: area: embeddings, name: auto:improvement

  - **Description:** 
Recently, this was introduced: https://github.com/langchain-ai/langchain/pull/10707

However, the changes are not taking effect unless you create the AzureOpenAIEmbeddings class with parameter openai_api_type="azure".. In the code it's already getting set by default if openAI version > 1 but it's never getting accessed when the embed_documents is actually happening because it was never surfaced up
  - **Issue:** Not so much an issue but leveraging existing default code of api type of azure when used.
  - **Twitter handle:** @kevin_neum
Updated VertexAIModelGarden class generate method vertexai.py to align with VertexAI API
Labels: name: auto:bug, name: area: models

Replace this entire comment with:
  - **Description:** corrects handling of Prediction object returned from the predict method of the PredictionServiceClient in _generate method of the VertexAIModelGarden class
  - **Issue:** #13370,
  - **Dependencies:** None
fix cli release
Labels: name: auto:nit

My thought is that the ==version would prevent pip from finding the package on regular [pypi.org](http://pypi.org/), so it would look at [test.pypi.org](http://test.pypi.org/) for that. Otherwise it'll pull package from [pypi.org](http://pypi.org/) (e.g. sub deps)

Right now, the cli release is failing because it's going to test.pypi.org by default, so it finds this incorrect FASTAPI package instead of the real one: https://test.pypi.org/project/FASTAPI/
fmt
Labels: name: auto:nit

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
VertexAIModelGarden _generate method not in sync with VertexAI API
Labels: name: auto:bug, name: area: models

### System Info

LangChain version: 
Platform: WSL for Windows (Linux 983G3J3 5.15.90.1-microsoft-standard-WSL2)
Python version: 3.10.6

### Who can help?

@agola11 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1a. Create a project in GCP and deploy an open source model from Vertex AI Model Garden (follow the provided Colab notebook to deploy to an endpoint)
1b. Instantiate a VertexAIModelGarden object 
```python
llm = VertexAIModelGarden(project=PROJECT_ID, endpoint_id=ENDPOINT_ID)
```
2. Create a prompt string
```python
prompt = "This is an example prompt"
```
3. Call the generate method on the VertexAIModelGarden object
```python
llm.generate([prompt])
```
4. The following error will be produced:
```python
../python3.10/site-packages/langchain/llms/vertexai.py", line 452, in <listcomp>
    [Generation(text=prediction[self.result_arg]) for prediction in result]
TypeError: string indices must be integers
```

### Expected behavior

Expecting the generate method to return an LLMResult object that contains the model's response in the 'generations' property

In order to align with Vertex AI api the _generate method should iterate through response.predictions and set text property of Generation object to the iterator variable since response.predictions is a list data type that contains the output strings.
```python
for result in response.predictions:
            generations.append(
                [Generation(text=result)]
            )
```
Fix the bedrock documentation
Labels: name: auto:documentation, name: area: models

  - **Description:** As I was working through bedrock llm for RetreivalQAChain I noticed that the imports were wrong and is corrected for documentation.
  - **Issue:** As mentioned in the documentation for langchain. The import says from `langchain.llms import Bedrock` which should rather be `from langchain.llms.bedrock import Bedrock`
  - **Dependencies:** None
module 'openai' has no attribute 'error'
Labels: name: area: embeddings, name: auto:bug

### System Info

openai==1.2.4
langchain==0.0.325
llama_index==0.8.69

### Who can help?

@hwchase17 @agola11 

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [X] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
/usr/local/lib/python3.10/site-packages/llama_index/indices/base.py:102: in from_documents
    return cls(
/usr/local/lib/python3.10/site-packages/llama_index/indices/vector_store/base.py:49: in __init__
    super().__init__(
/usr/local/lib/python3.10/site-packages/llama_index/indices/base.py:71: in __init__
    index_struct = self.build_index_from_nodes(nodes)
/usr/local/lib/python3.10/site-packages/llama_index/indices/vector_store/base.py:254: in build_index_from_nodes
    return self._build_index_from_nodes(nodes, **insert_kwargs)
/usr/local/lib/python3.10/site-packages/llama_index/indices/vector_store/base.py:235: in _build_index_from_nodes
    self._add_nodes_to_index(
/usr/local/lib/python3.10/site-packages/llama_index/indices/vector_store/base.py:188: in _add_nodes_to_index
    nodes = self._get_node_with_embedding(nodes, show_progress)
/usr/local/lib/python3.10/site-packages/llama_index/indices/vector_store/base.py:100: in _get_node_with_embedding
    id_to_embed_map = embed_nodes(
/usr/local/lib/python3.10/site-packages/llama_index/indices/utils.py:137: in embed_nodes
    new_embeddings = embed_model.get_text_embedding_batch(
/usr/local/lib/python3.10/site-packages/llama_index/embeddings/base.py:250: in get_text_embedding_batch
    embeddings = self._get_text_embeddings(cur_batch)
/usr/local/lib/python3.10/site-packages/llama_index/embeddings/langchain.py:82: in _get_text_embeddings
    return self._langchain_embedding.embed_documents(texts)
/usr/local/lib/python3.10/site-packages/langchain/embeddings/openai.py:490: in embed_documents
    return self._get_len_safe_embeddings(texts, engine=self.deployment)
/usr/local/lib/python3.10/site-packages/langchain/embeddings/openai.py:374: in _get_len_safe_embeddings
    response = embed_with_retry(
/usr/local/lib/python3.10/site-packages/langchain/embeddings/openai.py:100: in embed_with_retry
    retry_decorator = _create_retry_decorator(embeddings)
/usr/local/lib/python3.10/site-packages/langchain/embeddings/openai.py:47: in _create_retry_decorator
    retry_if_exception_type(openai.error.Timeout)
E   AttributeError: module 'openai' has no attribute 'error'
```

### Expected behavior

I suppose it should run, I'll provide some reproducible code here in a minute.
Add limit_to_domains to APIChain based tools
Labels: name: area: agent, name: lgtm, name: auto:improvement

  - **Description:**  Adds `limit_to_domains` param to the APIChain based tools  (open_meteo, TMDB, podcast_docs, and news_api)
  - **Issue:** I didn't open an issue, but after upgrading to 0.0.328 using these tools would throw an error.
  - **Dependencies:** N/A
  - **Tag maintainer:** @baskaryan 
  
  
 **Note**: I included the trailing / simply because the docs here did https://github.com/langchain-ai/langchain/blob/fc886cc3039c2479bb13287f48fcbdb097a44c70/docs/docs/use_cases/apis.ipynb#L246 , but I checked the code and it is using `urlparse`. SoI followed the docs since it comes down to stylee.
PGVector don't have indices what kills postgres in production
Labels: name: area: vector store, name: auto:bug, name: auto:improvement

### System Info

after getting million embedding records in postgres, everything became ridiculously slow and postgres cpu usage went to 100%
fix was simple:
```
CREATE INDEX CONCURRENTLY langchain_pg_embedding_collection_id ON langchain_pg_embedding(collection_id);
CREATE INDEX CONCURRENTLY langchain_pg_collection_name ON langchain_pg_collection(name);
```
I think it's important to include index creation in basic setup.

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Just run PGVector, and it will create tables without indices.

### Expected behavior

Should create indices