delayedkarma/langchain-issues huggingface dataset:
Why can't test117 be retrieved by vector and provide replies for reference to achieve long-term memory?
Labels: name: area: vector store, name: auto:bug

### Issue with current documentation:

import os
from langchain_community.chat_models import ChatOpenAI
from langchain.schema import Document
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import Qdrant
from langchain.chains import ConversationChain
from langchain.memory import ConversationBufferMemory
from qdrant_client import QdrantClient, models

os.environ['OPENAI_API_KEY'] = "mykey"
embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
client = QdrantClient(host="192.168.10.31", port=6333)
collection_name = "test117"

client.create_collection(
    collection_name=collection_name,
    vectors_config=models.VectorParams(size=384, distance=models.Distance.COSINE),
    optimizers_config=models.OptimizersConfigDiff(memmap_threshold=20000),
)
vectorstore = Qdrant(
    client=client,
    embedding=embeddings,
    location=":memory:",
    collection_name=collection_name,
)
llm = ChatOpenAI(
    temperature=0.7,
    model="gpt-3.5-turbo",
    max_tokens=100
)

memory = ConversationBufferMemory()
conversation = ConversationChain(llm=llm, memory=memory, verbose=True,)
collection_info = client.get_collection(collection_name)

while True:
    chat_history = memory.load_memory_variables({})
    question = input('ask：')
    question_vector = embeddings.embed_query(question)[0]

    search_results = client.search(
        collection_name=collection_name,
        query_vector=models.NamedSparseVector(
            name=collection_name,
            vector=models.SparseVector(
                indices=question_vector.indices,
                values=question_vector.values,
            ),
        ),
        limit=1,
    )
    if search_results and search_results[0].payload:
        response_vector = search_results[0].payload.vector
        response_str = vectorstore.retrieve_text(response_vector)
    else:
        response = conversation.predict(input=f"Q: {question}\nA:")
        response_str = response.choices[0].text.strip()
        response_vector = embeddings.embed_query(response_str)[0]
        vectorstore.add_documents([Document(vector=response_vector, text=response_str)])
    print("respones：", response_str)
    memory.add_memory_variables({"input": question, "output": response_str})
    if question.lower() == 'bye':
        break


### Idea or request for content:

Why can't test117 be retrieved by vector and provide replies for reference to achieve long-term memory?
Is it that the retriever is not written well?
Add Tuple support for messages placeholder

It's a bit annoying to work with LangChain messages on the frontend, which is why we [did things like this in chat langchain](https://github.com/langchain-ai/chat-langchain/blob/592d588a1bfe35fe03ed33ac8ba36afd9452bb6d/chain.py#L148). It would be convenient if the messages placeholder also accepted the same tuple format as the prompt template, so you could just pass the role and content values around.

Maybe not as useful in a real memory setup, but simplifies the user code a bit in some situations.
community: Fix LlamaContentFormatter to match actual input/output formatting
Labels: name: auto:bug, name: area: models, name: size:XS

 **Description:** The `LlamaContentFormatter` in the `langchain_community.chat_models.azureml_endpoint` does not work because it doesn't actually comply to the output of Llama models. The input/output formats are documented here: 

[Deploy Llama2 Chat Models - Chat API](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/deploy-models-llama?tabs=azure-studio#chat-api)

This PR creates a LlamaContentFormatter that works with the documented API.

 **Issue:** N/A
 **Dependencies:** N/A
 **Twitter handle:** parkerdhancock
refactor `langchain.retrievers.self_query`
Labels: name: auto:refactor, name: integration: chroma, name: size:XXL, name: partner

**Issue:** all classes in `langchain.retrievers.self_query` namespace belong to the `community` package.

**Following issue:**
We cannot just move these classes to `community` because of the dependency on the Visitor class and other classes from the `langchain.chains.query_constructor` namespace.
What is interesting is that `langchain.chains.query_constructor` artifacts are used ONLY in the `langchain.retrievers.self_query` artifacts.
It happens that `langchain.chains.query_constructor` artifacts do not depend on the `langchain.chains`. Actually the `langchain.chains.query_constructor` is quite independent. Because of that we can move `langchain.chains.query_constructor` under `langchain_core` as the `langchain_core.sql_constructor`
The reason for renaming `query_constructor` into `sql_constructor` is simple. It IS the SQL constructor. Moreover, the `query_..` is the ambiguous name because we use `query` in retrievers and many other places not as the SQL queries but as the queries in prompts. `sql` is unambiguous name here.

NOTE: The `langchain.chains.query_constructor.base.load_query_constructor_chain` is not moved! This is the only dependency to `chains`. It is not important because Chains will be depricated soon.
structured output parser on ChatPromptTemplate.from_template()
Labels: name: auto:question, name: area: models

### Checked other resources

- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a similar question and didn't find it.

### Example Code

```python
llm = VertexAI(model_name='gemini-pro', max_output_tokens=512, temperature=0.2)
data = loader.load()
embeddings = VertexAIEmbeddings(model_name='textembedding-gecko@003')
vectorstore = Chroma.from_documents(data, embeddings)
retriever = vectorstore.as_retriever()

template = """
Use the following pieces of context to answer the question at the end.
- If you think you need more information to find proper answer, ask user to clarify.
- If you cannot find a proper answer from the reference, just say we don't have a data from the reference.
Use three sentences maximum and keep the answer as concise as possible.

Context: {context}
Question: {question}
Helpful Answer:

**Resource:** [reference source name]
"""

found_response_schemas = [
    ResponseSchema(name="answer", description="answer to the user's question"),
    ResponseSchema(
        name="found",
        description="whether the model could find the proper answers or not.",
    ),
]
found_output_parser = StructuredOutputParser.from_response_schemas(found_response_schemas)
format_instructions = found_output_parser.get_format_instructions()

rag_prompt_custom = ChatPromptTemplate.from_template(template)
rag_chain = (
    {"context": retriever, "question": RunnablePassthrough()}
    | rag_prompt_custom
    | llm
    | StrOutputParser()
)

response = rag_chain.invoke(message)
    
print(response)
```

### Description

I am trying to add StructuredOutputParser with a specific ResponseSchema.
What I would like to add as a ResponseSchema is, adding boolean value to check whether the llm could find proper information from reference document in RAG.


### System Info

langchain==0.0.353

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [X] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async
RedisVectorStoreRetriever similarity search similarity_distance_threshold is not implemented asyncronously
Labels: name: area: vector store, name: auto:bug, name: integration: redis

### Checked other resources

- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a similar question and didn't find it.

### Example Code

```python
from langchain_community.vectorstores.redis import Redis

embedding = ...
redis_url = ...
schema_path = ...


vectorstore = Redis.from_existing_index(
    embedding,
    index_name='index',
    redis_url=redis_url,
    schema=schema_path,
)
retriever = vectorstore.as_retriever(search_type="similarity_distance_threshold", search_kwargs={"distance_threshold": 0.9, "k": 5})
retriever.aget_relevant_documents('query')
```

### Description

The asynchronous method `_aget_relevant_documents` is not defined for `RedisVectorStoreRetriever`. As such, the method of the parent of `RedisVectorStoreRetriever` (`VectorStoreRetriever`) is called. In `VectorStoreRetriever` however, `similarity_distance_threshold` is not available. This makes the chain crash when `RedisVectorStoreRetriever`  is called asynchronously in a chain.

I will make a PR to solve this

### System Info

```
langchain                         0.1.0
langchain-community               0.0.12
langchain-core                    0.1.10
```

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [x] Async
core:Add optional max_messages to MessagePlaceholder
Labels: name: area: memory, name: auto:improvement, name: size:S

- **Description:** Add optional max_messages to MessagePlaceholder
- **Issue:** [16096](https://github.com/langchain-ai/langchain/issues/16096)
- **Dependencies:** None
- **Twitter handle:** @davedecaprio

Sometimes it's better to limit the history in the prompt itself rather than the memory.  This is needed if you want different prompts in the chain to have different history lengths.
Allow MessagesPlaceholder to include a maximum number of messages
Labels: name: area: memory, name: auto:improvement

### Feature request

When using `MessagesPlaceholder`, it always includes all messages.  For some prompts I want the full message history, but for others, I want to limit it to just the most recent few messages.

This can be accomplished with `ConversationBufferWindowMemory`, but that limits the memory storage which is used for all my LLM prompts in the chain.  

I currently accomplish this with a custom prompt generator, but it would be easier if there were an optional max_messages parameter to MessagesPlaceholder that limits the history to the given number of messages.

### Motivation

I have chain with multiple LLM calls.  For one, the routing prompt, it works best when passed the last few messages in the conversation.  Right now I have to use a custom prompt generator to accomplish this.  I'd really like to be able to just add a limit to MessagesPlaceholder.



### Your contribution

I will submit a PR,
docs: Updated SQLite example to use LCEL and SQLChatMessageHistory
Labels: name: lgtm, name: auto:documentation, name: area: lcel, name: size:L

  - **Description:** Updated the SQLite example integration notebook to latest standards
  - **Issue:** [15664](https://github.com/langchain-ai/langchain/issues/15664)
  - **Dependencies:** None
  - **Twitter handle:** @davedecaprio
Add unit test for overridden lc_namespace
Labels: name: lgtm, name: area: doc loader, name: auto:improvement, name: size:XS

<!-- Thank you for contributing to LangChain!

Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified.

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes if applicable,
  - **Dependencies:** any dependencies required for this change,
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/docs/integrations` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Update elasticsearch.ipynb
Labels: name: area: vector store, name: auto:documentation, name: integration: elasticsearch, name: size:XS

Fixed a typo, the parameter used for the Elasticsearch API key was called api_key, but the parameter is called es_api_key.

<!-- Thank you for contributing to LangChain!

Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified.

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes if applicable,
  - **Dependencies:** any dependencies required for this change,
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/docs/integrations` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
PDFInfoNotInstalledError: Unable to get page count. Is poppler installed and in PATH?
Labels: name: area: doc loader, name: auto:bug

### Checked other resources

- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a similar question and didn't find it.

### Example Code

``` Python 
from langchain_community.document_loaders import DirectoryLoader
directory = '/content/drive/MyDrive/History_QA_dataset'

def load_docs(directory):
    return DirectoryLoader(directory).load()
documents = load_docs(directory)

```

### Description

```
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/pdf2image/pdf2image.py](https://localhost:8080/#) in pdfinfo_from_path(pdf_path, userpw, ownerpw, poppler_path, rawdates, timeout, first_page, last_page)
    580             env["LD_LIBRARY_PATH"] = poppler_path + ":" + env.get("LD_LIBRARY_PATH", "")
--> 581         proc = Popen(command, env=env, stdout=PIPE, stderr=PIPE)
    582 

19 frames
FileNotFoundError: [Errno 2] No such file or directory: 'pdfinfo'

During handling of the above exception, another exception occurred:

PDFInfoNotInstalledError                  Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/pdf2image/pdf2image.py](https://localhost:8080/#) in pdfinfo_from_path(pdf_path, userpw, ownerpw, poppler_path, rawdates, timeout, first_page, last_page)
    605 
    606     except OSError:
--> 607         raise PDFInfoNotInstalledError(
    608             "Unable to get page count. Is poppler installed and in PATH?"
    609         )

PDFInfoNotInstalledError: Unable to get page count. Is poppler installed and in PATH?
```

### System Info

Windows: 11
pip == 23.3.1
python == 3.10.10
langchain == 0.1.0
transformers == 4.36.2
sentence_transformers == 2.2.2
unstructured == 0.12.0

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async
Make research assistant template work with OpenAI models on AzureAI 
Labels: name: auto:enhancement, name: area: models

### Feature request

I would like to use the research assistant template with OpenAI models on Azure. Can this be integrated into the template somehow ? I can of course modify the code post template expansion (ChatOpenAI -> AzureChatOpenAI etc.), but that would be lost upon changes / enhancements to the templates.

### Motivation

Cannot use OpenAI models directly due to company policy.

### Your contribution

Happy to spend some effort if someone can point me to a good way of adding e.g. a switch to the template (don't want to duplicate the whole template).
langchain: support more comparators in Milvus self-querying retriever
Labels: name: area: vector store, name: auto:improvement, name: size:M

  - **Description:** Support IN and LIKE comparators in Milvus self-querying retriever,  based on [Boolean Expression Rules](https://milvus.io/docs/boolean.md)
  - **Issue:** No
  - **Dependencies:** No
  - **Twitter handle:** No
Why can't the response refer to the vector data of the historical conversation in test116 for achieving long-term memory in replies?
Labels: name: area: memory, name: auto:question

### Issue with current documentation:

import os
from docarray.index.backends.qdrant import qdrant_client
from langchain.schema import Document
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import Qdrant
from qdrant_client import QdrantClient
from langchain.chains.query_constructor.base import AttributeInfo
from langchain.retrievers.self_query.base import SelfQueryRetriever
from langchain.chat_models import ChatOpenAI
from langchain.chains import ConversationChain
from langchain.memory import ConversationBufferMemory
os.environ['OPENAI_API_KEY'] = "mykey"
embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
client = QdrantClient(host="192.168.0.22", port=6333)

conversation_doc = Document(
    id="1",
    page_content="openhistory",
    metadata={
        "timestamp": "2023-01-16",
        "user": "Ai",
    },
)
print(conversation_doc)
vectorstore = Qdrant.from_documents(
    [conversation_doc],
    client=client,
    embedding=embeddings,
    location=":memory:",
    collection_name="test116",
)
# client.create_collection(
#     collection_name="test116",
#     vectors_config=models.VectorParams(
#         distance=models.Distance.COSINE,
#         size=384),
#     optimizers_config=models.OptimizersConfigDiff(memmap_threshold=20000),
#     hnsw_config=models.HnswConfigDiff(on_disk=True, m=16, ef_construct=100)
# )
print("找到ㄉtable:")
collections = client.get_collections()
for collection in collections:
    print(collection)
metadata_field_info = [
    AttributeInfo(
        name="openhistory",
        description="talk_history_embeddings",
        type="string or list[string]",
    )
]
document_content_description = "chat_history"
llm = ChatOpenAI(
    temperature=0.7,
    model="gpt-3.5-turbo",
    max_tokens=100
)
retriever = SelfQueryRetriever.from_llm(
    llm, vectorstore, document_content_description, metadata_field_info, verbose=True
)
memory = ConversationBufferMemory()
conversation = ConversationChain(llm=llm, memory=memory, verbose=True, )
while True:
    chat_history = memory.load_memory_variables({})
    client.get_collection(collection_name="test116")
    print("成功載入history")
    question = input('冒險者：')
    response = conversation.predict(input=f"Q: {question}\nA:")
    print(response)
    response_str = str(response[0])
    response_vector = embeddings.embed_query(response_str)[0]
    conversation_doc = Document(
        id="1",
        page_content=response,
        metadata={
            "user": f"{question}",
            "response_vector": response_vector
        },
    )
    vectorstore.add_documents([conversation_doc])
    print(conversation_doc)
    if question.lower() == 'bye':
        break


### Idea or request for content:

_No response_
Input to ChatPromptTemplate is missing variables {'input', 'agent_scratchpad'}. Expected: ['agent_scratchpad', 'input']
Labels: name: integration: weaviate, name: area: embeddings, name: auto:bug

### Checked other resources

- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a similar question and didn't find it.

### Example Code

This code gives me an weird error:

```
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnableLambda, RunnablePassthrough

def create_weaviate_new_RAG(client, embeddings, llm, class_name_rich, class_properties_rich, class_text_key):
    
    vector_store = Weaviate(client=client,
            index_name=class_name_rich,
            text_key=class_text_key,
            embedding=embeddings,
            attributes=class_properties_rich,
            #by_text=False,
        )

    retriever=vector_store.as_retriever(
                                search_type= "mmr", 
                                search_kwargs={
                                    "k": 4,
                                    "fetch_k": 20,
                                }
                            )

    prompt = ChatPromptTemplate.from_messages(
        [
            ("system", "You are a helpful assistant who retrieves information from documents"),
            ("user", "{input}"),
            MessagesPlaceholder(variable_name="agent_scratchpad"),
        ]
    )

    rag_pipe = (
        {"context": retriever, "question": RunnablePassthrough()}
        | prompt
        | llm
        | StrOutputParser()
    )

    return rag_pipe

rag_pipe = create_weaviate_new_RAG(client, embeddings, llm_chat, class_name, ["chunk_id", "doc_name", "doc_path"], "chunk_text")
text = "what is ai policy?"
rag_pipe.invoke(text)

```

Gives me this error:

```
"Input to ChatPromptTemplate is missing variables {'input', 'agent_scratchpad'}.  Expected: ['agent_scratchpad', 'input'] Received: ['context', 'question']"

```

I am using openai==1.7.0 and langchain==0.1.0

This inly started the appear after adding the new rag code from langchain. 

Can someone help?

Thank you :)

### Description

I expect the function to just run smoothly. Mot of the code is copied from the langchain documentation. 

### System Info

openai==1.7.0 and langchain==0.1.0

### Related Components

- [X] LLMs/Chat Models
- [X] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [X] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async
DOCS: Fix StreamlitChatMessageHistory docs to latest API
Labels: name: area: memory, name: auto:documentation, name: size:M

  - **Description:** Update [this page](https://python.langchain.com/docs/integrations/memory/streamlit_chat_message_history) to use the latest API
  - **Issue:** https://github.com/langchain-ai/langchain/issues/13995
  - **Dependencies:** None
  - **Twitter handle:** @OhSynap