delayedkarma/langchain-issues huggingface dataset:
Use "parse_file" in class "ConversationBufferMemory" raise a ValidationError
Labels: name: area: memory, name: auto:bug

### System Info

Platform:Windows
LangChain:0.0.335
python:3.9.12

### Who can help?

@eyurtsev @hwchase17 @hwchase17

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [X] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Here is my code:

from langchain.memory import ConversationBufferMemory
import langchain.load

ChatMemory = ConversationBufferMemory()
ChatMemory.chat_memory.add_ai_message("Hello")
ChatMemory.chat_memory.add_user_message("HI")
tojson = ChatMemory.json()

with open('E:\\memory.json', 'w') as file:
    file.write(tojson)
    file.close()

ParserChatMemory = ConversationBufferMemory.parse_file('E:\\memory.json')

### Expected behavior

Regarding the parse_file method in ConversationBufferMemory, there is not much introduction in the document, and I have not found a similar problem description. However, saving Memory into a file/or generating it from a file is a normal requirement, but when I execute parse_file This method throws a "ValidationError", the complete description is as follows:

ValidationError: 1 validation error for ConversationBufferMemory
chat_memory
  instance of BaseChatMessageHistory expected (type=type_error.arbitrary_type; expected_arbitrary_type=BaseChatMessageHistory)

If there is any violation of my operation, please tell me, thank you very much
qa retrieval
Labels: name: area: embeddings, name: auto:question, name: area: models

### Issue you'd like to raise.

i should qa over an document using retrieval. i should not use any large language model. i can only use embedding model. here is my code:

```
# import
from langchain.embeddings.sentence_transformer import SentenceTransformerEmbeddings
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import Chroma
from langchain.document_loaders import TextLoader
from silly import no_ssl_verification
from langchain.embeddings.huggingface import HuggingFaceEmbeddings
from langchain.chat_models import ChatOpenAI
from langchain.prompts import ChatPromptTemplate
from langchain.schema import StrOutputParser
from langchain.schema.runnable import RunnablePassthrough
from langchain.chains import RetrievalQA
from langchain.document_loaders import TextLoader
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.llms import OpenAI
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import Chroma


with no_ssl_verification():
    # load the document and split it into chunks
    loader = TextLoader("state_of_the_union.txt")
    documents = loader.load()

    # split it into chunks
    text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
    docs = text_splitter.split_documents(documents)

    # create the open-source embedding function
    embedding_function = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")
    # hfemb = HuggingFaceEmbeddings()
    # load it into Chroma
    db = Chroma.from_documents(docs, embedding_function)
    # query it
    query = "What did the president say about Ketanji Brown Jackson"
    docs = db.similarity_search(query)

    # print results
    print(docs[0].page_content)
    qa = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type="stuff", retriever=docsearch.as_retriever())

    query = "What did the president say about Ketanji Brown Jackson"
    qa.run(query)









```

how can i change this code with not using any llm in "qa = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type="stuff", retriever=docsearch.as_retriever())"




### Suggestion:

_No response_
turkish embedding
Labels: name: area: embeddings, name: auto:question, name: integration: chroma

### Issue you'd like to raise.

i have the following code for q&a system with retrieval mechanism:

```
# import
from langchain.embeddings.sentence_transformer import SentenceTransformerEmbeddings
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import Chroma
from langchain.document_loaders import TextLoader
from silly import no_ssl_verification
from langchain.embeddings.huggingface import HuggingFaceEmbeddings

with no_ssl_verification():
    # load the document and split it into chunks
    loader = TextLoader("paul_graham/paul_graham_essay.txt")
    documents = loader.load()

    # split it into chunks
    text_splitter = CharacterTextSplitter(chunk_size=2000, chunk_overlap=0)
    docs = text_splitter.split_documents(documents)

    # create the open-source embedding function
    embedding_function = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")
    # hfemb = HuggingFaceEmbeddings()
    # load it into Chroma
    db = Chroma.from_documents(docs, embedding_function)

    # query it
    query = "What were the two main things the author worked on before college?"
    docs = db.similarity_search(query)

    # print results
    print(docs[0].page_content)
```


i should do this retrieval in turkish dataset. so i should use turkish embeddings. how can i do that in my code?

### Suggestion:

_No response_
FAISS save_local / load_local don't be aware of avx2
Labels: name: area: vector store, name: auto:bug

### System Info

```
cat /proc/version
=> Linux version 5.15.107+
cat /etc/debian_version
=> 10.13
import langchain
langchain.__version__
=> '0.0.334'
```

I spent some time to debug why the function signature `search` is different between linux / macos m1.
I found that in macos, because it has arm arch, doesn't have swigfaiss_avx2.py file on download wheel.
Besides, in linux, because it has amd64 arch, which support avx2, has swigfaiss_avx2.py.

Here's the amd64 example:
If I make faiss vector store using `from_documents` like

```py
from langchain.embeddings import OpenAIEmbeddings

docs = [] # some documents
vs = FAISS.from_documents(docs, OpenAIEmbeddings())
vs.index
# <faiss.swigfaiss_avx2.IndexFlatL2; proxy of <Swig Object of type 'faiss::IndexFlatL2 *' at> >
vs.index.search
<bound method handle_Index.<locals>.replacement_search of <faiss.swigfaiss_avx2.IndexFlatL2; proxy of <Swig Object of type 'faiss::IndexFlatL2 *' at> >>
```
It is delived from swigfaiss_avx2's IndexFlatL2, and it's search method is replaced by `replacement_search` in faiss.__init__.py intentionally.

But, when we save this to local and load like:

```py
vs.save_local("faiss", "abcd.index")
vs2 = FAISS.load_local("faiss", OpenAIEmbeddings(), "abcd.index")
vs2.index
<faiss.swigfaiss.IndexFlat; proxy of <Swig Object of type 'faiss::IndexFlat *' at> >
vs2.index.search
<bound method IndexFlat.search of <faiss.swigfaiss.IndexFlat; proxy of <Swig Object of type 'faiss::IndexFlat *' at> >>
```

We can see that the vector store : vs2 is the IndexFlat of swigfaiss.py, not swigfaiss_avx2.py

I don't know whether the save process is problem, or the load process is problem, but I think it's quite a big bug because two `search` signatures are totally different.

```py
import inspect

inspect.signature(vs.index.search)
# <Signature (x, k, *, params=None, D=None, I=None)>
inspect.signature(vs2.index.search)
# <Signautre (n, x, k, distances, labels, params=None)>
```

Besides, I found the `FAISS_NO_AVX2` is not work, too because when using the flag,
the `replacement_search` is not wrapped the original `search` method at all.
like explained in https://github.com/langchain-ai/langchain/issues/8857

So, how can I use the local save / load mechanism in amd64 arch?
(arm doesn't have any problem) 

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

You can refer to my issue above.

### Expected behavior

Can use save/load faiss vectorstore machanism with avx2 IndexFlat in amd64 so that method `search` can wrapped properly by `replacement_search`
update pdf document loaders' metadata source to url for online pdf
Labels: name: lgtm, name: area: doc loader, name: auto:improvement

  - **Description:** Update 5 pdf document loaders in `langchain.document_loaders.pdf`, to store a url in the metadata (instead of a temporary, local file path) if the user provides a web path to a pdf: `PyPDFium2Loader`, `PDFMinerLoader`, `PDFMinerPDFasHTMLLoader`, `PyMuPDFLoader`, and `PDFPlumberLoader` were updated.
      - The updates follow the approach used to update `PyPDFLoader` for the same behavior in #12092 
      - The `PyMuPDFLoader` changes required additional work in updating `langchain.document_loaders.parsers.pdf.PyMuPDFParser` to be able to process either an `io.BufferedReader` (from local pdf) or `io.BytesIO` (from online pdf)
      - The `PDFMinerPDFasHTMLLoader` change used a simpler approach since the metadata is assigned by the loader and not the parser
  - **Issue:** Fixes #7034
  - **Dependencies:** None


```python
# PyPDFium2Loader example:
# old behavior
>>> from langchain.document_loaders import PyPDFium2Loader
>>> loader = PyPDFium2Loader('https://arxiv.org/pdf/1706.03762.pdf')
>>> docs = loader.load()
>>> docs[0].metadata
{'source': '/var/folders/7z/d5dt407n673drh1f5cm8spj40000gn/T/tmpm5oqa92f/tmp.pdf', 'page': 0}

# new behavior
>>> from langchain.document_loaders import PyPDFium2Loader
>>> loader = PyPDFium2Loader('https://arxiv.org/pdf/1706.03762.pdf')
>>> docs = loader.load()
>>> docs[0].metadata
{'source': 'https://arxiv.org/pdf/1706.03762.pdf', 'page': 0}
```
AWS LangChain using bedrock: Setting Relevance Score Function
Labels: name: area: embeddings, name: area: vector store, name: auto:bug, name: integration: aws

### System Info

Thank you for your great help. I have issue with setting the score function (I have AWS LangChain using bedrock). What I have is:

from langchain.vectorstores import FAISS
loader = CSVLoader("./rag_data/a.csv") 
documents_aws = loader.load() 
docs = CharacterTextSplitter(chunk_size=2000, chunk_overlap=400, separator=",").split_documents(documents_aws)

def custom_score(i):
    #return 1 - 1 / (1 + np.exp(i))
    return 1

vectorstore_faiss_aws = FAISS.from_documents(documents=docs,embedding = br_embeddings)
vectorstore_faiss_aws.relevance_score_function=custom_score


This made no difference in the scores (did not give any errors either as I am getting large negative numbers and the vectorstore_faiss_aws.similarity_search_with_relevance_scores is indifferent to the score_threshold value)


Then I tried:
vectorstore_faiss_aws = FAISS(relevance_score_fn=custom_score).from_documents(documents=docs,embedding = br_embeddings)

and it gave the following error:
 FAISS.__init__() missing 4 required positional arguments: 'embedding_function', 'index', 'docstore', and 'index_to_docstore_id'


Your advice is highly appreciated

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [X] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

The code is included:

from langchain.vectorstores import FAISS
loader = CSVLoader("./rag_data/a.csv") 
documents_aws = loader.load() 
docs = CharacterTextSplitter(chunk_size=2000, chunk_overlap=400, separator=",").split_documents(documents_aws)

def custom_score(i):
    #return 1 - 1 / (1 + np.exp(i))
    return 1

vectorstore_faiss_aws = FAISS.from_documents(documents=docs,embedding = br_embeddings)
vectorstore_faiss_aws.relevance_score_function=custom_score

### Expected behavior

The score to be in the range of 0 and 1 and the vectorstore_faiss_aws.similarity_search_with_relevance_scores to react to score_threshold
https://python.langchain.com/docs/modules/model_io/models/llms/integrations/sagemaker 
Labels: name: area: embeddings, name: auto:bug, name: auto:documentation, name: integration: aws

### Issue with current documentation:

I am  try to use sagemaker endpoint to use the embedding model, but I am confused about using EmbeddingsContentHandler, after trying I figure out how to define the transform_input function, but the transform_output did not work as expected, it give out some errors, which like 
    **embeddings = response_json[0]["embedding"]
TypeError: list indices must be integers or slices, not str**,  I would be really garteful it someone konw how to sove it
my ContentHandler function is :
class ContentHandler(EmbeddingsContentHandler):
    content_type = "application/json"
    accepts = "application/json"

    def transform_input(self, prompt: str, model_kwargs={}) -> bytes:
        input_str = json.dumps({"inputs": prompt, **model_kwargs})
        return input_str.encode("utf-8")

    def transform_output(self, output: bytes) -> str:
        response_json = json.loads(output.read().decode("utf-8"))
        embeddings = response_json[0]["embedding"]
        return embeddings


### Idea or request for content:

_No response_
Chain/Agent Builder
Labels: name: area: agent, name: area: embeddings, name: auto:enhancement, name: auto:documentation

### Feature request

The LangChail modules can be naturally used to build something like OpenAI GPTs builder.
My understanding is, LangChain needs to add descriptions and description_embeddings to all integrations/chains/agents (not only to the tools). It allows to build the Super Agent aka Agent Builder. 

### Motivation

LangChain's space (in terms of integrations/chains/agents) is bigger than OpenAI's. Let's use this.

### Your contribution

I can help with documentations, examples,ut-s
Support Vald secure connection
Labels: name: lgtm, name: area: vector store, name: auto:improvement, name: size:L

 **Description:** 
When using Vald, only insecure grpc connection was supported, so secure connection is now supported.
In addition, grpc metadata can be added to Vald requests to enable authentication with a token.

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Issue: gpt3.5-turbo-1106 responds differently from the OpenAI playground 
Labels: name: auto:bug, name: area: models

### Issue you'd like to raise.

- Langchain version: 0.0.335
- Affected LLM: chatgpt 3.5 turbo 1106 (NOT affecting: gpt 3.5 turbo 16k, gpt4, gpt4 turbo preview)
- Code template, similar to:

`llm=ChatOpenAI(model_name='gpt-3.5-turbo-1106',
                         openai_api_key=openai_key
                         )`
`memory = ConversationBufferMemory()`
`conversation = ConversationChain(
        llm=llm, memory=memory
    )`

`conversation.predict(input='my instructions')
`

**The problem**
When the identical instructions and parameters are used in the API and the open ai playground, model responds radically different. The open ai playground responds as expected, producing answers. While the API through langchain always returns 'I cannot fufil your request'.

**suspected causes**
Open AI made an upgrade to its API to 1.x, and as far as I know Langchain now only works with open ai 0.28.1. Could that be the reason?

Thanks


**UPDATE 13 Nov:**
I further tested my prompts with OpenAI api 1.2.3 alone, without langchain and everything works fine:
- First I put my prompts into the OpenAI playground, which produced responses
- Second I 'view source' > 'copy', put the python code into a .py file, edited out the last AI response and ran the code = I got the response I wanted.
- Third I tested the same prompt with langchain 0.0.335 (openai 0.28.1) this time, and I always get 'I cannot fulfil this request'.

I know that langchain adds a little bit of context to every request and my prompts did consist of two rounds of chats and thus my code uses memory. I don't know which part of this is problematic.

Again, this problem does not occur with gpt-3.5-turbo, or gpt-4, or gpt-4-turbo, only **'gpt-3.5-turbo-1106'**


### Suggestion:

_No response_
Updated DocugamiLoader with better support for hierarchical chunks
Labels: name: lgtm, name: area: doc loader, name: auto:improvement, name: size:XXL

There are the following main changes in this PR:

1. Rewrite of the DocugamiLoader to not do any XML parsing of the DGML format internally, and instead use the dgml-utils library we are separately working on. This is a very lightweight dependency.
2. Added MMR search type as an option to multi-vector retriever, similar to other retrievers. MMR is especially useful when using Docugami for RAG since we deal with large sets of documents within which a few might be duplicates and straight similarity based search doesn't give great results in many cases.

We are @docugami on twitter, and I am @tjaffri
Import error in lanchain/vectorstores/azuresearch.py
Labels: name: area: vector store, name: auto:bug

I am testing a simple RAG implementation with Azure Cognitive Search. I am seeing a "cannot import name 'Vector' from azure.search.documents.models" error when I invoke my chain. Origin of my error is line 434 in lanchain/vectorstores/azuresearch.py (from azure.search.documents.models import Vector)

this is the relevant code snippet, I get the import error when I execute rag_chain.invoke(question)

from langchain.schema.runnable import RunnablePassthrough
from langchain.prompts import ChatPromptTemplate
from langchain.chat_models.azure_openai import AzureChatOpenAI

question = "my question.."

-- vector_store is initialized using AzureSearch(), not including that snippet here --
retriever = vector_store.as_retriever()

template = ''' 
Answer the question based on the following context: 
{context}

Question: {question} 
'''

prompt = ChatPromptTemplate.from_template(template=template)

llm = AzureChatOpenAI( deployment_name='MY_DEPLOYMENT_NAME', model_name='MY_MODEL', openai_api_base=MY_AZURE_OPENAI_ENDPOINT, openai_api_key=MY_AZURE_OPENAI_KEY, openai_api_version='2023-05-15', openai_api_type='azure' )

rag_chain = {'context' : retriever, 'question' : RunnablePassthrough} | prompt | llm 
rag_chain.invoke(question)

--------------

my package versions

langchain==0.0.331
azure-search-documents==11.4.0b11
azure-core==1.29.5
openai==0.28.1

_Originally posted by @yallapragada in https://github.com/langchain-ai/langchain/discussions/13245_
Add remove_comments option (default True): do not extract html comments
Labels: name: auto:improvement

  - **Description:** add `remove_comments` option (default: True): do not extract html _comments_, 
  - **Issue:** None,
  - **Dependencies:** None,
  - **Tag maintainer:** @nfcampos ,
  - **Twitter handle:** peter_v

I ran `make format`, `make lint` and `make test`.

Discussion: I my use case, I prefer to not have the comments in the extracted text:
* e.g. from a Google tag that is added in the html as comment
* e.g. content that the authors have temporarily hidden to make it non visible to the regular reader

Removing the comments makes the extracted text more alike the intended text to be seen by the reader.


**Choice to make:** do we prefer to make the default for this `remove_comments` option to be True or False?
I have changed it to True in a second commit, since that is how I would prefer to use it by default. Have the
cleaned text (without technical Google tags etc.) and also closer to the actually visible and intended content.
I am not sure what is best aligned with the conventions of langchain in general ...


INITIAL VERSION (new version above):
~**Choice to make:** do we prefer to make the default for this `ignore_comments` option to be True or False?
I have set it to False now to be backwards compatible. On the other hand, I would use it mostly with True.
I am not sure what is best aligned with the conventions of langchain in general ...~
Added support for a Pandas DataFrame OutputParser
Labels: name: auto:enhancement, name: size:XL

**Description:**

Added support for a Pandas DataFrame OutputParser with format instructions, along with unit tests and a demo notebook. Namely, we've added the ability to request data from a DataFrame, have the LLM parse the request, and then use that request to retrieve a well-formatted response.

Within LangChain, it seamlessly integrates with language models like OpenAI's `text-davinci-003`, facilitating streamlined interaction using the format instructions (just like the other output parsers).

This parser structures its requests as `<operation/column/row>[<optional_array_params>]`. The instructions detail permissible operations, valid columns, and array formats, ensuring clarity and adherence to the required format.

For example:

- When the LLM receives the input: "Retrieve the mean of `num_legs` from rows 1 to 3."
- The provided format instructions guide the LLM to structure the request as: "mean:num_legs[1..3]".

The parser processes this formatted request, leveraging the LLM's understanding to extract the mean of `num_legs` from rows 1 to 3 within the Pandas DataFrame.

This integration allows users to communicate requests naturally, with the LLM transforming these instructions into structured commands understood by the `PandasDataFrameOutputParser`. The format instructions act as a bridge between natural language queries and precise DataFrame operations, optimizing communication and data retrieval.

**Issue:**

- https://github.com/langchain-ai/langchain/issues/11532

**Dependencies:**

No additional dependencies :)

**Tag maintainer:**

@baskaryan 

**Twitter handle:**

No need. :)