delayedkarma/langchain-issues huggingface dataset:
How to use MongoDB to store vector embeddings and during extracting the relevant document, using `FAISS` for similarity search based document relevancy?
Labels: name: area: vector store, name: auto:question

### Issue you'd like to raise.

I am using FAISS vector store for storing the vector embeddings of the documents and retrieving the relevant document with respect to the query. My question is that `FAISS` does not store the embeddings permanently, instead it stores it in RAM and when running the code once again, it simply overwrites on the previous one or simply adds another index for it (I am not sure on this one). What I want is that to store these embeddings, say on `MongoDB` so that it gets stored there for longer period of time and when extracting those embeddings, `FAISS` is the algorithm to be used for similarity search based extraction (as it is best for large documents). Is it possible?

### Suggestion:

I tried `MongoDBAtlasVectorSearch` for this and came to know that it does store the vector embeddings in MongoDB database but when extracting the relevant document with respect to a query it does not use `FAISS` based similarity search algorithm (which is, I believe, better than the one used by `MongoDBAtlasVectorSearch`).
Bagatur/replicate stop
Labels: name: auto:improvement

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Cannot use CombinedMemory with VectorStoreRetrieverMemory and ConversationBufferMemory for ConversationalRetrievalChain

### System Info

1. Python 3.10
2. Langchain 0.0.240
3. Mac OSX Monterey Intel Processor

### Who can help?

@hwchase17 
@ago

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [X] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Hey so I've been trying to use `CombinedMemory` to combine the `VectorStoreRetrieverMemory` and `ConversationSummaryBufferMemory` and set it as the memory of a `ConversationalRetrievalChain`, but I keep running into issues and could not find any examples of this in the documentation. is there an example that could be provided by the documentation
Supporting Multiple Schemas for SQL Query Generation in Databricks
Labels: name: auto:improvement

### Feature request

There might be scenarios where transformations and joins will need to be applied on tables from different schemas but currently, the code only supports a single schema query structure.

### Motivation

I am working on Databricks Unity Catalog Query Assistant feature for a Product and need this change for the same, there are business scenarios where multiple schemas will need to be accessed to make complicated business rules to be applied.

### Your contribution

I can make the PR for this change, I have identified the code changes required
fix:get bilibili subtitles
Labels: name: lgtm, name: area: doc loader, name: auto:bug


  - Description: fix the Loader 'BiliBiliLoader'
  - Issue: the API response was changed
![image](https://github.com/langchain-ai/langchain/assets/2113954/91216793-82f8-4c82-a018-d49f36f5f6aa)
The previously used API no longer returns the "subtitle_url" property.
![image](https://github.com/langchain-ai/langchain/assets/2113954/a8ec2a7a-f40d-4c2a-b7d0-0ccdf2b327cc)
We should use another API to get `subtitle_url` property. 
The `subtitle_url` returned by this API does not include the http schema and needs to be added.

  - Dependencies: Nope
  - Tag maintainer: @rlancemartin
Add support for load_chat_model
Labels: name: auto:improvement

- Description:
  - Add `load_chat_model()` and `load_chat_model_from_config()` in `langchain.chat_model.loading` module
  - Update `_load_llm_chain()` to use the above functions in case of LLMs that are Chat Models
  - Fix warning while loading `AzureChatOpenAI` and `ChatOpenAI`: WARNING! engine is not default parameter.
  - **TODO**: I'll add/update unit tests once I get a go-ahead from maintainers for these new additions.
- Issue:
  - https://github.com/langchain-ai/langchain/issues/2627
  - Related PR that was closed: https://github.com/langchain-ai/langchain/pull/1715
- Dependencies: None
- Tag maintainer: @baskaryan
- Twitter handle: None
Adding Adobe PDF extraction API as an additional langchain document loader
Labels: name: area: doc loader, name: auto:enhancement

### Feature request

Hi langchain team!

I'd like to contribute this feature to the langchain document loaders. 

There are multiple pros for using Adobe API instead of the existing libraries for converting pdf to text and other metadata; e.g, adobe API allows for extraction of tables and figures in pdf documents as separate .csv and .png files, respectively. 
These are of course in addition to raw text extraction which in my opinion works better most of the time compared to other libraries. 

Here are a few links to examples and source code:
[Adobe python API example](https://developer.adobe.com/document-services/docs/overview/pdf-extract-api/quickstarts/python/)
[Adobe SDK](https://github.com/adobe/pdfservices-python-sdk-samples/tree/main/src/extractpdf)

### Motivation

I needed to extract the tables in pdf files in a standard format to extract the data in them more carefully. I was also not satisfied by the existing libraries performance on raw text extraction.

### Your contribution

I'd like to contribute this feature as a new document loader in langchain. Let me know if I can start working on it.
Thanks!
Add support for load_chat_model

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Migration OpenAIEmbedding from Azure to OpenAI reproduce gives an error message "ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions."
Labels: name: area: embeddings, name: auto:bug

### System Info

python = ">=3.9,<3.12"
fire = "0.5.0"
pandas = "2.0.2"
sqlalchemy = "2.0.15"
rich = "13.4.1"
mysqlclient = "2.1.1"
pandera = "0.15.1"
openai = "0.27.7"
guidance = "0.0.62"
langchain = "0.0.221"
textual = "^0.29.0"

### Who can help?

@hwchase17 @agola11 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [X] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. Prepare data. In our case, there are 3576 documents.
2. Migrate API
before migration:
```python
embeddings = OpenAIEmbeddings(
    model="text-embedding-ada-002",
    deployment=<MY_DEPLOYMENT_NAME>,
    chunk_size=1,
    openai_api_base=os.environ["OPENAI_API_BASE"],
    openai_api_key=os.environ["OPENAI_API_KEY"],
    openai_api_type="azure",
)
```
after migration:
```python
embeddings = OpenAIEmbeddings(
    model="text-embedding-ada-002",
    openai_api_key=os.environ["OPENAI_API_KEY"],
    openai_api_type="openai",
)
```
3. Make FAISS DB
The code are the same as that of before-migration version.
db = FAISS.from_documents(docs, embeddings)

4. Error occurs
```
Traceback (most recent call last):
  File "/path/make_dataframe.py", line 54, in <module>
    db = FAISS.from_documents(docs, embeddings)
  File "/path/.venv/lib/python3.9/site-packages/langchain/vectorstores/base.py", line 337, in from_documents
    return cls.from_texts(texts, embedding, metadatas=metadatas, **kwargs)
  File "/path/.venv/lib/python3.9/site-packages/langchain/vectorstores/faiss.py", line 554, in from_texts
    return cls.__from(
  File "/path/.venv/lib/python3.9/site-packages/langchain/vectorstores/faiss.py", line 506, in __from
    vector = np.array(embeddings, dtype=np.float32)
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (3576,) + inhomogeneous part.
```

### Expected behavior

Correctly construct FAISS DB without error.
Error when importing ChatOutputParser: no validator found for <class 're.Pattern'>
Labels: name: area: agent, name: auto:bug

### System Info

from langchain.agents.chat.output_parser import ChatOutputParser
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/chatpdf/Backend/backenv/lib/python3.10/site-packages/langchain/__init__.py", line 6, in <module>
    from langchain.agents import MRKLChain, ReActChain, SelfAskWithSearchChain
  File "/home/ubuntu/chatpdf/Backend/backenv/lib/python3.10/site-packages/langchain/agents/__init__.py", line 10, in <module>
    from langchain.agents.agent_toolkits import (
  File "/home/ubuntu/chatpdf/Backend/backenv/lib/python3.10/site-packages/langchain/agents/agent_toolkits/__init__.py", line 6, in <module>
    from langchain.agents.agent_toolkits.csv.base import create_csv_agent
  File "/home/ubuntu/chatpdf/Backend/backenv/lib/python3.10/site-packages/langchain/agents/agent_toolkits/csv/base.py", line 4, in <module>
    from langchain.agents.agent_toolkits.pandas.base import create_pandas_dataframe_agent
  File "/home/ubuntu/chatpdf/Backend/backenv/lib/python3.10/site-packages/langchain/agents/agent_toolkits/pandas/base.py", line 18, in <module>
    from langchain.agents.types import AgentType
  File "/home/ubuntu/chatpdf/Backend/backenv/lib/python3.10/site-packages/langchain/agents/types.py", line 5, in <module>
    from langchain.agents.chat.base import ChatAgent
  File "/home/ubuntu/chatpdf/Backend/backenv/lib/python3.10/site-packages/langchain/agents/chat/base.py", line 6, in <module>
    from langchain.agents.chat.output_parser import ChatOutputParser
  File "/home/ubuntu/chatpdf/Backend/backenv/lib/python3.10/site-packages/langchain/agents/chat/output_parser.py", line 12, in <module>
    class ChatOutputParser(AgentOutputParser):
  File "pydantic/main.py", line 229, in pydantic.main.ModelMetaclass.__new__
  File "pydantic/fields.py", line 491, in pydantic.fields.ModelField.infer
  File "pydantic/fields.py", line 421, in pydantic.fields.ModelField.__init__
  File "pydantic/fields.py", line 542, in pydantic.fields.ModelField.prepare
  File "pydantic/fields.py", line 804, in pydantic.fields.ModelField.populate_validators
  File "pydantic/validators.py", line 723, in find_validators
RuntimeError: no validator found for <class 're.Pattern'>, see `arbitrary_types_allowed` in Config

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [X] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

from langchain.agents.chat.output_parser import ChatOutputParser

### Expected behavior

Import corrrectly.
Retain links when extracting text using PDFLoader
Labels: name: area: doc loader, name: auto:bug

### System Info

Hi,

I'm using version 0.0.105 of langchain (nodejs). 

I'm using PDFLoader to embed PDF documents, and I've noticed that the extracted documents do not retain the hyperlinks that were present in the files.

Is there a way to retain them? I couldn't find it in the document or the PDFLoader class.

### Who can help?

@eyurtsev @hwchase17 @agola11

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

var pdfLoader = new PDFLoader(filePath, {
    splitPages: false,
});

var rawDocs = await pdfLoader.load();
console.log(rawDocs);

### Expected behavior

I was expecting PDFLoader to convert the hyperlinks in the PDF to either raw URLs or markdown URL.
Unbreak devcontainer

Codespaces and devcontainer was broken by the [repo restructure](https://github.com/langchain-ai/langchain/discussions/8043).



  - Description: Add libs/langchain to container so it can be built without error.
  - Issue: -
  - Dependencies: -
  - Tag maintainer: @hwchase17 @baskaryan 
  - Twitter handle: @finnless

The failed build log says:
```
#10 [langchain-dev-dependencies 2/2] RUN poetry install --no-interaction --no-ansi --with dev,test,docs
#10 sha256:e850ee99fc966158bfd2d85e82b7c57244f47ecbb1462e75bd83b981a56a1929
2023-07-23 23:30:33.692Z: #10 0.827 
#10 0.827 Directory libs/langchain does not exist
2023-07-23 23:30:33.738Z: #10 ERROR: executor failed running [/bin/sh -c poetry install --no-interaction --no-ansi --with dev,test,docs]: exit code: 1
```

The new pyproject.toml imports from libs/langchain:
https://github.com/langchain-ai/langchain/blob/77bf75c236351edf47d3a76a522bb45ccc90d299/pyproject.toml#L14-L16

But libs/langchain is never added to the dev.Dockerfile:

https://github.com/langchain-ai/langchain/blob/77bf75c236351edf47d3a76a522bb45ccc90d299/libs/langchain/dev.Dockerfile#L37-L39
Function calling logging fixup

Fix bad overwriting of "functions" arg in invocation params.
Cleanup precedence in the dict
Clean up some inappropriate types (mapping should be dict)


Example: https://dev.smith.langchain.com/public/9a7a6817-1679-49d8-8775-c13916975aae/r

![image](https://github.com/langchain-ai/langchain/assets/13333726/94cd0775-b6ef-40c3-9e5a-3ab65e466ab9)
Optimize the cosine_similarity_top_k function performance
Labels: name: area: embeddings, name: auto:improvement

Optimizing important numerical code and making it run faster.

Performance went up by 1.48x (148%). Runtime went down from 138715us to 56020us

Optimization explanation:

The `cosine_similarity_top_k` function is where we made the most significant optimizations. 
Instead of sorting the entire score_array which needs considering all elements, `np.argpartition` is utilized to find the top_k largest scores indices, this operation has a time complexity of O(n), higher performance than sorting. Remember, `np.argpartition` doesn't guarantee the order of the values. So we need to use argsort() to get the indices that would sort our top-k values after partitioning, which is much more efficient because it only sorts the top-K elements, not the entire array. Then to get the row and column indices of sorted top_k scores in the original score array, we use `np.unravel_index`. This operation is more efficient and cleaner than a list comprehension. 

The code has been tested for correctness by running the following snippet on both the original function and the optimized function and averaged over 5 times.
```
def test_cosine_similarity_top_k_large_matrices():
    X = np.random.rand(1000, 1000)
    Y = np.random.rand(1000, 1000)
    top_k = 100
    score_threshold = 0.5
    gc.disable()
    counter = time.perf_counter_ns()
    return_value = cosine_similarity_top_k(X, Y, top_k, score_threshold)
    duration = time.perf_counter_ns() - counter
    gc.enable()
```

@hwaking @hwchase17 @jerwelborn 

Unit tests pass, I also generated more regression tests which all passed.
MultiRetrievalQAChain async mode
Labels: name: auto:question, name: auto:enhancement

### Feature request

When I testing MultiRetrievalQAChain in synchronous mode, it worked perfectly. It can choose QA from the right retriever. But when I implement MultiRetrievalQAChain into production with streaming, I need mode async.

anyone can help me ?
TIA 

### Motivation

I'm quite frustated when I found MultiRetrievalQAChain not yet support mode async. Because the launch time for the product is halted.

### Your contribution

yes, I'd love to help testing the new features from langchain.
YoutubeLoader for nodejs 
Labels: name: area: doc loader, name: auto:enhancement

### Feature request

Would like to have YoutubeLoader on Nodejs api

https://api.python.langchain.com/en/latest/document_loaders/langchain.document_loaders.youtube.YoutubeLoader.html#langchain.document_loaders.youtube.YoutubeLoader

### Motivation

The motivation of this is to have the feature to load the transcript of youtube video and summarize it

### Your contribution

no
add faiss test for score threshold
Labels: name: area: vector store, name: auto:improvement

# What
- Add faiss vector search test for score threshold
- Fix failing faiss vector search test; filtering with list value is wrong.

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: Add faiss vector search test for score threshold; Fix failing faiss vector search test; filtering with list value is wrong.
  - Issue: None
  - Dependencies: None
  - Tag maintainer: @rlancemartin, @eyurtsev
  - Twitter handle: @MlopsJ

Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Issue: Why condense the conversational history when Open AI provides a 'messages' field in its API?
Labels: name: auto:question, name: auto:improvement

### Issue you'd like to raise.

Hi there, 

I've found many scenarios where ConversationalRetrievalChain's attempt to condense a conversation history into a stand-alone question fails, either because the history is too complex or, for example, if the user simply provides an affirmation.

Why do we attempt to condense the conversational history when Open AI's chat API provides a [messages](https://platform.openai.com/docs/api-reference/chat/create#chat/create-messages) field for this purpose?

Is there a way to construct a retrieval chain such that the messages field in Open AI's chat API can be used instead of attempting to condense the chat history into a stand-alone question?


### Suggestion:

Provide a way to construct a retrieval chain such that Open AI's chat API's messages field can be used instead of attempting to condense the question.
fix-typo
Labels: name: auto:question

more general questions about this function:

```
        return {
            **super()._get_invocation_params(stop=stop, **kwargs),
            **self._default_params,
            "model": self.model_name,
            "functions": kwargs.get("functions"),
        }
```

1. `**super()._get_invocation_params(stop=stop, **kwargs)` returns everything in kwargs, including functions (if present). do we do `"functions": kwargs.get("functions"),` because we explicitly need to pass functions through even when not present? (im assuming this is not needed (because its been broken and we havent noticed) so can we just remove
2. This ordering of 

```
**super()._get_invocation_params(stop=stop, **kwargs),
            **self._default_params,
```

means that if an arbitrary param is passed in through `kwargs` that is same as one in `**self._default_params,`, the one in defuaut params gets logged. imo we want to give preference to everything in kwargs right? why is this ordering the same?