delayedkarma/langchain-issues huggingface dataset:
Update s3_directory.py
Labels: name: area: doc loader, name: auto:improvement, name: integration: aws, name: size:M

read all the files inside the sub-directory of the given bucket or given prefix . 
since the files are getting loaded via unstructured loader

 a fix of issue  #11854

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Issue: Is langchain automatically adjusting its prompts to account for required formats of models such as llama2?
Labels: name: auto:question, name: area: models

### Issue you'd like to raise.

I am wondering if I need to adjust the prompt template of the combine_docs_chain (combine_docs_chain.llm_chain.prompt.template) to account for the needs of llama 2 models that require the following format:


 
 You are a HR-assistant that answers questions:
 
 
{query} 


### Suggestion:

_No response_
Extraction of contents from docx files through UnstructuredWordDocumentLoader
Labels: name: area: doc loader, name: auto:bug

### System Info

langchain=0.0.208
platform=Windows
python=3.9.16

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

from langchain.document_loaders import UnstructuredWordDocumentLoader

loader = UnstructuredWordDocumentLoader(docx_file_path, mode="elements")
data = loader.load()
data

### Expected behavior

Page numbers of the contents extracted using UnstructuredWordDocumentLoader(docx_file_path, mode="elements") are not in sync with the actual page number of the contents which are there in the docx file.
Request to add OpenSearchVectorSearch delete index method
Labels: name: area: vector store, name: auto:improvement

### Feature request

In OpenSearchVectorSearch langchain wrapper there is no way to delete the indexes from vector DB.


### Motivation

I was working on a project and had to delete some indexes because they already existed in DB. 
Had to write the code myself to do that.

### Your contribution

Yes I can create a PR for the issue.
Issue: rephrasing of question in Conversational retrieval chain which delays responses and users see it
Labels: name: auto:bug, name: auto:improvement, name: area: models

### Issue you'd like to raise.

When you give a query, it rephrases the query(which isn't the issue), but the rephrased query gets shown which is very problematic because it delays response and the rephrase gets shown to the user.
example, if I ask What is an apple is, i get shown something like " What is the definition of an apple
Getting the rephrased question delays the response to obtain the first token, and this rephrased question gets shown to the user first before the real answer comes out, which isn't good at all

 I want a way where it can rephrase the question, but it doesn't need to increase response time and it doesn't need to show it

### Suggestion:

_No response_
Chroma vectorstore get_ids 
Labels: name: area: vector store, name: auto:improvement, name: integration: chroma, name: size:XS

Simply added a get_ids method, that returns a list of all ids in the chroma vectorstore. 

To dynamically add, delete and update documents in a vectorstore you need to know which ids are in the vectorstore. With this function, it's just a bit easier to access them.

Thought about creating an abstract method in the Vectorstore interface. So it's available per default. But I don't know how to access the ids of the other vectorstore types.

@jarib
Initialize HuggingFacePipeline parameters when transformers pipeline passed in directly
Labels: name: auto:bug, name: area: models

When a transformers pipeline is passed directly into `HuggingFacePipeline`, it always sets `model_id` to 'gpt2' no matter what model has been loaded into the pipeline. 

To replicate the problem: 
```python
model_id = "Salesforce/codegen-350M-mono"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=10, temperature=0.3, top_p=0.95)
hf = HuggingFacePipeline(pipeline=pipe)
print(hf)
print(hf.pipeline.model.name_or_path)
```
`print(hf)` returns
```python
HuggingFacePipeline
Params: {'model_id': 'gpt2', 'model_kwargs': None, 'pipeline_kwargs': None}
```
despite the pipeline model having correctly loaded `Salesforce/codegen-350M-mono`. 

This is confusing and can lead to folks assuming that the pipeline model has been loaded incorrectly. 
Additionally, `model_id` and `kwargs` are convenient parameters to use when initialized
This issue was reported in #3451 and #8280

This PR addresses this by initializing the `HuggingFacePipeline` parameters whenever a pipeline object is passed directly. 
After this change, using the above example: 

```python
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=10, temperature=0.3, top_p=0.95)
hf = HuggingFacePipeline(pipeline=pipe)
print(hf)
```
returns
```python
HuggingFacePipeline
Params: {'model_id': 'Salesforce/codegen-350M-mono', 'model_kwargs': None, 'pipeline_kwargs': {'max_new_tokens': 10, 'temperature': 0.3, 'top_p': 0.95}}
```




<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
DOC: docs/expression_language/cookbook/tools --next leads to a previous page
Labels: name: auto:bug, name: auto:documentation

### Issue with current documentation:

Converting https://python.langchain.com/ to a pdf using https://github.com/jean-humann/docs-to-pdf runs forever because https://python.langchain.com/docs/expression_language/cookbook/tools --next leads to a previous page.
`npx docs-to-pdf --initialDocURLs="https://python.langchain.com/" --contentSelector="article" --paginationSelector="a.pagination-nav__link.pagination-nav__link--next" --coverImage="https://upload.wikimedia.org/wikipedia/commons/3/3f/LangChain_logo.png" --coverTitle="LangChain"`

### Idea or request for content:

_No response_
Free knowledge base pod information update
Labels: name: lgtm, name: area: vector store, name: auto:documentation

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->

We updated MyScale free knowledge base, where you can try your RAG with 36 million paragraphs from wikipedia and 2 million paragraphs from ArXiv.

The pod has two tables
```sql
CREATE TABLE default.ChatArXiv (
    `abstract` String, 
    `id` String, 
    `vector` Array(Float32), 
    `metadata` Object('JSON'), 
    `pubdate` DateTime,
    `title` String,
    `categories` Array(String),
    `authors` Array(String), 
    `comment` String,
    `primary_category` String,
    VECTOR INDEX vec_idx vector TYPE MSTG('metric_type=Cosine'), 
    CONSTRAINT vec_len CHECK length(vector) = 768) 
ENGINE = ReplacingMergeTree ORDER BY id;

CREATE TABLE wiki.Wikipedia (
    `id` String, 
    `title` String, 
    `text` String,
    `url` String,
    `wiki_id` UInt64,
    `views` Float32,
    `paragraph_id` UInt64,
    `langs` UInt32, 
    `emb` Array(Float32), 
    VECTOR INDEX emb_idx emb TYPE MSTG('metric_type=Cosine'), 
    CONSTRAINT emb_len CHECK length(emb) = 768) 
ENGINE = ReplacingMergeTree ORDER BY id;
```

You can connect those two tables using credentials below (just the same to the old one)
URL: `msc-4a9e710a.us-east-1.aws.staging.myscale.cloud`
Port: `443`
Username: `chatdata`
Password: `myscale_rocks`

It's FREE and you can also use it with 
ChatData: https://github.com/myscale/ChatData
Retrieval-QA-Benchmark: https://github.com/myscale/Retrieval-QA-Benchmark
... and also LangChain!

Request for review @baskaryan
Langchain ConfluenceLoader.load(): Get "when" field in metadata
Labels: name: area: doc loader, name: auto:bug

### System Info

langchain: 0.0.311


### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

We're trying to load confluence pages using `ConfluenceLoader.load()`.
If we load pages by list of page_ids:
```
loader = ConfluenceLoader(
     url=baseUrl,
     username=username, 
     api_key= api_key
)

documents = loader.load(
    page_ids = ["2081227164", "2096890841", "2081227310"], 
    include_attachments=False, 
    limit=50,
    max_pages=1000
 )
```
We are able to get `when` field in metadata.
However, if we reuqest pages by space
```
documents = loader.load(
    space_key = spaceName,
    include_attachments=False, 
    limit=50,
    max_pages=1000
 )
```
The when field will not be in metadata.





### Expected behavior

Is there any methods we can request pages by space and also get `when` field? This is really important for us.
Thanks :)
Bagatur/tiledb
Labels: name: lgtm, name: area: vector store, name: auto:enhancement

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
migrate
Labels: name: auto:documentation

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
PLEASE make all subsequent versions of Langchain backwards compatible!!!
Labels: name: auto:improvement

### Feature request

Example of what NOT to do:
https://github.com/langchain-ai/langchain/issues/12507

Langchain, PLEASE stop doing this. PLEASE make all your updates backwards compatible. As it is, if you keep making changes like moving modules around, existing scripts break with every new update of Langchain.

### Motivation

It is so frustrating to have to try and guess which version of Langchain will actually work with my code.

### Your contribution

Just stop moving modules around.  You cause way more harm than good when you do that, because tutorials for Langchain have to be rewritten and new versions of Langchain are not compatible with Langchain code written just a few months ago.
__getattr__ should rase AttributeError not ImportError on missing attributes
Labels: name: area: agent, name: auto:bug

[The python spec](https://docs.python.org/3/reference/datamodel.html#object.__getattr__) requires that `__getattr__` throw `AttributeError` for missing attributes but there are several places throwing `ImportError` in the current code base. This causes a specific problem with `hasattr` since it calls `__getattr__` then looks only for `AttributeError` exceptions. At present, calling `hasattr` on any of these modules will raise an unexpected exception that most code will not handle as `hasattr` throwing exceptions is not expected. 

In our case this is triggered by an exception tracker (Airbrake) that attempts to collect the version of all installed modules with code that looks like: `if hasattr(mod, "__version__"):`. With `HEAD` this is causing our exception tracker to fail on all exceptions.

I only changed instances of unknown attributes raising `ImportError` and left instances of known attributes raising `ImportError`. It feels a little weird but doesn't seem to break anything.
Automatically add configurable key to config_schema if config_specs i…
Labels: name: auto:improvement

…s present

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Add KDBAI vector store
Labels: name: area: vector store, name: auto:enhancement, name: size:XL

Addition of KDBAI vector store (https://kdb.ai).

Dependencies: `kdbai_client` v0.1.2 Python package.

Sample notebook: `docs/docs/integrations/vectorstores/kdbai.ipynb`

Tag maintainer: @bu2kx
Twitter handle: @kxsystems
Add support for Cohere embed V3
Labels: name: area: embeddings, name: auto:enhancement, name: auto:documentation

### Feature request

The new cohere V3 embed endpoints have a new required parameter - input_type.
Langchain currently doesn't support this parameter

https://txt.cohere.com/introducing-embed-v3/

### Motivation

Being able to use the newest multilingual embedding capabilities is something users who are developing an international product would find very important

### Your contribution

Documentation!
Add mask API key to Petals LLM
Labels: name: auto:improvement, name: area: models

- **Description:** Add masking of API Key for `Petals` LLM when printed.
   -  Added unit tests to validate API key handling and representation in the `Petals` class.
- **Issue:** https://github.com/langchain-ai/langchain/issues/12165
- **Dependencies:** None
- **Tag maintainer:** @eyurtsev

```py
# output of `__repr__()` method
# llm = Petals(model_name="bigscience/bloom-petals")
# llm

Petals(client=DistributedBloomForCausalLM(
  (transformer): DistributedBloomModel(
    (word_embeddings): Embedding(250880, 14336)
    (word_embeddings_layernorm): LayerNorm((14336,), eps=1e-05, elementwise_affine=True)
    (h): RemoteSequential(modules=bigscience/bloom-petals.0..bigscience/bloom-petals.69)
    (ln_f): LayerNorm((14336,), eps=1e-05, elementwise_affine=True)
  )
  (lm_head): LMHead()
), tokenizer=BloomTokenizerFast(name_or_path='bigscience/bloom-petals', vocab_size=250680, model_max_length=1000000000000000019884624838656, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'bos_token': '
', 'eos_token': '
', 'unk_token': '<unk>', 'pad_token': '<pad>'}, clean_up_tokenization_spaces=False),  added_tokens_decoder={
	0: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	1: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	2: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	3: AddedToken("<pad>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}, huggingface_api_key=SecretStr('**********'))

```
Update llamacpp.ipynb
Labels: name: auto:documentation

HuggingFace -> Hugging Face

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Issue: Amazon Bedrock Cohere Command LLM - Malformed input request: 2 schema violations found
Labels: name: area: embeddings, name: auto:bug, name: area: models, name: integration: pinecone, name: integration: aws

### System Info

Sample Snippet:
```
from langchain.chains.question_answering import load_qa_chain

chain = load_qa_chain(llm, chain_type = "stuff")

query = "decentralized identity"
docs = docsearch.similarity_search(query)

text = chain({"input_documents": docs, "question": query, "verbose": True})
```

Error:
```
ValueError: Error raised by bedrock service: An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: 2 schema violations found, please reformat your input and try again.
```



### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [X] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
from langchain.llms.bedrock import Bedrock
from langchain.embeddings import BedrockEmbeddings

modelId = 'cohere.command-text-v14'

bedrock_runtime = boto3.client(service_name='bedrock-runtime', region_name='us-east-1')
llm = Bedrock(model_id=modelId, client=bedrock_runtime )

docs = read_and_split_docs(data_directory)


bedrock_embeddings = BedrockEmbeddings(model_id="amazon.titan-embed-text-v1", client=bedrock_runtime)
docsearch = Pinecone.from_texts(
    [t.page_content for t in docs],
    bedrock_embeddings,
    index_name=index_name
)
from langchain.chains.question_answering import load_qa_chain

chain = load_qa_chain(llm, chain_type = "stuff")

query = "Human: decentralized identity. Assistant:"
docs = docsearch.similarity_search(query)

text = chain({"input_documents": docs, "question": query, "verbose": True})
```

### Expected behavior

return some response