delayedkarma/langchain-issues huggingface dataset:
Add start index to metadata in TextSplitter

<!--
Thank you for contributing to LangChain! Your PR will appear in our release under the title you set. Please make sure it highlights your valuable contribution.

Replace this with a description of the change, the issue it fixes (if applicable), and relevant context. List any dependencies required for this change.

After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost.

Finally, we'd love to show appreciation for your contribution - if you'd like us to shout you out on Twitter, please also include your handle!
-->

#### Add start index to metadata in TextSplitter

- Modified method `create_documents` to track start position of each chunk
- The `start_index` is included in the metadata if the `add_start_index` parameter in the class constructor is set to `True`

This enables referencing back to the original document, particularly useful when a specific chunk is retrieved.

<!-- If you're adding a new integration, please include:

1. a test for the integration - favor unit tests that does not rely on network access.
2. an example notebook showing its use


See contribution guidelines for more information on how to write tests, lint
etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->

#### Who can review?

Tag maintainers/contributors who might be interested:
@eyurtsev @agola11
<!-- For a quicker response, figure out the right person to tag with @

  @hwchase17 - project lead

  Tracing / Callbacks
  - @agola11

  Async
  - @agola11

  DataLoaders
  - @eyurtsev

  Models
  - @hwchase17
  - @agola11

  Agents / Tools / Toolkits
  - @vowelparrot

  VectorStores / Retrievers / Memory
  - @dev2049

 -->
chroma.from_documents AuthenticationError

### System Info

In Google Collab

What I have installed
%pip install requests==2.27.1
%pip install chromadb==<compatible version>
%pip install langchain duckdb unstructured chromadb openai tiktoken

MacBook M1

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. from langchain.embeddings import OpenAIEmbeddings
embeddings = OpenAIEmbeddings() << PASS

create a vector database and use it to index the embeddings
2. class Document:
    def __init__(self, text, metadata):
        self.page_content = text
        self.metadata = metadata << PASS

3.documents = [Document(text, metadata) for text, metadata in zip(texts, metadata_list)] << PASS

4.from langchain.vectorstores import Chroma

db = Chroma.from_documents(documents, embeddings, model='davinci') << AuthenticationError: <empty message>



---------------------------------------------------------------------------
AuthenticationError                       Traceback (most recent call last)
[<ipython-input-87-ea7b035908f9>](https://rzac4prlyba-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230606-060135-RC00_538140384#) in <cell line: 3>()
      1 from langchain.vectorstores import Chroma
      2 
----> 3 db = Chroma.from_documents(documents, embeddings, model='davinci')

17 frames
[/usr/local/lib/python3.10/dist-packages/openai/api_requestor.py](https://rzac4prlyba-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230606-060135-RC00_538140384#) in _interpret_response_line(self, rbody, rcode, rheaders, stream)
    761         stream_error = stream and "error" in resp.data
    762         if stream_error or not 200 <= rcode < 300:
--> 763             raise self.handle_error_response(
    764                 rbody, rcode, resp.data, rheaders, stream_error=stream_error
    765             )

AuthenticationError: <empty message>


What should I do?

FYI. I try to follow this link https://medium.com/mlearning-ai/using-chatgpt-for-question-answering-on-your-own-data-afa33d82fbd0
seems like he has no issue TT

### Expected behavior

pass
Add desired expected language response as a function param.

### Feature request

I could work around the call from `load_summarize_chain` to add response in certain languages

https://github.com/hwchase17/langchain/blob/master/langchain/chains/summarize/map_reduce_prompt.py

The work around was something like this:

```python
from langchain.prompts import PromptTemplate

prompt_template = """Write a concise summary of the following text.  
The following text is in 'pt-br' and you should respond in 'pt-br':


"{text}"


CONCISE SUMMARY:"""
PROMPT = PromptTemplate(template=prompt_template, input_variables=["text"])
```

Is it possible to add a language param in some functions to add this approach, if not, the answers would be always in English as its the use language in requests.

### Motivation

Help people with responses in the desired language

### Your contribution

I can help of course, even with a PR.  Just need to get more familiar with the tool
PubmedQueryRun not working

### System Info

langchain               0.0.194
langchainplus-sdk       0.0.6

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [x] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. code from langchain.tools import PubmedQueryRun
2. Error:

ImportError: cannot import name 'PubmedQueryRun' from 'langchain.tools' (C:\Users\USER\anaconda3\lib\site-packages\langchain\tools\__init__.py)
Traceback:
File "C:\Users\USER\anaconda3\lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 552, in _run_script
    exec(code, module.__dict__)
File "C:\Users\USER\Dropbox\AI\Cloned Repis\Langchain-Crash-Course\pubmed.py", line 31, in <module>
    from langchain.tools import PubmedQueryRun

### Expected behavior

It seems PubmedQueryRun is not defined.
Issue: langchainplus-sdk dependency

### Issue you'd like to raise.

Currently, the newest version `0.0.194` is not building on conda-forge since `langchainplus-sdk` is missing.
I can't add that to conda-forge, though, since I have neither found source code nor a licence file for that package - other than the PyPI release.

Is `langchainplus-sdk` a strongly needed dependency or can it be made optional like many others as well?
This prevents the newest version from being installable through conda/mama/micromamba etc.

### Suggestion:

Remove the dependency on `langchainplus-sdk` with `optional=True`.
Weaviate: Batch embed texts
Labels: name: lgtm

When a custom Embeddings object is set, embed all given texts in a batch instead of passing them through individually. Any code calling add_texts can then appropriately size the chunks of texts that are passed through to take full advantage of the hardware it's running on.

<!-- If you're adding a new integration, please include:

1. a test for the integration - favor unit tests that does not rely on network access.
2. an example notebook showing its use


See contribution guidelines for more information on how to write tests, lint
etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->

#### Who can review?

Tag maintainers/contributors who might be interested:

<!-- For a quicker response, figure out the right person to tag with @

  @hwchase17 - project lead

  Tracing / Callbacks
  - @agola11

  Async
  - @agola11

  DataLoaders
  - @eyurtsev

  Models
  - @hwchase17
  - @agola11

  Agents / Tools / Toolkits
  - @vowelparrot

  VectorStores / Retrievers / Memory
  - @dev2049

 -->
@dev2049
Fix IndexError in RecursiveCharacterTextSplitter

Fixes (not reported) an error that may occur in some cases in the RecursiveCharacterTextSplitter.

An empty `new_separators` array ([]) would end up in the else path of the condition below and used in a function where it is expected to be non empty.

```python
if new_separators is None:
    ...
else:
   # _split_text() expects this array to be non-empty!
   other_info = self._split_text(s, new_separators)

```
resulting in an `IndexError`

```python
def _split_text(self, text: str, separators: List[str]) -> List[str]:
        """Split incoming text and return chunks."""
        final_chunks = []
        # Get appropriate separator to use
>       separator = separators[-1]
E       IndexError: list index out of range

langchain/text_splitter.py:425: IndexError
```

#### Who can review?
@hwchase17 @eyurtsev
Arxiv Tool validation error (api_wrapper field required (type=value_error.missing))

### System Info

python in Azure Function, langchain 0.0.194

this code:

```
tools = [
                PubmedQueryRun(),
                ArxivQueryRun(),
            ]
```

and then loading it into an agent:

```
agent_chain = initialize_agent(tools, llm, agent=AgentType.CHAT_CONVERSATIONAL_REACT_DESCRIPTION, verbose=True, memory=memory, return_intermediate_steps=False,
                                        max_iterations=15, max_execution_time=100, early_stopping_method="generate")
```

is giving this error:

```
 1 validation error for ArxivQueryRun
[2023-06-08T18:28:08.426Z] api_wrapper
[2023-06-08T18:28:08.426Z]   field required (type=value_error.missing)

[2023-06-08T18:28:08.609Z] Executed 'Functions.HttpTriggerCallNavigator' (Failed, Id=70088244-eda1-456d-b9bc-1acedfe71af2, Duration=3756ms)
[2023-06-08T18:28:08.609Z] System.Private.CoreLib: Exception while executing function: Functions.HttpTriggerCallNavigator. System.Private.CoreLib: Result: Failure
[2023-06-08T18:28:08.609Z] Exception: TypeError: unable to encode outgoing TypedData: unsupported type "<class 'azure.functions.http.HttpResponseConverter'>" for Python type "NoneType"
[2023-06-08T18:28:08.609Z] Stack:   File "/usr/lib/azure-functions-core-tools-4/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py", line 506, in _handle__invocation_request
[2023-06-08T18:28:08.610Z]     return_value = bindings.to_outgoing_proto(
[2023-06-08T18:28:08.610Z]   File "/usr/lib/azure-functions-core-tools-4/workers/python/3.9/LINUX/X64/azure_functions_worker/bindings/meta.py", line 152, in to_outgoing_proto
[2023-06-08T18:28:08.610Z]     datum = get_datum(binding, obj, pytype)
[2023-06-08T18:28:08.610Z]   File "/usr/lib/azure-functions-core-tools-4/workers/python/3.9/LINUX/X64/azure_functions_worker/bindings/meta.py", line 110, in get_datum
[2023-06-08T18:28:08.610Z]     raise TypeError(
[2023-06-08T18:28:08.610Z] .
```

### Who can help?

@hwchase17 @vowelparrot 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
tools = [
                PubmedQueryRun(),
                ArxivQueryRun(),
            ]

agent_chain = initialize_agent(tools, llm, agent=AgentType.CHAT_CONVERSATIONAL_REACT_DESCRIPTION, verbose=True, memory=memory, return_intermediate_steps=False,
                                        max_iterations=15, max_execution_time=100, early_stopping_method="generate")
```

### Expected behavior

 1 validation error for ArxivQueryRun
[2023-06-08T18:28:08.426Z] api_wrapper
[2023-06-08T18:28:08.426Z]   field required (type=value_error.missing)
SQLDatabaseChain and agent no longer working at all. Code was working for past 4 weeks. no changes made to my code and everything errored out as of this morning (6/8/24 ~11am ET). It worked flawlessly just yesterday (6/7) for a demo i gave. 

when i try and run this code: 

from langchain import OpenAI, SQLDatabase, SQLDatabaseChain
db = SQLDatabase.from_uri("sqlite:///my_data.db")
llm = OpenAI(temperature=0, verbose=True)


i get this error: typing.ClassVar[typing.Collection[str]] is not valid as type argument

I have tested it on a snowflake connection too with the exact same error. 

I checked the langchain documentation and nothing has been updated or changed from what i can tell.
Issue: <RetrievalQA.from_chain_type with custom prompt template>

### Issue you'd like to raise.

Hello everyone! I'm having trouble setting up the successful usage of a custom QA prompt template that includes input variables with my RetrievalQA.from_chain_type.

This issue  is similar to https://github.com/hwchase17/langchain/pull/3425


I am using LangChain v0.0.192 with FAISS vectorstore.  

As below my custom prompt has three input 

**My_loader_
made_corrections_
output_format_instructions_**



My code looks like that for now:


pdf_template_stuff = """
You are a Contract Review Specialist. You have been given a dataloader as 
langchain.document_loaders.directory.DirectoryLoader: {my_loader_}.
Use the following pieces of context to answer the question at the end.
If you don't know the answer, just say that you don't know, don't try to make up an answer.
Use as much detail as possible when responding and try to make answer in markdown format as much as possible.
Based on this data do the following:
1-    Read the input PDFs and extract the relevant information.
2-    Answer the question using the extracted information.
3-    Cross-reference the answer with other PDF files to ensure accuracy.
4-    Based on the historic corrections: {made_corrections_} make corrections to the extracted answers
5-    provide the answer in alignment with the {output_format_instructions_} format.

Context: {context}
Question:{question}

Helpful Answer:
"""



pdf_prompt_template = PromptTemplate( input_variables=
["context","question","my_loader_", "made_corrections_","output_format_instructions_"],
    template=pdf_template_stuff
)




my_chain_type='stuff'

my_qa_chain = RetrievalQA.from_chain_type(llm=my_specific_llm, 
                                  chain_type=my_chain_type, 
                                  retriever=my_retriever, 
                                  return_source_documents=False,
                                  chain_type_kwargs={"prompt": pdf_prompt_template}
                                                                                  
                                         )

final_result=my_qa_chain(query="MY SAMPLE QUESTION",my_loader_=my_loader, made_corrections_=made_corrections , Output_format_instructions_=output_format_instructions)



ValueError: Missing some input keys: {'output_format_instructions_', 'my_loader_', 'made_corrections_'}


Thanks for your help 


### Suggestion:

_No response_
Add support for tags

<!--
Thank you for contributing to LangChain! Your PR will appear in our release under the title you set. Please make sure it highlights your valuable contribution.

Replace this with a description of the change, the issue it fixes (if applicable), and relevant context. List any dependencies required for this change.

After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost.

Finally, we'd love to show appreciation for your contribution - if you'd like us to shout you out on Twitter, please also include your handle!
-->

<!-- Remove if not applicable -->

Fixes # (issue)

#### Before submitting

<!-- If you're adding a new integration, please include:

1. a test for the integration - favor unit tests that does not rely on network access.
2. an example notebook showing its use


See contribution guidelines for more information on how to write tests, lint
etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->

#### Who can review?

Tag maintainers/contributors who might be interested:

<!-- For a quicker response, figure out the right person to tag with @

  @hwchase17 - project lead

  Tracing / Callbacks
  - @agola11

  Async
  - @agola11

  DataLoaders
  - @eyurtsev

  Models
  - @hwchase17
  - @agola11

  Agents / Tools / Toolkits
  - @vowelparrot

  VectorStores / Retrievers / Memory
  - @dev2049

 -->
Issue: TokenTextSplitter processes passages less than chunk_size and creates duplicate overlaps

### Issue you'd like to raise.

**Observation:** TokenTextSplitter processes passages smaller than specified chunk_size, and creates a lot more smaller chunks with duplicated overlaps.

**Impact:** The smaller chunks with duplicated overlaps may be harmful to storage size, computation time, and it may confuse the retrieval model.

**For example:** The second TokenTextSplitter creates 366 more small chunks, even if new_chunks1 doesn't have any passages longer than chunk_size=500.
<img width="950" alt="Screen Shot 2023-06-08 at 10 58 52 AM" src="https://github.com/hwchase17/langchain/assets/103061109/d05c28a9-b6dc-4e44-ad00-c892457055d8">


### Suggestion:

In source code for langchain.text_splitter, split_text_on_tokens function, can we add the check for

`If len(input_ids) <= tokenizer.tokens_per_chunk:  splits.append(tokenizer.decode(input_ids)) `
DeepLakeVectorStore new updates

Added new functionalities to DeepLake vectorstore based on our latest release. Now you can:

1. run searching in 3 different modes: `python`, `tensor_db`, `compute_engine`
2. Data ingestion on small data is now faster
3. Code refactoring

<!-- Remove if not applicable -->

Fixes # (issue)
1. Fixed typos in docstrings

#### Before submitting

<!-- If you're adding a new integration, please include:

1. a test for the integration - favor unit tests that does not rely on network access.
2. an example notebook showing its use


See contribution guidelines for more information on how to write tests, lint
etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->

#### Who can review?

Tag maintainers/contributors who might be interested:

<!-- For a quicker response, figure out the right person to tag with @

  @hwchase17 - project lead

  Tracing / Callbacks
  - @agola11

  Async
  - @agola11

  DataLoaders
  - @eyurtsev

  Models
  - @hwchase17
  - @agola11

  Agents / Tools / Toolkits
  - @vowelparrot

  VectorStores / Retrievers / Memory
  - @dev2049

 -->
When inialztion VertexAI() all passed parameters got ignored

### System Info

langchain=0.0.194
python=3.11.3

### Who can help?

@hwchase17 
@agola11 

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Run:
`VertexAI(project="my_project_name")`

### Expected behavior

The client will connect to the supplied project_id
Add a new vectorstore - AwaDB

Added AwaDB vector store, which is a wrapper over the AwaDB, that can be used as a vector storage and has an efficient similarity search.
Added integration tests for the vector store
Added jupyter notebook with the example

Complete the notebook about AwaDB which had not notebook in the last pr(https://github.com/hwchase17/langchain/pull/5687)

Please check, Thanks!

@dev2049 
@hwchase17
Issue: The 'chain' field of ConstitutionalChain is limited to 'LLM'

### Issue you'd like to raise.

Failed to pass in an `AgentExecutor` as `chain` field in a `ConsitutionalChain` like this:

```python
constitutional_chain = ConsitutionalChain.from_llm(chain = intialize_agent(<some inputs>), <other inputs>)
```

Digging into the source code, I noticed that `AgentExecutor` is a `Chain` type, but the parameter `chain` requires `LLMChain`, which is also a child of `Chain`. 

Any suggestions for this situation?

### Suggestion:

_No response_
MediaWiki docloader improvements +  unit tests
Labels: name: lgtm

Starting over from #5654 because I utterly borked the poetry.lock file.

Adds new paramerters for to the MWDumpLoader class:

* skip_redirecst (bool) Tells the loader to skip articles that redirect to other articles. False by default.
* stop_on_error (bool) Tells the parser to skip any page that causes a parse error. True by default.
* namespaces (List[int]) Tells the parser which namespaces to parse. Contains namespaces from -2 to 15 by default.

Default values are chosen to preserve backwards compatibility.

Sample dump XML and full unit test coverage (with extended tests that pass!) also included!

Tagging people for checks / review:
@hwchase17
fix: TypeError when loading confluence pages by cql
Labels: name: lgtm

The Confluence loader uses the wrong API (`Confluence.cql()` provided by `atlassian-python-api`) to load pages by CQL.
`Confluence.cql()` is a wrapper of the `/rest/api/search` API which searches for entities in Confluence.

To search for pages in Confluence, the loader can use the `/rest/api/content/search` API.

#### Who can review?

Tag maintainers/contributors who might be interested: @eyurtsev
<!-- For a quicker response, figure out the right person to tag with @

  @hwchase17 - project lead

  Tracing / Callbacks
  - @agola11

  Async
  - @agola11

  DataLoaders
  - @eyurtsev

  Models
  - @hwchase17
  - @agola11

  Agents / Tools / Toolkits
  - @vowelparrot

  VectorStores / Retrievers / Memory
  - @dev2049

 -->
#### References
##### Cloud API
https://developer.atlassian.com/cloud/confluence/rest/v1/api-group-content/#api-wiki-rest-api-content-search-get
https://developer.atlassian.com/cloud/confluence/rest/v1/api-group-search/#api-wiki-rest-api-search-get

##### Server API
https://docs.atlassian.com/ConfluenceServer/rest/8.3.1/#api/content-search
https://docs.atlassian.com/ConfluenceServer/rest/8.3.1/#api/search