delayedkarma/langchain-issues huggingface dataset:
Text Chunks are 1/2 the token length specified when using split text with CharacterTextSplitter.from_tiktoken_encoder and separator=" "

### System Info

langchain-0.0.202
Python 3.11.0
Windows 10 Pro




### Who can help?

@hwchase17

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. Use `CharacterTextSplitter.from_tiktoken_encoder` and set the separator to `" "`, and the chunk size to anything
2. Split some text using `split_text`
3. Notice that the resulting chunked text is nearly half the token length specified by the chunk size
```py
import tiktoken
from langchain.text_splitter import CharacterTextSplitter

lorem_text = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Metus dictum at tempor commodo ullamcorper a lacus vestibulum sed. Integer quis auctor elit sed vulputate. Quis blandit turpis cursus in hac. Pellentesque pulvinar pellentesque habitant morbi tristique senectus et netus et."

text_splitter = CharacterTextSplitter.from_tiktoken_encoder(
    chunk_size=100, chunk_overlap=0, separator=" ", model_name="gpt-3.5-turbo"
)

texts = text_splitter.split_text(lorem_text)

encoding = tiktoken.encoding_for_model("gpt-3.5-turbo")
print(len(encoding.encode(texts[0]))) # 44 Tokens ; expected around 100
```

### Expected behavior

I would expect the resulting text chunks after splitting to be around 100 tokens.

I think the issue arises from text_splitter.py, commit #1511 : lines 135 and 160. `total += _len + (separator_len if len(current_doc) > 1 else 0)` assumes that the separator (in this case " ") is it's own token, but I think " " is often combined with the characters next to it. From the tokenizer webpage from OpenAI : 
![image](https://github.com/hwchase17/langchain/assets/16768177/f6f4a856-c7fb-40d5-bf95-5b7761b81524)

You can see that for the " " and "d" from "dolor" are combined into one token, instead of always being separate as assumed by the above `total`.
Sliding window of intermediate actions for agents

### Feature request

Allow tweaking with the history window / intermediate actions that are being sent to the LLM:
* Send a sliding window if N last actions
* Only send a specific snapshot (can be useful for code generation tasks - for example where the agent needs to perfect the code until it works).

### Motivation

Currently, agents use the entire length of intermediate actions whenever they call the LLM.
This means that long-running agents can quickly reach the token limit.

### Your contribution

I'm willing to write a PR for this if the feature makes sense for the community
Issue:current not support gpt-3.5-turbo-0613 model

### Issue you'd like to raise.

execute method modelname_to_contextsize had an exception~! online 546 "Unknown model: gpt-3.5-turbo-0613. Please provide a valid OpenAI model name."

source code location:langchain/llms/openai.py(BaseOpenAI.modelname_to_contextsize)

### Suggestion:

_No response_
Modified PromptLayerChatOpenAI class to support function call
Labels: name: lgtm

Introduction of newest function calling feature doesn't work properly with PromptLayerChatOpenAI model since on the `_generate` method, functions argument are not even getting passed to the `ChatOpenAI` base class which results in empty `ai_message.additional_kwargs` 

Fixes  #6365


  - @hwchase17
  - @agola11
PromptLayerChatOpenAI does not support the newest function calling feature

### System Info

Python Version: 3.11
Langchain Version: 0.0.209

### Who can help?

@hwchase17 @agola11 

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Steps to reproduce:

```
llm = PromptLayerChatOpenAI(model="gpt-3.5-turbo-0613", pl_tags=tags, return_pl_id=True)
predicted_message = self.llm.predict_messages(messages, functions=self.functions, callbacks=callbacks)
```

`predicted_message.additional_kwargs` attribute appears to have a empty dict, because the `functions` kwarg not even passed to the parent class.

### Expected behavior

Predicted AI Message should have a `function_call`  key on `additional_kwargs` attribute.
The chatCompletion operation does not work with the specified model appears when I use AzureChatOpenAI, but it does exist

### Issue you'd like to raise.

openai.error.InvalidRequestError: The chatCompletion operation does not work with the specified model, text-embedding-ada-002. Please choose different model and try again. You can learn more about which models can be used with each operation here: https://go.microsoft.com/fwlink/?linkid=2197993.


### Suggestion:

_No response_
Support for Redis Cluster

### Feature request

The current python client `langchain.vectorstores.redis` lacks support for RedisCluster. Handling redirection with try-expect on receiving `MOVED` error also doesn't work in this case because the code `Redis.from_documents(docs, llmembeddings, redis_url=redis_url, index_name=index_name)` internally makes more calls to Redis, which eventually throw MOVED error because the client is not configured for RedisCluster.

### Motivation

As users of Redis Cluster at Salesforce, we aim to integrate it seamlessly and develop a chatbot powered by LLM. Enabling Redis API support in the Vector Redis Client would enhance performance, streamline development workflows, and provide a unified client library for interacting with Redis databases. We believe this addition would greatly benefit our organization and the wider community utilizing Redis in their applications.

### Your contribution

We will look at adding RedisCluster support in langchain vector client.
Improved Documentation: Eliminating Redundancy in the Introduction.mdx

When the documentation was originally written there was a redundant typing of the word "using the"

<!--
Thank you for contributing to LangChain! Your PR will appear in our release under the title you set. Please make sure it highlights your valuable contribution.

Replace this with a description of the change, the issue it fixes (if applicable), and relevant context. List any dependencies required for this change.

After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost.

Finally, we'd love to show appreciation for your contribution - if you'd like us to shout you out on Twitter, please also include your handle!
-->

<!-- Remove if not applicable -->

Fixes # (issue)

#### Before submitting

<!-- If you're adding a new integration, please include:

1. a test for the integration - favor unit tests that does not rely on network access.
2. an example notebook showing its use


See contribution guidelines for more information on how to write tests, lint
etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->

#### Who can review?

Tag maintainers/contributors who might be interested:

<!-- For a quicker response, figure out the right person to tag with @

  @hwchase17 - project lead

  Tracing / Callbacks
  - @agola11

  Async
  - @agola11

  DataLoaders
  - @eyurtsev

  Models
  - @hwchase17
  - @agola11

  Agents / Tools / Toolkits
  - @hwchase17

  VectorStores / Retrievers / Memory
  - @dev2049

 -->
The cost calculation of tokens for open ai models looks like is only considering output tokens

### System Info

I am using gpt-3.5-turbo for which the price of the tokens are as below

4K Context
0.0015/1K tokens - for input
0.002/1k tokens - output

From the call back I get the below

Cost and token usage :Tokens Used: 222
	Prompt Tokens: 171
	Completion Tokens: 51
Successful Requests: 1
Total Cost (USD): $0.00044400000000000006

Going by the pricing, it should be 0.000375942

It looks like the program is only looking at output cost




### Who can help?

@agola11 

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [X] Callbacks/Tracing
- [ ] Async

### Reproduction

run the LLM with callback using any of the open ai models

        with get_openai_callback() as cb:
            response = chat_chain.predict(input=_input.to_string())

            print(response)
            resp_json = out_parser.parse(response)

            print("Cost and token usage :{cb}".format(cb=cb))

            return resp_json

### Expected behavior

The cost calculation should consider the input tokens also
Relax the action input check for actions that require no input

<!--
Thank you for contributing to LangChain! Your PR will appear in our release under the title you set. Please make sure it highlights your valuable contribution.

Replace this with a description of the change, the issue it fixes (if applicable), and relevant context. List any dependencies required for this change.

After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost.

Finally, we'd love to show appreciation for your contribution - if you'd like us to shout you out on Twitter, please also include your handle!
-->

<!-- Remove if not applicable -->



#### Before submitting

<!-- If you're adding a new integration, please include:

1. a test for the integration - favor unit tests that does not rely on network access.
2. an example notebook showing its use


See contribution guidelines for more information on how to write tests, lint
etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->
When the tool requires no input, the LLM often gives something like this:
```json
{
    "action": "just_do_it"
}
```
I have attempted to enhance the prompt, but it doesn't appear to be functioning effectively. Therefore, I believe we should consider easing the check a little bit.

#### Who can review?

Tag maintainers/contributors who might be interested: @hwchase17
DOC: Updating documentation relating to Models->llms->integrations->*

### Issue with current documentation:

The page I am referring with this issue is [Llama.cpp](https://python.langchain.com/docs/modules/model_io/models/llms/integrations/llamacpp.html). I am hoping to update the documentation regarding some windows specific instructions which others might find useful.

I've forked the repo and my working changes I would push there and make a pull request here.

I understand that to do this; `langchain/docs/_dist/docs_skeleton/docs/modules/model_io/models/llms/integrations/llamacpp.ipynb` needs to be updated.

It also appears that `docs/_dist` is now ignored by `.gitignore`.

Could you please let me know if I am looking in the wrong place? If so please correct me.

### Idea or request for content:

_No response_
Be able to use Codey models on Vertex AI
Labels: name: enhancement, name: lgtm

Added the functionality to leverage 3 new Codey models from Vertex AI:
- code-bison - Code generation using the existing LLM integration
- code-gecko - Code completion using the existing LLM integration
- codechat-bison - Code chat using the existing chat_model integration

Tag maintainers/contributors who might be interested:

  - @hwchase17
  - @agola11
  - @vowelparrot 
  
  Had to close and create a new PR to resolve conflicts.
Docusaurus Document Loader

### Feature request

I propose the integration of a **Docusaurus Document Loader** for the LangChain Python repository. By integrating a **[Docusaurus](https://docusaurus.io/) Document Loader**, we can extend the documentation capabilities of LangChain and provide a more comprehensive resource for developers who use Docusaurus (similar to ReadTheDocs).

### Motivation

My motivation for this feature request is to enhance and extend the document loading functionalities of LangChain. Currently, LangChain [integrates with ReadTheDocs](https://python.langchain.com/docs/modules/data_connection/document_loaders/integrations/readthedocs_documentation), and while this is a powerful tool, incorporating a **Docusaurus Document Loader** can offer an alternative loader. Plus, LangChain docs ([Python](https://python.langchain.com/docs) and [JavaScript)](https://js.langchain.com/docs) are already hosted on Docusaurus.

### Your contribution

As the one initiating this feature request, I am willing to help by creating this issue and providing initial suggestions for the implementation of the **Docusaurus Document Loader**.
Feature/generate custom context
Labels: name: area: vector store, name: auto:improvement

A vectorstore can be used to provide an LLM with extra context by including it in the prompt.
by default context is created by joining all the documents returned from the store seperated by a seperator e.g

    <page_content>
    <document_separator>
    <page_content>
    <document_separator>
    <page_content>

However often the logic we want to apply is much more bespoke.
For exmaple imagine we are storing information about ppl in a vector store.

The first entry would look like:

    name,date of birth,job
    Paul,23-12-1985   ,cleaner

another entry would look like:

    name,date of birth,job
    John,23-12-1985   ,driver


Obviously both of these embeddings contain the same headers.

By default the context would be created as such

    name,date of birth,job
    Paul,23-12-1985   ,cleaner
    <document_separator>
    name,date of birth,job
    John,23-12-1985   ,driver


However we want to reduce the header duplication and also just create it as one table for the LLM to understand

    name,date of birth,job
    Paul,23-12-1985   ,cleaner
    John,23-12-1985   ,driver



I have created a test where I show that we could use the metadata of a document to enrich the context. 


#### Who can review?
@dev2049
Fix duplicated sentence in documentation's introduction 

Fix duplicated sentence in documentation's introduction 
<!--
Thank you for contributing to LangChain! Your PR will appear in our release under the title you set. Please make sure it highlights your valuable contribution.

Replace this with a description of the change, the issue it fixes (if applicable), and relevant context. List any dependencies required for this change.

After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost.

Finally, we'd love to show appreciation for your contribution - if you'd like us to shout you out on Twitter, please also include your handle!
-->

<!-- Remove if not applicable -->

Fixes # (issue)

#### Before submitting

<!-- If you're adding a new integration, please include:

1. a test for the integration - favor unit tests that does not rely on network access.
2. an example notebook showing its use


See contribution guidelines for more information on how to write tests, lint
etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->

#### Who can review?

Tag maintainers/contributors who might be interested:

<!-- For a quicker response, figure out the right person to tag with @

  @hwchase17 - project lead

  Tracing / Callbacks
  - @agola11

  Async
  - @agola11

  DataLoaders
  - @eyurtsev

  Models
  - @hwchase17
  - @agola11

  Agents / Tools / Toolkits
  - @hwchase17

  VectorStores / Retrievers / Memory
  - @dev2049

 -->
nit

<!--
Thank you for contributing to LangChain! Your PR will appear in our release under the title you set. Please make sure it highlights your valuable contribution.

Replace this with a description of the change, the issue it fixes (if applicable), and relevant context. List any dependencies required for this change.

After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost.

Finally, we'd love to show appreciation for your contribution - if you'd like us to shout you out on Twitter, please also include your handle!
-->

<!-- Remove if not applicable -->

Fixes # (issue)

#### Before submitting

<!-- If you're adding a new integration, please include:

1. a test for the integration - favor unit tests that does not rely on network access.
2. an example notebook showing its use


See contribution guidelines for more information on how to write tests, lint
etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->

#### Who can review?

Tag maintainers/contributors who might be interested:

<!-- For a quicker response, figure out the right person to tag with @

  @hwchase17 - project lead

  Tracing / Callbacks
  - @agola11

  Async
  - @agola11

  DataLoaders
  - @eyurtsev

  Models
  - @hwchase17
  - @agola11

  Agents / Tools / Toolkits
  - @hwchase17

  VectorStores / Retrievers / Memory
  - @dev2049

 -->
Issue: langchain pandas df agent, not taking full df in context

### Issue you'd like to raise.

HI all,
As can be seen in this screenshot
![SCR-20230617-uajg](https://github.com/hwchase17/langchain/assets/62832721/8bca1a16-4c8d-47a3-8e40-22bd88b3bac8)

I am using pandas_df_agent, but instead of taking whole dataframe, which is around 27000 lines, it is creating itself a Sample data, and doing operations on that.
Isn't it weird, ? Kindly help me with this..

Thanks

### Suggestion:

_No response_
getting the "This model's maximum context length is 4097 tokens" error using gpt-4 and gpt-3.5-turbo-16k model

### System Info

This is strange, since these models are with 8k and 16k context length
my code is
```
llm = ChatOpenAI(model_name="gpt-4", temperature=0)
        agent_executor = create_custom_agent(
            llm= llm,
            tools=toolkit.get_tools()[0:1],
            verbose=True,
            # prefix=PREFIX,
        )
```

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
llm = ChatOpenAI(model_name="gpt-4", temperature=0)
        agent_executor = create_custom_agent(
            llm= llm,
            tools=toolkit.get_tools()[0:1],
            verbose=True,
            # prefix=PREFIX,
        )
```

### Expected behavior

the agent runs without error or not producing "4097 context length" error
Supporting Similarity Search with Threshold on PGVector retriever

### Feature request

Implementing `_similarity_search_with_relevance_scores` on PGVector so users can set search_type to "similarity_score_threshold" without raising **NotImplementedError**.

`
    retriever = pgvector.as_retriever(search_type="similarity_score_threshold", search_kwargs={"score_threshold": 0.7})
    results = retriever.get_relevant_documents(query)
`

### Motivation

Using the search threshold on PGVector to avoid unrelated documents in the results.

### Your contribution

Pull request will be submitted.
Make lckwargs private

<!--
Thank you for contributing to LangChain! Your PR will appear in our release under the title you set. Please make sure it highlights your valuable contribution.

Replace this with a description of the change, the issue it fixes (if applicable), and relevant context. List any dependencies required for this change.

After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost.

Finally, we'd love to show appreciation for your contribution - if you'd like us to shout you out on Twitter, please also include your handle!
-->

<!-- Remove if not applicable -->

Fixes # (issue)

#### Before submitting

<!-- If you're adding a new integration, please include:

1. a test for the integration - favor unit tests that does not rely on network access.
2. an example notebook showing its use


See contribution guidelines for more information on how to write tests, lint
etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->

#### Who can review?

Tag maintainers/contributors who might be interested:

<!-- For a quicker response, figure out the right person to tag with @

  @hwchase17 - project lead

  Tracing / Callbacks
  - @agola11

  Async
  - @agola11

  DataLoaders
  - @eyurtsev

  Models
  - @hwchase17
  - @agola11

  Agents / Tools / Toolkits
  - @hwchase17

  VectorStores / Retrievers / Memory
  - @dev2049

 -->
Tools with only default arguments still require a parameter to `run()`

### Issue you'd like to raise.

This issue is created at langchain v0.0.202

Consider the [ListDirectoryTool](https://github.com/hwchase17/langchain/blob/0475d015fe0eb4a997c7d37867e316a23dde8aaa/langchain/tools/file_management/list_dir.py#L21)

Its only parameter has a default value as the `.` current dir. 

Its json-schema suggests no parameter is required 

```
>>> pprint.pprint(langchain.tools.format_tool_to_openai_function(t))
{'description': 'List files and directories in a specified folder',
 'name': 'list_directory',
 'parameters': {'description': 'Input for ListDirectoryTool.',
                'properties': {'dir_path': {'default': '.',
                                            'description': 'Subdirectory to '
                                                           'list.',
                                            'title': 'Dir Path',
                                            'type': 'string'}},
                'title': 'DirectoryListingInput',
                'type': 'object'}}
```
                
 However calling the `run` function without any parameters creates a runtime error 
 
```python
import langchain
t  = langchain.tools.ListDirectoryTool(root_dir="~/misc")
t.run()
```

With this error 
```
>>> t.run()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: BaseTool.run() missing 1 required positional argument: 'tool_input'
```

This is not a huge problem but at least it creates an inconsistency expectations by the schema and the `run` function call. 


### Suggestion:

If the schema had no required arguments, the run function should not create a runtime error if called without any arguments. At least that would be more expected.