delayedkarma/langchain-issues huggingface dataset:
Inconsistent behavior of `CharacterTextSplitter` when changing `keep_separator` for special regex characters
Labels: name: auto:bug, name: auto:documentation

### System Info

LangChain v0.0.225, Windows, Python 3.10

### Who can help?

@hwchase17

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction


The behavior for `CharacterTextSplitter` when changing `keep_separator` when using normal characters is like this:

```python
text_splitter = CharacterTextSplitter(
    chunk_size=4,
    chunk_overlap=0,
    separator="_",
    keep_separator=False,
)

text_splitter.split_text("foo_bar_baz_123")
# ['foo', 'bar', 'baz', '123']
```

```python
text_splitter = CharacterTextSplitter(
    chunk_size=4,
    chunk_overlap=0,
    separator="_",
    keep_separator=True,
)

text_splitter.split_text("foo_bar_baz_123")
# ['foo', '_bar', '_baz', '_123']
```

However, when using special regex characters like `.` or `*` the splitter breaks when `keep_separator` is `False`.

```python
text_splitter = CharacterTextSplitter(
    chunk_size=4,
    chunk_overlap=0,
    separator=r"\.",
    keep_separator=False,
)

text_splitter.split_text("foo.bar.baz.123")
# ['foo.bar.baz.123']
```

The special characters should be escaped, otherwise it raises an error. For example, the following code raises an error.

```python
text_splitter = CharacterTextSplitter(
    chunk_size=4,
    chunk_overlap=0,
    separator=r"?",
    keep_separator=True,
)

text_splitter.split_text("foo?bar?baz?123")
```

I'll make a PR to fix this.

Also, the documentation never mentions that the separator should be a regex, I only found out the hard way after getting regex errors on one of the `RecursiveTextSplitter` splitters after updating LangChain. I think we should add a note about this in the documentation or the code.


### Expected behavior


 ```python
text_splitter = CharacterTextSplitter(
    chunk_size=4,
    chunk_overlap=0,
    separator=r"\.",
    keep_separator=False,
)

text_splitter.split_text("foo.bar.baz.123")
# ['foo', 'bar', 'baz', '123']
```
Chroma DB Error
Labels: name: auto:bug

### System Info
Error: Please install chromadb as a dependency with, e.g. `npm install -S chromadb`
    at Function.imports (file:///home/aqib/backend-11/vision/node_modules/langchain/dist/vectorstores/chroma.js:160:19)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Chroma.ensureCollection (file:///home/aqib/backend-11/vision/node_modules/langchain/dist/vectorstores/chroma.js:61:42)
    at async Chroma.addVectors (file:///home/aqib/backend-11/vision/node_modules/langchain/dist/vectorstores/chroma.js:88:28)
    at async Chroma.addDocuments (file:///home/aqib/backend-11/vision/node_modules/langchain/dist/vectorstores/chroma.js:56:9)
    at async Function.fromDocuments (file:///home/aqib/backend-11/vision/node_modules/langchain/dist/vectorstores/chroma.js:145:9)
    at async add (file:///home/aqib/backend-11/vision/vectorStore/db.js:24:5)
    at async run (file:///home/aqib/backend-11/vision/vectorStore/insertDocs.js:6:3)
node:internal/process/promises:279
            triggerUncaughtException(err, true /* fromPromise */);
            ^

**_Got this when I console Logged the error_**
ReferenceError: fetch is not defined
    at Object.<anonymous> (/home/aqib/backend-11/vision/node_modules/chromadb/dist/main/generated/runtime.js:17:24)
    at Module._compile (node:internal/modules/cjs/loader:1196:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1250:10)
    at Module.load (node:internal/modules/cjs/loader:1074:32)
    at Function.Module._load (node:internal/modules/cjs/loader:909:12)
    at Module.require (node:internal/modules/cjs/loader:1098:19)
    at require (node:internal/modules/cjs/helpers:108:18)
    at Object.<anonymous> (/home/aqib/backend-11/vision/node_modules/chromadb/dist/main/generated/api.js:17:19)
    at Module._compile (node:internal/modules/cjs/loader:1196:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1250:10)



### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Call any Chroma method to reproduce this issue

### Expected behavior

It should've inserted the documents
Issue: confusion about `Document` or string input for `embed_documents` function 
Labels: name: auto:bug, name: auto:improvement

### Issue you'd like to raise.

I'm using the `OpenAIEmbeddings` object to create embeddings for my list of documents. While it expects a list of strings, the `embed_documents` function seems misleading as to accept a list of documents. As a result, passing a list of documents raises `AttributeError` since `Document` objects do not have a `replace` method.

### Suggestion:

1. Add a `replace` method to `Document` (or)
2. Extend the function `embed_documents` to handle `Document` objects (or)
3. Rename the function to suit handling a list of strings

I will be glad to implement one of them as a first issue!
docs: Update documentation for Wikipedia tool to use WikipediaQueryRun
Labels: name: auto:documentation

**Description**
In the following page, "Wikipedia" tool is explained.
https://python.langchain.com/docs/modules/agents/tools/integrations/wikipedia

However, the WikipediaAPIWrapper being used is not a tool. This PR updated the documentation to use a tool WikipediaQueryRun.

**Issue**
None

**Tag maintainer**
Agents / Tools / Toolkits: @hinthornw
Update supabase match_docs ddl and notebook to use expected id type
Labels: name: auto:improvement, name: auto:documentation

  - Description: Switch supabase match function DDL to use expected uuid type instead of bigint
  - Issue: https://github.com/hwchase17/langchain/issues/6743, https://github.com/hwchase17/langchain/issues/7179
  - Tag maintainer:  @rlancemartin, @eyurtsev
  - Twitter handle: https://twitter.com/ShantanuNair
OpenAI Functions Agent not asking required parameter value
Labels: name: auto:bug

I was using openai functions agent with custom functions , the custom function(loan eligibility) needs three augments , state, age and income .

when i run the agent with question "how much can i borrow in state CA?" , it is directly calling function without asking age and income from the user .

below is the error 

pydantic.error_wrappers.ValidationError: 2 validation errors for LoanEligibilitySchema
age
  field required (type=value_error.missing)
income
  field required (type=value_error.missing)


how to fix this ?

### Suggestion:

_No response_
Structured Chat Agent cannot save_context when memory has additional input variables
Labels: name: auto:bug

### System Info

LangChain 0.0.225, Python 3.10, Windows

### Who can help?

@hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [X] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I am creating a structured chat agent using an `AgentExecutor.from_agent_and_tools`. I have added a custom variable `new_variable` to `input_variables`, and followed the instructions on how to add memory as per here: https://python.langchain.com/docs/modules/agents/agent_types/structured_chat

`def get_agent_executor(llm: ChatOpenAI, tools: list[Tool], chat_history: MessagesPlaceholder, memory: ConversationBufferMemory) -> AgentExecutor:
    input_variables = ["input", "agent_scratchpad", "chat_history", "new_variable"]
    prefix = CUSTOM_PREFIX
    suffix = CUSTOM_SUFFIX
    custom_prompt = StructuredChatAgent.create_prompt(tools, prefix=prefix, suffix=suffix,
                                                      input_variables=input_variables, memory_prompts=[chat_history])
    llm_chain = LLMChain(llm=llm, prompt=custom_prompt, verbose=True)
    convo_agent = StructuredChatAgent(llm_chain=llm_chain)
    agent_executor = AgentExecutor.from_agent_and_tools(agent=convo_agent, tools=tools, verbose=True, max_iterations=1,
                                                        memory=memory, memory_prompts=[chat_history], input_variables=input_variables,
                                                        handle_parsing_errors="Check your output and make sure it is a markdown code snippet of a json blob with a single action!")

    return agent_executor`
    
    This agent crashes every time at the end of the first iteration:
    `final_outputs: Dict[str, Any] = self.prep_outputs(
    self.memory.save_context(inputs, outputs)
      input_str, output_str = self._get_input_output(inputs, outputs)
    prompt_input_key = get_prompt_input_key(inputs, self.memory_variables)
  raise ValueError(f"One input key expected got {prompt_input_keys}")
ValueError: One input key expected got ['example_good_page', 'input']`

### Expected behavior

save_context without hiccup
Add verbose parameter for llamacpp
Labels: name: lgtm, name: auto:improvement

**Title:** Add verbose parameter for llamacpp

**Description:**
This pull request adds a 'verbose' parameter to the llamacpp module. The 'verbose' parameter, when set to True, will enable the output of detailed logs during the execution of the Llama model. This added parameter can aid in debugging and understanding the internal processes of the module.

The verbose parameter is a boolean that prints verbose output to stderr when set to True. By default, the verbose parameter is set to True but can be toggled off if less output is desired. This new parameter has been added to the `validate_environment` method of the `LlamaCpp` class which initializes the `llama_cpp.Llama` API:

```python
class LlamaCpp(LLM):
    ...
    @root_validator()
    def validate_environment(cls, values: Dict) -> Dict:
        ...
        model_param_names = [
            ...
            "verbose",  # New verbose parameter added
        ]
        ...
        values["client"] = Llama(model_path, **model_params)
        ...
```

**Issue:** 
Not applicable (If there is an issue that this PR resolves, please link to it here)

**Dependencies:** 
No new dependencies introduced.

**Maintainer:** 
Tagging @hinthornw, as this change relates to Tools / Toolkits.

**Twitter handle:** 
(If you want a shout-out on Twitter and have a Twitter handle, mention it here.)

This change does not introduce any new features or integrations, so no new tests or notebooks are provided. However, existing tests should cover this new parameter.

Maintainers, please review at your earliest convenience. Thank you for considering this contribution!
How to disable the OpenAI initialization when you're not using an OpenAI model
Labels: name: auto:bug

### Feature request

how to disable the OpenAI initialization when you're not using an OpenAI model.
[Please check this issue](https://github.com/hwchase17/langchain/issues/7189#issuecomment-1621931461)

### Motivation

 I am trying to build a VectorstoreIndexCreator using the following configuration

embeddings= SentenceTransformerEmbeddings embeddings
vectorstore_cls = Chroma
llm = HuggingfaceHub Model

Note: I am not using any openai model as llm or embedding purpose

here is the code
```
  from langchain.indexes import VectorstoreIndexCreator
  from langchain.vectorstores import Chroma
  from langchain.text_splitter import CharacterTextSplitter
  from langchain.embeddings import SentenceTransformerEmbeddings
  from langchain.llms import HuggingFaceHub
  from langchain.document_loaders import PyPDFLoader
  loader = PyPDFLoader("pdffile.pdf")

  embeddings =  SentenceTransformerEmbeddings(model_name="paraphrase-MiniLM-L6-v2")
  model = HuggingFaceHub(repo_id="facebook/mbart-large-50",
                       model_kwargs={"temperature": 0, "max_length":200},
                       huggingfacehub_api_token=HUGGING_FACE_API_KEY)
  index = VectorstoreIndexCreator(
                                  embedding=embeddings,
                                  vectorstore_cls=Chroma,
                                  text_splitter=CharacterTextSplitter(chunk_size=100,chunk_overlap=0)
                                  ).from_loaders([loader])
  result = index.query(llm=model,question=query,chain_type="refine")
```
But still iam getting the open_ai key dependency error when i run the code

```
Did not find openai_api_key, please add an environment variable `OPENAI_API_KEY` which contains it, or pass  `openai_api_key` as a named parameter. (type=value_error)
```
using langchain version: langchain==0.0.219

### Your contribution

To disable the OpenAI initialization when you're not using an OpenAI model
Does LlamaCPP currently not support the gpu acceleration of mps when n_gpu_layer to 1?
Labels: name: auto:bug

### System Info

langchain : 0.0.223
os: mac Ventura 13.4.1 max
python: 3.11.3

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I have compiled the llama.cpp project into mps, and tested the support for gpu acceleration under mps through the command, but when calling the cpp model in langchain, I found， When I set n_gpu_layer to 1, the gpu acceleration of mps is not turned on， below  is the code and output by it.

callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])
    llm=LlamaCpp(model_path='./zh-models/33B/ggml-model-q4_K.bin', n_ctx="2048", n_gpu_layers=1, callback_manager = callback_manager, verbose = True)
    llm('tell me a joke')

`llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 49954
llama_model_load_internal: n_ctx      = 2048
llama_model_load_internal: n_embd     = 6656
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 52
llama_model_load_internal: n_layer    = 60
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: n_ff       = 17920
llama_model_load_internal: model size = 30B
llama_model_load_internal: ggml ctx size =    0.14 MB
llama_model_load_internal: mem required  = 19884.88 MB (+ 3124.00 MB per state)
llama_new_context_with_model: kv self size  = 3120.00 MB
AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | VSX = 0 | 
`


### Expected behavior

support mps gpu acceleration
Make tqdm for OpenAIEmbeddings optional
Labels: name: auto:improvement, name: auto:documentation

  - Description: I have added a `show_progress_bar` parameter (defaults.to `False`) to the `OpenAIEmbeddings`. If the user sets `show_progress_bar` to `True`, a progress bar will be displayed.
  - Issue: #7246
  - Dependencies: N/A
  - Tag maintainer: @hwchase17, @baskaryan
  - Twitter handle: N/A
Make tqdm optional for OpenAIEmbeddings
Labels: name: auto:improvement

### Feature request

Add `show_progress_bar` within `OpenAIEmbeddings` class.

### Motivation

Simply speaking, 
 1. Showing progress bar within existing progress bar is generally not a good practice. Most of time it will break.
 2. There might be people who want to keep their console quiet.


### Your contribution

I will make a PR
DOC: Pinecone functions need more documentation
Labels: name: auto:documentation

### Issue with current documentation:

A continuation of #7126,

Pinecone features in Langchain that are not mentioned in Langchain's documentation. https://python.langchain.com/docs/modules/data_connection/vectorstores/integrations/pinecone

### Idea or request for content:

Add documentation for the other Pinecone functions.
Add missing token_max in reduce.py acombine_docs
Labels: name: lgtm, name: auto:bug

Replace this comment with:
  - Description: reduce.py reduce chain implementation's acombine_docs call does not propagate token_max. Without this, the async call will end up using 3000 tokens, the default, for the collapse chain.
  - Tag maintainer: @hwchase17 @agola11 @baskaryan 
  - Twitter handle: https://twitter.com/ShantanuNair

Related PR: https://github.com/hwchase17/langchain/pull/7201 and https://github.com/hwchase17/langchain/pull/7204
Updating for anthropic v0.3.2
Labels: name: auto:improvement

In the latest version of Anthropic-python-sdk - the api_url variable in the anthropic.Client has been updated to base_url variable. But if we access the same using langchain - it results in an error as explained in the - #6400 . In this pull request, I have updated the variable, which helps us to stay upto mark with Anthropic v0.3.2 

Maintainer responsibilities:
  - Models / Prompts: @hwchase17, @baskaryan
OpenAPISpec functions can get stuck in a loop
Labels: name: auto:bug

### System Info

langchain: '0.0.218'
windows 10

### Who can help?

@dev2049 @homanp

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [X] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Run the following:
```py
from langchain.chains.openai_functions.openapi import get_openapi_chain
chain = get_openapi_chain("https://chat-web3-plugin.alchemy.com/openapi.yaml")
chain.run("DOES NOT MATTER")
```
Results in endless loop
```shell
Attempting to load an OpenAPI 3.0.0 spec.  This may result in degraded performance. Convert your OpenAPI spec to 3.1.* spec for better support.
Attempting to load an OpenAPI 3.0.0 spec.  This may result in degraded performance. Convert your OpenAPI spec to 3.1.* spec for better support.
Attempting to load an OpenAPI 3.0.0 spec.  This may result in degraded performance. Convert your OpenAPI spec to 3.1.* spec for better support.
Attempting to load an OpenAPI 3.0.0 spec.  This may result in degraded performance. Convert your OpenAPI spec to 3.1.* spec for better support.
Attempting to load an OpenAPI 3.0.0 spec.  This may result in degraded performance. Convert your OpenAPI spec to 3.1.* spec for better support.
Attempting to load an OpenAPI 3.0.0 spec.  This may result in degraded performance. Convert your OpenAPI spec to 3.1.* spec for better support.
Attempting to load an OpenAPI 3.0.0 spec.  This may result in degraded performance. Convert your OpenAPI spec to 3.1.* spec for better support.
Attempting to load an OpenAPI 3.0.0 spec.  This may result in degraded performance. Convert your OpenAPI spec to 3.1.* spec for better support.
...
```

### Expected behavior

Loop should break
ZeroShotAgent prompt fixes
Labels: name: auto:improvement

  - Description: Reformatting prompt for ZeroShotAgent. The current prompt for the ZeroShotAgent presents limitations when getting the response from the agents. The purpose of this PR is to be more specific in its prompt in order to achieve the response in the expected format.
  - Issue: None
  - Dependencies: None
  - Tag maintainer: @hinthornw (for Agents) @hwchase17 @baskaryan (for Prompts)
  - Twitter handle: @goyoreyes
store template and template args in prompt output
Labels: name: auto:improvement

This exposes the original template + variables used in prompts to downstream consumers such as the callbacks system or LLM libraries. It does this by altering the template-formatting system to use a subclass of `str` rather than just a `str`. This is the most pythonic way I could find to accomplish this (There is also UserString, but that doesn't subclass `str` so it fails `isinstance` checks)

What this specifically allows is that libraries that monitor API libraries like the OpenAI SDK, can look at the prompt that the API was called with (a completion text or a messages array) and inspect the objects to pull out the template (on `.template`) or the template arguments (on `.template_args`)

(I spent some time trying to do this with callbacks, but the problem I ran into is that you have access to the prompt + variables in `on_chain_start` but there isn't an easy way to tie this back to the actual request sent to the OpenAI and this data doesn't seem to be captured by `on_llm_start`, and these are called in a way that makes it hard to capture any data and have it visible from the openai `create()` methods)

If this approach feels right, I can go ahead and write some unit tests for it as a part of this PR.

Not sure who to tag for this, so tagging @baskaryan and @hwchase17
feat: support generate as an early stopping method for `OpenAIFunctionsAgent`
Labels: name: lgtm, name: auto:enhancement

This PR proposes an implementation to support `generate` as an `early_stopping_method` for the new `OpenAIFunctionsAgent` class.  

The motivation behind is to facilitate the user to set a maximum number of actions the agent can take with `max_iterations` and force a final response with this new agent (as with the `Agent` class).

The following changes were made:

- The `OpenAIFunctionsAgent.return_stopped_response` method was overwritten to support `generate` as an `early_stopping_method`
- A boolean `with_functions` parameter was added to the `OpenAIFunctionsAgent.plan` method

This way the `OpenAIFunctionsAgent.return_stopped_response` method can call the `OpenAIFunctionsAgent.plan` method with `with_function=False` when the `early_stopping_method` is set to `generate`, making a call to the LLM with no functions and forcing a final response from the `"assistant"`.

  - Relevant maintainer: @hinthornw
  - Twitter handle: @aledelunap
Salesforce XGen integration
Labels: name: auto:enhancement

### Feature request

Still new here so I mimicked the setup of `llm/llamacpp.py` wrapper for the draft of Salesforce's new LLM [XGen](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/discussions/91). 

### Motivation

Expand the LLM sets integrated in Langchain.

### Your contribution

PR #7221
Draft: Starting Salesforce XGen model integration
Labels: name: auto:enhancement

Description: Salesforce recently released the LLM [XGen](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/discussions/91). This change is to begin integration of the model into Langchain.
Issue: the issue # it fixes (if applicable),
Dependencies: transformers
Tag maintainer: @hwchase17 @baskaryan 

Still new here so marking as Draft to ensure further development is okay. Mimicked design of llamacpp wrapper to start.