delayedkarma/langchain-issues huggingface dataset:
Embedding very sensitive to punctuation
Labels: name: area: embeddings, name: area: vector store, name: auto:question

### System Info

I have been using OpenAI Embeddings specifically text-embedding-ada-002 and noticed it was very sensitive to punctuation even. I have around 1000 chunks and need to extract each time the 15 most similar chunks to my query. I have been testing my query without punctuation and when I add a dot '.' at the end of my query it changes the initial set I got from the retriever with the query without punctuation (some chunks are the same but new ones may appear or the initial order is different).

- Have you noticed anything similar ?
- Is it the basic behaviour of this embedding to be that sensitive to punctuation ?
- Is there a way to make it more robust to minor changes in the query ?

FYI: I am using PGvector to store my chunks vectors

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [X] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

-

### Expected behavior

-
`PythonREPL` removal from langchain library
Labels: name: auto:bug

### System Info

Langchain version: 0.0.346
Python version: 3.9.16

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

`PythonREPL` which has been moved to `experimental` still exists in based library under path:

`libs/langchain/langchain/utilities/python.py`

which triggers security scans vulnerabilities (`exec()` call) and doesn't allow us to use the package on the production environment.

Since 

https://nvd.nist.gov/vuln/detail/CVE-2023-39631

Should be most likely closed soon, this is only vulnerability that would have to be addressed so we can freely use `langchain`.

### Expected behavior

`PythonREPL` should only exist in `experimental` version of `langchain`
Support SageMaker Inference Component of multi model endpoint
Labels: name: auto:enhancement, name: area: models, name: integration: aws

### Feature request

I have created a multi model inference endpoint in the new version of SageMaker studio (the original one now is called studio classic). I don't see there is a place that I can set the `inference component` in the `SagemakerEndpoint` class. So I ended up get an expected error from SageMaker.

`An error occurred (ValidationError) when calling the InvokeEndpointWithResponseStream operation: Inference Component Name header is required for endpoints to which you plan to deploy inference components. Please include Inference Component Name header or consider using SageMaker models.`

### Motivation

To support multi model endpoint in SageMaker which is cost efficient way to run models.

### Your contribution

I can test and verify.
Specific prompt adds false input variables
Labels: name: area: agent, name: auto:bug, name: area: models

### System Info

python3.10.13
langchain==0.0.346
langchain-core==0.0.10

### Who can help?

@agola11, @hwchase17

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction
```

from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.agents.output_parsers import OpenAIFunctionsAgentOutputParser
from langchain.chat_models import ChatOpenAI
from langchain.agents.format_scratchpad import format_to_openai_function_messages

text_with_call = """You are a helpful assistant. Here is a function call that you should not imitate: <functioncall> {"name":"generate_anagram", "arguments": {"word": "listen"}} 
    """

prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            text_with_call,
        ),
        ("user", "{input}"),
        MessagesPlaceholder(variable_name="agent_scratchpad"),
    ]
)

llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)

agent = (
    {
        "input": lambda x: x["input"],
        "agent_scratchpad": lambda x: format_to_openai_function_messages(
            x["intermediate_steps"]
        ),
    }
    | prompt
    | llm
    | OpenAIFunctionsAgentOutputParser()
)
output = agent.invoke(
    {
        "input": "Hello",
        "intermediate_steps": [],
    }
)
```

### Expected behavior

I would expect not to get an error.

I get error: `KeyError: 'Input to ChatPromptTemplate is missing variable \'"name"\'.  Expected: [\'"name"\', \'agent_scratchpad\', \'input\'] Received: [\'input\', \'agent_scratchpad\']'`

> I think this error is cause by the f-string prompt template recognising the brackets inside the prompt as other variables needed an input. 
I tried using Jinja2 template for my prompt instead, but I cannot setup this option in `ChatPromptTemplate`.
I understand this could be for security reasons as mentioned in [this issue](https://github.com/langchain-ai/langchain/issues/4394)

So the possible solutions I see :
- Use PromptTemplate like:
```
prompt = PromptTemplate.from_template(
    text_with_call, template_format="jinja2"
)
```
But I would like to make use of the `agent_scratchpad`, so the ChatPromptTemplate is needed in my case.
- Change ChatPromptTemplate class to support jinja2 template, which I understand could not be done
- Re-implement my custom ChatPromptTemplate
- find another way to accept this prompt without falsely flagging prompt elements as input variables.

Do you have any ideas? Thanks for your help 😃
Error: 
Labels: name: area: vector store, name: auto:bug, name: integration: chroma

### System Info

I try this example code

```
from langchain.retrievers import ParentDocumentRetriever
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.storage import InMemoryStore

# This text splitter is used to create the parent documents
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)
# This text splitter is used to create the child documents
# It should create documents smaller than the parent
child_splitter = RecursiveCharacterTextSplitter(chunk_size=400)
# The vectorstore to use to index the child chunks
vectorstore = Chroma(embedding_function=OpenAIEmbeddings())
# The storage layer for the parent documents
store = InMemoryStore()

vectorstore = Chroma(collection_name="test", embedding_function=OpenAIEmbeddings())
```

# Initialize the retriever
parent_document_retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=store,
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)

but I encountered an error:

```
 1 # Initialize the retriever
----> 2 parent_document_retriever = ParentDocumentRetriever(
      3     vectorstore=vectorstore,
      4     docstore=store,
      5     child_splitter=child_splitter,

TypeError: MultiVectorRetriever.__init__() got an unexpected keyword argument 'child_splitter'
```

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
from langchain.retrievers import ParentDocumentRetriever
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.storage import InMemoryStore

# This text splitter is used to create the parent documents
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)
# This text splitter is used to create the child documents
# It should create documents smaller than the parent
child_splitter = RecursiveCharacterTextSplitter(chunk_size=400)
# The vectorstore to use to index the child chunks
vectorstore = Chroma(embedding_function=OpenAIEmbeddings())
# The storage layer for the parent documents
store = InMemoryStore()

vectorstore = Chroma(collection_name="test", embedding_function=OpenAIEmbeddings())
# Initialize the retriever
parent_document_retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=store,
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)
```

### Expected behavior

I can run.
When I install the template "self-query-qdrant" I get this error: UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 132: character maps to <undefined>
Labels: name: auto:bug

### System Info

Windows 10 Pro - 22H2 - Build 9045.3693 - Windows Feature Experience Pack 1000.19053.1000.0
Python 3.11.5
langchain-cli 0.0.19


### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. creating a virtual environments with python - m venv .venv and activate it;
2. installing langchain-cli with pip install -U "langchain-cli[serve]"
3. launch langchain app new qdrant-app --package self-query-qdrant 

### Expected behavior

As for other templates that I have installed, I expected to find the .py files of the app in the packages directory, but it's empty.
[log.txt](https://github.com/langchain-ai/langchain/files/13580197/log.txt)
Requests to the Creates a completion for the chat message Operation under Azure OpenAI API version 2023-03-15-preview have exceeded token rate limit of your current OpenAI S0 pricing tier. Please go here: https://aka.ms/oai/quotaincrease if you would like to further increase the default rate limit.
Labels: name: area: agent, name: auto:question, name: area: models

### System Info

langchain 0.0.299 python 3.8  gpt-4
when AzureChatOpenAI use gpt-4 ,I use agents to deal with problems,raised a error about RateLimitError; but gpt-3.5 do not appear this quesition 



### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

llm = AzureChatOpenAI(
            openai_api_base=api_base,
            openai_api_version=api_version,
            deployment_name=deployment_name,
            openai_api_key=api_token,
            openai_api_type="azure",
            max_tokens = max_tokens,
            model_name = model_name,
          )

memory = ConversationBufferMemory(memory_key="chat_history")

agent = create_pandas_dataframe_agent(
    llm,
    df,
    agent=AgentType.CONVERSATIONAL_REACT_DESCRIPTION,
    verbose=True,
    memory=memory
)
question = "what are the top 2 most polluted counties"

res = agent.run(question)


### Expected behavior

Tell me what caused this error and how to avoid it
api_base parameter not properly propagated in client calls in ChatLiteLLM
Labels: name: auto:bug, name: area: models

### System Info

- LangChain version: 0.0.346
- Platform: Mac mini M1 16GB - macOS Sonoma 14.0
- Python version: 3.11
- LiteLLM version: 1.10.6

### Who can help?

@hwchase17
@agola11

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
from langchain.chat_models import ChatLiteLLM

# Code for initializing the ChatLiteLLM instance
chat_model = ChatLiteLLM(api_base="https://custom.endpoints.huggingface.cloud", model="huggingface/Intel/neural-chat-7b-v3-1")

# Make a call to LiteLLM
text = "What would be a good company name for a company that makes colorful socks?"
messages = [HumanMessage(content=text)]

print(chat_model(messages).content)
```

Error:

```
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/litellm/utils.py", line 4919, in handle_huggingface_chunk
    raise ValueError(chunk)
ValueError: {"error":"The model Intel/neural-chat-7b-v3-1 is too large to be loaded automatically (14GB > 10GB). Please use Spaces (https://huggingface.co/spaces) or Inference Endpoints (https://huggingface.co/inference-endpoints)."}
```

Is the same error if:
`chat_model = ChatLiteLLM(model="huggingface/Intel/neural-chat-7b-v3-1")`

So api_base parameter not properly propagated in client calls in ChatLiteLLM.

### Expected behavior

I would expect the ChatLiteLLM instance to correctly utilize the api_base parameter when making requests to the LiteLLM client. This should enable using models larger than the default size limit without encountering the error message about model size limits.

Notably, if I explicitly add the api_base argument in chat_models/litellm.py on line 239 (e.g., `return self.client.completion(api_base=self.api_base, **kwargs)`), the problem is resolved. This suggests that the api_base argument is not being correctly passed through **kwargs.
Get JSON output from non-OpenAI models
Labels: name: auto:bug, name: auto:question, name: area: models

### Issue with current documentation:

I'm exploring the use of open-source models via Ollama excellent application, I'm facing quite some challenges in adapting the instructions that were made for OpenAI-type models (they are perfectly interchangeable, I must say). 

Is the format_instructions generated from the PydanticOutputParser supposed to work also for non-Openai models? 
With Zephyr the model keeps returning the answer AND the JSON schema, even when I try to stop it from doing that... 

### Idea or request for content:

_No response_
Exclude `max_tokens` from request if it's None
Labels: name: lgtm, name: auto:bug, name: area: models, name: size:S

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->


We found a request with `max_tokens=None` results in the following error in Anthropic:

```
HTTPError: 400 Client Error: Bad Request for url: https://oregon.staging.cloud.databricks.com/serving-endpoints/corey-anthropic/invocations. 
Response text: {"error_code":"INVALID_PARAMETER_VALUE","message":"INVALID_PARAMETER_VALUE: max_tokens was not of type Integer: null"}
```

This PR excludes `max_tokens` if it's None.
Add Alpha Vantage API Tool
Labels: name: auto:enhancement, name: size:L

### Description
This implementation adds functionality from the AlphaVantage API, renowned for its comprehensive financial data. The class encapsulates various methods, each dedicated to fetching specific types of financial information from the API. 

### Implemented Functions

- **`search_symbols`**: 
  - Searches the AlphaVantage API for financial symbols using the provided keywords.

- **`_get_market_news_sentiment`**: 
  - Retrieves market news sentiment for a specified stock symbol from the AlphaVantage API.

- **`_get_time_series_daily`**: 
  - Fetches daily time series data for a specific symbol from the AlphaVantage API.

- **`_get_quote_endpoint`**: 
  - Obtains the latest price and volume information for a given symbol from the AlphaVantage API.

- **`_get_time_series_weekly`**: 
  - Gathers weekly time series data for a particular symbol from the AlphaVantage API.

- **`_get_top_gainers_losers`**: 
  - Provides details on top gainers, losers, and most actively traded tickers in the US market from the AlphaVantage API.

  ### Issue: 
  - #11994 
  
### Dependencies: 
  - 'requests' library for HTTP requests. (import requests)
  - 'pytest' library for testing. (import pytest)
No module named 'langchain.callbacks.tracers.log_stream'
Labels: name: auto:bug

### System Info

I already create venv and re-install langchain: `pip install langchain`   but error:
>     from langserve import add_routes
>   File "/usr/local/lib/python3.10/dist-packages/langserve/__init__.py", line 7, in <module>
>     from langserve.client import RemoteRunnable
>   File "/usr/local/lib/python3.10/dist-packages/langserve/client.py", line 29, in <module>
>     from langchain.callbacks.tracers.log_stream import RunLogPatch
> ModuleNotFoundError: No module named 'langchain.callbacks.tracers.log_stream'
> 

### Who can help?

@agola11

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [X] Callbacks/Tracing
- [ ] Async

### Reproduction

$ python3 -m venv venv
$ source venv/bin/activate
$ pip install langchain
$ pip install google-generativeai
$ pip install fastapi
$ pip install uvicorn
$ pip install langserve
$ `uvicorn main:app --host 0.0.0.0 --port 8000`

### Expected behavior

App run with langserve without error
Issue: The purpose to validate_tools_single_input in CHAT_CONVERSATIONAL_REACT_DESCRIPTION agent
Labels: name: area: agent, name: auto:question

### Issue you'd like to raise.

Cannot figure out why validate_tools_single_input is necessary. I've tried to adopt multi-inputs tools in CHAT_CONVERSATIONAL_REACT_DESCRIPTION agents by just commenting out the implementation of such function. Everything seemed fun. 
So what's the purpose of such design, and if it is possible to use multi-inputs tools in a more flexible setup?

### Suggestion:

_No response_
Allow PresidioAnonymizer() to redact data instead of anonymizing.
Labels: name: auto:enhancement

### Feature request

The langchain library only supports anonymizing it seems. The native microsoft library can redact data:

```
analyzer_results = analyzer.analyze(text=text_to_anonymize,
         entities=["PERSON", "PHONE_NUMBER", "EMAIL_ADDRESS", "URL", "LOCATION"],
         ad_hoc_recognizers=ad_hoc_recognizers,
         language='en')

  anonymized_results = anonymizer.anonymize(
       text=text_to_anonymize,
       analyzer_results=analyzer_results,
       operators={"DEFAULT": OperatorConfig("redact", {})}
 )
```

I don't see a way to do this via langchain_experimental.data_anonymizer

### Motivation

Redaction is cleaner than anonymizing. It's better than replacing first names with gibberish for my. use case

### Your contribution

I can help provide use-cases, like i did above.
Issue: <Please write a coImportError: cannot import name 'open_filename' from 'pdfminer.utils' (/home/nickjtay/Projects/langtest1/lib/python3.10/site-packages/pdfminer/utils.py)mprehensive title after the 'Issue: ' prefix>
Labels: name: area: doc loader, name: auto:bug

### Issue you'd like to raise.

I tried to run the notebook [Semi_structured_multi_modal_RAG_LLaMA2.ipynb](https://github.com/langchain-ai/langchain/blob/master/cookbook/Semi_structured_multi_modal_RAG_LLaMA2.ipynb). I installed and imported the required libraries, but an error message was returned as a result of running the below code: 

```
from typing import Any

from pydantic import BaseModel
from unstructured.partition.pdf import partition_pdf

path = "/home/nickjtay/LLaVA/"

raw_pdf_elements = partition_pdf(
    filename=path + "LLaVA.pdf",
    # Using pdf format to find embedded image blocks
    extract_images_in_pdf=True,
    # Use layout model (YOLOX) to get bounding boxes (for tables) and find titles
    # Titles are any sub-section of the document
    infer_table_structure=True,
    # Post processing to aggregate text once we have the title
    chunking_strategy="by_title",
    # Chunking params to aggregate text blocks
    # Attempt to create a new chunk 3800 chars
    # Attempt to keep chunks > 2000 chars
    # Hard max on chunks
    max_characters=4000,
    new_after_n_chars=3800,
    combine_text_under_n_chars=2000,
    image_output_dir_path=path,
)
```

```
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[5], line 4
      1 from typing import Any
      3 from pydantic import BaseModel
----> 4 from unstructured.partition.pdf import partition_pdf
      5 import pdfminer
      6 print(pdfminer.utils.__version__)

File ~/Projects/langtest1/lib/python3.10/site-packages/unstructured/partition/pdf.py:40
     38 from pdfminer.pdfparser import PSSyntaxError
     39 from pdfminer.pdftypes import PDFObjRef
---> 40 from pdfminer.utils import open_filename
     41 from PIL import Image as PILImage
     43 from unstructured.chunking.title import add_chunking_strategy

ImportError: cannot import name 'open_filename' from 'pdfminer.utils' (/home/nickjtay/Projects/langtest1/lib/python3.10/site-packages/pdfminer/utils.py)
​
```
### Suggestion:

_No response_
docs[patch]: fix ipynb links
Labels: name: auto:documentation, name: size:S

Keeping it simple for now.

Still iterating on our docs build in pursuit of making everything mdxv2 compatible for docusaurus 3, and the fewer custom scripts we're reliant on through that, the less likely the docs will break again.

Other things to consider in future:

Quarto rewriting in ipynbs: https://quarto.org/docs/extensions/nbfilter.html (but this won't do md/mdx files)

Docusaurus plugins for rewriting these paths