delayedkarma/langchain-issues huggingface dataset:
Myscale/improve string pattern match
Labels: name: auto:improvement

- Description: Improved prompts for comparator `LIKE` and `CONTAIN`. They are ambiguous under some circumstances like "Answer this questions using papers published by Geoffrey Hinton". In that example LLM tends to use `LIKE` instead of `CONTAIN` to column `authors`, which is actually a column with type list of strings.

We revised the prompt to address this problem:
1. Changed prompt for comparator `LIKE` to `string_pattern_like`
2. Changed prompt for comparator `CONTAIN` to `list_contain`

@hwchase17 
<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  3. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @dev2049
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @dev2049
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @vowelparrot
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Fixed bug in AnalyticDB Vector Store caused by upgrade SQLAlchemy version

Fixed bug in AnalyticDB VectorStore caused by the SQLAlchemy upgrade version to 1.4.47

The new version no longer supports the `connection.commit()` function, resulting in the error: AttributeError: 'Connection' object has no attribute 'commit'. Replace it with `with conn.begin():`.

This commit is related to VectorStores. @rlancemartin @eyurtsev
Update the token_max value in mp_reduce
Labels: name: auto:improvement

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: When I use the gpt-3.5-turbo-16k model, I often encounter this ValueError 'A single document was so long it could not be combined' due to the small token_max.I always need to modify a larger value for token_max to resolve the issue.I hope to enhance the functionality by dynamically modifying the value of token_max through obtaining the model used by the current chain.
  - Issue: The issue of encountering frequent errors with long documents when using the large tokens model has been addressed and fixed.
  - Dependencies: We need a more dynamic way of obtaining the model name in order to retrieve the maximum tokens of the model. This aspect still requires further improvement.
  - Tag maintainer: @hwchase17 @dev2049 @vowelparrot
feat: fix the Chinese characters in the solution content will be conv…
Labels: name: lgtm

fix the Chinese characters in the solution content will be converted to ascii encoding, resulting in an abnormally long number of tokens


<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @dev2049
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @dev2049
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @vowelparrot
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Throw error if trying to deserialize python prompts
Labels: name: area: doc loader, name: auto:bug

Removes the ability to deserialize Python prompts and throws an error informing of the security issues with a link to documentation on how to use JSON and YAML instead. 

The docs don't even talk about Python, and the saving function in the base prompt class doesn't support saving to Python either. I'm guessing this is something left in for backwards compatibility, but I think due to the CVEs we should just remove it. 


Fixes #6627
Fixes #4849
TypeError: __init__() takes exactly 1 positional argument (2 given)
Labels: name: bug

### System Info

langchain:0.0.215

### Who can help?

@hwchase17 

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
from langchain.indexes import VectorstoreIndexCreator
from langchain.embeddings import OpenAIEmbeddings
index = VectorstoreIndexCreator(embedding=OpenAIEmbeddings('gpt2')).from_loaders([loader])


```

error:
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[6], line 3
      1 from langchain.indexes import VectorstoreIndexCreator
      2 from langchain.embeddings import OpenAIEmbeddings
----> 3 index = VectorstoreIndexCreator(embedding=OpenAIEmbeddings('gpt2')).from_loaders([loader])

File [~/micromamba/envs/openai/lib/python3.11/site-packages/pydantic/main.py:332], in pydantic.main.BaseModel.__init__()

TypeError: __init__() takes exactly 1 positional argument (2 given)
```


### Expected behavior

no
Grobid parser for Scientific Articles from PDF
Labels: name: enhancement, name: area: doc loader

### Scientific Article PDF Parsing via Grobid

`Description:`
This change adds the GrobidParser class, which uses the Grobid library to parse scientific articles into a universal XML format containing the article title, references, sections, section text etc. The GrobidParser uses a local Grobid server to return PDFs document as XML and parses the XML to optionally produce documents of individual sentences or of whole paragraphs. Metadata includes the text, paragraph number, pdf relative bboxes, pages (text may overlap over two pages), section title (Introduction, Methodology etc), section_number (i.e 1.1, 2.3), the title of the paper and finally the file path.
      
Grobid parsing is useful beyond standard pdf parsing as it accurately outputs sections and paragraphs within them. This allows for post-fitering of results for specific sections i.e. limiting results to the methodology section or results. While sections are split via headings, ideally they could be classified specifically into introduction, methodology, results, discussion, conclusion. I'm currently experimenting with chatgpt-3.5 for this function, which could later be implemented as a textsplitter. 

`Dependencies:`
For use, the grobid repo must be cloned and Java must be installed, for colab this is:

```
!apt-get install -y openjdk-11-jdk -q
!update-alternatives --set java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
!git clone https://github.com/kermitt2/grobid.git
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-11-openjdk-amd64"
os.chdir('grobid')
!./gradlew clean install
```

Once installed the server is ran on localhost:8070 via
```
get_ipython().system_raw('nohup ./gradlew run > grobid.log 2>&1 &')
```

@rlancemartin, @eyurtsev

Twitter Handle: @Corranmac

Grobid Demo Notebook is [here](https://colab.research.google.com/drive/1X-St_mQRmmm8YWtct_tcJNtoktbdGBmd?usp=sharing).
prevent DuckDuckGoSearchAPIWrapper from consuming top result

remove the `next` call that checks for None on the results generator

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: remove the `next` call that checks for None on the results generator
  - Issue: #6724 ,
  - Dependencies: none,
  - Tag maintainer: @vowelparrot,
  - Twitter handle: gabrielaltay
  - 
If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @dev2049
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @dev2049
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @vowelparrot
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
ValueError: Unknown encoding text-embedding-ada-002
Labels: name: bug

### System Info

no

### Who can help?

@hwchase17 @agola11 

### Information

- [x] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [x] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction



when running the following code:

```
from langchain.embeddings import OpenAIEmbeddings

embedding_model = OpenAIEmbeddings()
embeddings = embedding_model.embed_documents(
    [
        "Hi there!",
        "Oh, hello!",
        "What's your name?",
        "My friends call me World",
        "Hello World!"
    ]
)
```



such errors occurred:

```
ValueError                                Traceback (most recent call last)
Cell In[9], line 1
----> 1 embeddings = embedding_model.embed_documents(
      2     [
      3         "Hi there!",
      4         "Oh, hello!",
      5         "What's your name?",
      6         "My friends call me World",
      7         "Hello World!"
      8     ]
      9 )
     10 len(embeddings), len(embeddings[0])

File ~/micromamba/envs/openai/lib/python3.11/site-packages/langchain/embeddings/openai.py:305, in OpenAIEmbeddings.embed_documents(self, texts, chunk_size)
    293 """Call out to OpenAI's embedding endpoint for embedding search docs.
    294 
    295 Args:
   (...)
    301     List of embeddings, one for each text.
    302 """
    303 # NOTE: to keep things simple, we assume the list may contain texts longer
    304 #       than the maximum context and use length-safe embedding function.
--> 305 return self._get_len_safe_embeddings(texts, engine=self.deployment)

File ~/micromamba/envs/openai/lib/python3.11/site-packages/langchain/embeddings/openai.py:225, in OpenAIEmbeddings._get_len_safe_embeddings(self, texts, engine, chunk_size)
    223 tokens = []
    224 indices = []
--> 225 encoding = tiktoken.get_encoding(self.model)
    226 for i, text in enumerate(texts):
    227     if self.model.endswith("001"):
    228         # See: https://github.com/openai/openai-python/issues/418#issuecomment-1525939500
    229         # replace newlines, which can negatively affect performance.

File ~/micromamba/envs/openai/lib/python3.11/site-packages/tiktoken/registry.py:60, in get_encoding(encoding_name)
     57     assert ENCODING_CONSTRUCTORS is not None
     59 if encoding_name not in ENCODING_CONSTRUCTORS:
---> 60     raise ValueError(f"Unknown encoding {encoding_name}")
     62 constructor = ENCODING_CONSTRUCTORS[encoding_name]
     63 enc = Encoding(**constructor())

ValueError: Unknown encoding text-embedding-ada-002
```

how to fix it?

### Expected behavior

no
DuckDuckGoSearchAPIWrapper Consumes results w/o returning them
Labels: name: bug, name: needs work

### System Info

langchain: 0.0.215
platform: ubuntu 22.04 LTS
python: 3.10

### Who can help?

@eyurtsev :) 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

https://colab.research.google.com/drive/1vraycYUuF-BZ0UA8EoUT0hPt15HoeGPV?usp=sharing

### Expected behavior

The first result of DuckDuckGoSearch to be returned in the `get_snippets` and `results` methods of the DuckDuckGoSearchAPIWrapper.
https://python.langchain.com/docs/use_cases/question_answering/document-context-aware-QA code is not working,
Labels: name: bug, name: documentation

### System Info

colab

### Who can help?

@hwchase17 hi i am trying to use Context aware text splitting and QA / Chat


the code doesnt work starting the vectore indes

# Build vectorstore and keep the metadata
from langchain.vectorstores import Chroma
vectorstore = Chroma.from_documents(texts=all_splits,metadatas=all_metadatas,embedding=OpenAIEmbeddings())






### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

i am only following the notebood 

### Expected behavior

it should work
Handling larger responses in the API chain
Labels: name: enhancement

### Feature request

Currently, the API chain takes the result from the API and gives it directly to a LLM chain. If the result is larger then the context length of the LLM, the chain will give an error. To address this issue I propose that the chain should split the API result using a text splitter, then give the result to combine documents chain that answers the question. 

### Motivation

I have found that certain question given to the API chain gives results from the API that exede the context length of the standard openai model. For example:  
```python
from langchain.chains import APIChain
from langchain.prompts.prompt import PromptTemplate
from langchain.llms import OpenAI

llm = OpenAI(temperature=0)

from langchain.chains.api import open_meteo_docs
chain_new = APIChain.from_llm_and_api_docs(llm, open_meteo_docs.OPEN_METEO_DOCS, verbose=True)

chain_new.run('What is the weather of [latitude:52.52, longitude:13.419998]?')
```
The LLM creates the this URL: https://api.open-meteo.com/v1/forecast?latitude=52.52&longitude=13.419998&hourly=temperature_2m,weathercode,snow_depth,freezinglevel_height,visibility.  The content from this URL is kind of large. This causes the prompt to have 6382 tokens and be lager then the max context length. I think the chain should be able to handle larger responses. 

### Your contribution

I can try to submit a PR to implement this solution.
ChromaDB Chroma.from_documents error:   AttributeError: 'Collection' object has no attribute 'upsert'
Labels: name: bug

### System Info

windows 11, python 3.9.16 , langchain-0.0.215

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

The loaded documents succesfully get indexd in FAISS, Deeplake and Pinecone, so I guess the issue is not there. When using  Chroma.from_documents the error is thrown:

```
  File ~\...langchain\vectorstores\chroma.py:149 in add_texts
    self._collection.upsert(
        AttributeError: 'Collection' object has no attribute 'upsert'
```
Code:
```python
with open('docs.pkl', 'rb') as file:
    documents= pickle.load(file)   
text_splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=200)
docs = text_splitter.split_documents(documents)
chroma_dir = base_dir + "ChromaDB" + "_" + str(chunk_size) + "_" + str(overlap)
db2 = Chroma.from_documents(docs, embeddings, persist_directory=chroma_dir)
```
Uploaded the pickle zipped as .pkl is not supported for upload.
[docs.zip](https://github.com/hwchase17/langchain/files/11860249/docs.zip)


### Expected behavior

Index the docs
search specific vector by tag or key
Labels: name: enhancement, name: question

Hi teams,

Can I make the embedding with tag or any key, I want to make user with different auth. to query specific vector of document.

I was trying to use Redis as vectorstore.Would it recommend to be my vectorstore if I want to implement something authentication.

I'll embed my document into vectorstore and each embedding has its permission. And then I'll have users with different permission. The user who be granted the permission of document can query that vector of document.

For example, Embed three docuemnts into vectorstore and there are two user with different permission.
| Docs   | key | 
| :----- | :--: | 
| Docs_A |  A  | 
| Docs_B |  B  | 
| Docs_C |  C |

| User| permission | 
| :----- | :--: | 
| User_A |  A, C |  
| User_B |  B | 

At this use case, User_A can query the vector of Docs_A and Docs_C, and User_B just Docs_B
DOC: some upgrades to  https://langchainers.hashnode.dev/getting-started-with-langchainjs
Labels: name: documentation, name: for-langchain-js

### Issue with current documentation:

i've made some comments on https://langchainers.hashnode.dev/getting-started-with-langchainjs

and https://langchainers.hashnode.dev/learn-how-to-integrate-language-models-llms-with-sql-databases-using-langchainjs
some things to update to make it working with the last langchain JS/TS packages

### Idea or request for content:

_No response_
Pinecone hybrid search is incomplete, missing add_documents method
Labels: name: bug, name: enhancement

### System Info

All regular retrievers have an add_document method that handles adding a list of documents, and this one retriever only handles add_text, a list of strings and not langchain documents. 

For comparison, weaviate hybrid search is able to handle document lists:

`   def add_documents(self, docs: List[Document], **kwargs: Any) -> List[str]:
        """Upload documents to Weaviate."""
        from weaviate.util import get_valid_uuid

        with self._client.batch as batch:
            ids = []
            for i, doc in enumerate(docs):
                metadata = doc.metadata or {}
                data_properties = {self._text_key: doc.page_content, **metadata}

                # If the UUID of one of the objects already exists
                # then the existing objectwill be replaced by the new object.
                if "uuids" in kwargs:
                    _id = kwargs["uuids"][i]
                else:
                    _id = get_valid_uuid(uuid4())

                batch.add_data_object(data_properties, self._index_name, _id)
                ids.append(_id)
        return ids`

### Who can help?

@hw

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Import PineconeHybridSearch


### Expected behavior

retriever.add_documents(DocumentList)
cannot import name 'get_callback_manager' from 'langchain.callbacks' 
Labels: name: bug

### System Info

Langchain 0.0.125, Python 3.9 within Databricks

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [X] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [X] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

from langchain import PromptTemplate, HuggingFaceHub, LLMChain

### Expected behavior

Installation without issues
Update the token_max value in mp_reduce.ValueError:"A single document was so long it could not be combined "、"A single document was longer than the context length,"
Labels: name: enhancement

### Feature request

The value of token_max in mp_reduce should vary according to the model, not the fixed 3000.
![image](https://github.com/hwchase17/langchain/assets/38323944/58ab4760-9787-40e1-a831-6b3a895d4806)


### Motivation

When I use the gpt-3.5-turbo-16k model, I often encounter this ValueError 'A single document was so long it could not be combined' due to the small token_max.I always need to modify a larger value for token_max to resolve the issue.
![image](https://github.com/hwchase17/langchain/assets/38323944/e4c8b4cc-23b8-48e0-ba09-160ab4888c80)


### Your contribution

I hope to enhance the functionality by dynamically modifying the value of token_max through obtaining the model used by the current chain.
![image](https://github.com/hwchase17/langchain/assets/38323944/a289a1c9-677b-4227-b3f4-b5d8c7620eb6)
A swift langchain copy, for ios or mac apps.
Labels: name: enhancement

Hi there, awesome project!

https://github.com/buhe/langchain-swift is a swift langchain copy, for ios or mac apps.
Chatbots , QA bot and  Agent is done.
Current step:
- [ ] LLMs
  - [x] OpenAI
- [ ] Vectorstore
  - [x] Supabase
- [ ] Embedding
  - [x] OpenAI
- [ ] Chain
  - [x] Base
  - [x] LLM
- [ ] Tools
  - [x] Dummy
  - [x] InvalidTool
  - [ ] Serper
  - [ ] Zapier
- [ ] Agent
  - [x] ZeroShotAgent
- [ ] Memory
    - [x] BaseMemory
    - [x] BaseChatMemory
    - [x] ConversationBufferWindowMemory
- [ ] Text Splitter
    - [x] CharacterTextSplitter
- [ ] Document Loader
    - [x] TextLoader

### Suggestion:

_No response_
How can i get execution order inside agent?
Labels: name: question

I defined some agents and some tools. 
 How can i get execution order about llm, tool inside agent? Use Callback or other? 
If i use  callback print, there results is in order? 
```
agent_orange = ChatAgent(llm_chain=orange_chain, allowed_tools=tool_names, output_parser=MyChatOutputParser())
agent_chatgpt = ChatAgent(llm_chain=chatgpt_chain, allowed_tools=tool_names, 

output_parser=MyChatOutputParser())
agent_executor = MyAgentExecutor.from_agent_and_tools(agent=[agent_orange, agent_chatgpt],
    tools=tools,
    verbose=True
)
result = agent_executor.run("my query", callbacks=[my_handle])
```