delayedkarma/langchain-issues huggingface dataset:
Wfh/whatsapp loader
Labels: name: area: doc loader, name: auto:enhancement

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->
Issue: vectorstore hence indexes and embeddings are persisting when they should not be.
Labels: name: area: vector store, name: auto:bug

### Issue you'd like to raise.

langchain==0.0.250
platform = macOS
Python version 3.11

I am not sure if this is by design, I think it is not and hence reporting this as an issue.

When looping through a list of pdf files to get a summary for each I am creating an index using - VectorstoreIndexCreator().from_documents(pages). 

The issue is for each subsequent file, data(documents) from previous file are also being retrieved to be passed on to GPT and end up in the summary.

Code:
    from langchain.indexes import VectorstoreIndexCreator
    from langchain.chains import RetrievalQA
    from langchain.chat_models import ChatOpenAI
    for file in files:
            loader = PyPDFLoader(file['path'])
            pages = loader.load_and_split()

            index = VectorstoreIndexCreator().from_documents(pages)

            retriever = index.vectorstore.as_retriever(search_type='mmr')
            retriever.search_kwargs['k'] = 10

            llm = ChatOpenAI()
            aca_qa = RetrievalQA.from_chain_type(
                   llm=llm,
                   retriever=retriever,
                   chain_type='stuff',
                   # return_source_documents=True,
            )
            result = aca_qa({'query': summary_query})


### Suggestion:

Temp fix:

The resolution is to include - index = '' at the beginning of each loop cycle
Add new document_loader: AssemblyAIAudioTranscriptLoader
Labels: name: lgtm, name: area: doc loader, name: auto:enhancement

This PR adds a new document loader `AssemblyAIAudioTranscriptLoader` that allows to transcribe audio files with the [AssemblyAI API](https://www.assemblyai.com) and loads the transcribed text into documents.

- Add new document_loader with class `AssemblyAIAudioTranscriptLoader`
- Add optional dependency `assemblyai`
- Add unit tests (using a Mock client)
- Add docs notebook

This is the equivalent to the JS integration already available in LangChain.js. See the [LangChain JS docs AssemblyAI page](https://js.langchain.com/docs/modules/data_connection/document_loaders/integrations/web_loaders/assemblyai_audio_transcription).

At its simplest, you can use the loader to get a transcript back from an audio file like this:

```python
from langchain.document_loaders.assemblyai import AssemblyAIAudioTranscriptLoader

loader =  AssemblyAIAudioTranscriptLoader(file_path="./testfile.mp3")
docs = loader.load()
```

To use it, it needs the `assemblyai` python package installed, and the
environment variable `ASSEMBLYAI_API_KEY` set with your API key. Alternatively, the API key can also be passed as an argument.

Twitter handles to shout out if so kindly 🙇
[@AssemblyAI](https://twitter.com/AssemblyAI) and [@patloeber](https://twitter.com/patloeber)
Initial commit for comprehend moderator
Labels: name: auto:enhancement

This PR implements a custom chain that wraps Amazon Comprehend API calls. The custom chain is aimed to be used with LLM chains to provide moderation capability that let’s you detect and redact PII, Toxic and Intent content in the LLM prompt, or the LLM response. The implementation accepts a configuration object to control what checks will be performed on a LLM prompt and can be used in a variety of setups using the LangChain expression language to not only detect the configured info in chains, but also other constructs such as a retriever. 
The included sample notebook goes over the different configuration options and how to use it with other chains.

###  Usage sample
```python
from langchain_experimental.comprehend_moderation import BaseModerationActions, BaseModerationFilters

moderation_config = { 
        "filters":[ 
                BaseModerationFilters.PII, 
                BaseModerationFilters.TOXICITY,
                BaseModerationFilters.INTENT
        ],
        "pii":{ 
                "action": BaseModerationActions.ALLOW, 
                "threshold":0.5, 
                "labels":["SSN"],
                "mask_character": "X"
        },
        "toxicity":{ 
                "action": BaseModerationActions.STOP, 
                "threshold":0.5
        },
        "intent":{ 
                "action": BaseModerationActions.STOP, 
                "threshold":0.5
        }
}

comp_moderation_with_config = AmazonComprehendModerationChain(
    moderation_config=moderation_config, #specify the configuration
    client=comprehend_client,            #optionally pass the Boto3 Client
    verbose=True
)

template = """Question: {question}

Answer:"""

prompt = PromptTemplate(template=template, input_variables=["question"])

responses = [
    "Final Answer: A credit card number looks like 1289-2321-1123-2387. A fake SSN number looks like 323-22-9980. John Doe's phone number is (999)253-9876.", 
    "Final Answer: This is a really shitty way of constructing a birdhouse. This is fucking insane to think that any birds would actually create their motherfucking nests here."
]
llm = FakeListLLM(responses=responses)

llm_chain = LLMChain(prompt=prompt, llm=llm)

chain = ( 
    prompt 
    | comp_moderation_with_config 
    | {llm_chain.input_keys[0]: lambda x: x['output'] }  
    | llm_chain 
    | { "input": lambda x: x['text'] } 
    | comp_moderation_with_config 
)

response = chain.invoke({"question": "A sample SSN number looks like this 123-456-7890. Can you give me some more samples?"})

print(response['output'])


```
### Output
```
> Entering new AmazonComprehendModerationChain chain...
Running AmazonComprehendModerationChain...
Running pii validation...
Found PII content..stopping..
The prompt contains PII entities and cannot be processed
```
Wfh/whatsapp loader
Labels: name: area: doc loader, name: auto:improvement

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->
docs: fix PromptGuard docs
Labels: name: lgtm, name: auto:documentation, name: area: models

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->

Fix PromptGuard docs. Noticed several trivial issues on the docs when integrating the new class.
cc @baskaryan
Could not parse LLM output: `Action: json_spec_list_keys(data)`
Labels: name: area: agent, name: auto:bug

### System Info

- Version: 0.0.271
- Platform: Macbook Pro M1 macos 13.5
- Python Version: 3.11.4

### Who can help?

@...

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Steps to reproduce the behavior:

1. Copy the existing code from [Langchain Document - Json Agent](https://python.langchain.com/docs/integrations/toolkits/json)
2. Replace the llm model from OpenAI to GPT4All
3. Use the model [ggml-gpt4all-j-v1.3-groovy.bin](https://gpt4all.io/models/ggml-gpt4all-j-v1.3-groovy.bin)
4. Download the [JSON FIle](https://github.com/OAI/OpenAPI-Specification/blob/main/examples/v2.0/json/petstore.json) and provide the path in the script
5. Replace the question with `What are the required parameters in the request body to the /pets endpoint?`.
6. Output: `File venv/lib/python3.11/site-packages/langchain/agents/mrkl/output_parser.py", line 61, in parse
    raise OutputParserException(
langchain.schema.output_parser.OutputParserException: Could not parse LLM output: 'Action: json_spec_list_keys(data)'`

### Expected behavior

The agent should parse through and provide the answer.
Error in the code given in the prompt template
Labels: name: area: doc loader, name: auto:bug, name: auto:documentation

### Issue with current documentation:
Modules -> prompt templates

This set code which is there in documentation is throwing error
`from langchain import PromptTemplate`
`invalid_prompt = PromptTemplate(`
   ` input_variables=["adjective"],`
   ` template="Tell me a {adjective} joke about {content}."`
`)`

Because here in the input variables 'content' word is missing , as it is also one of the input variable to the prompt template.

error -
![image](https://github.com/langchain-ai/langchain/assets/69832170/925674a0-0d87-4525-af33-4ece4f9af924)


**update the code -**

`from langchain import PromptTemplate`
`invalid_prompt = PromptTemplate(`
   ` input_variables=["adjective","content"],`
   ` template="Tell me a {adjective} joke about {content}."`
`)`
output_parser has a bug while the output string has ``` code ```
Labels: name: auto:bug, name: area: models

### System Info

langchain 0.0.235 / python 3.10.11 / 

File:  libs/langchain/langchain/output_parsers/json.py

There is a bug in the function `parse_json_markdown`.
When the input json_string contains  \`\`\` $code  \`\`\` . It mistakenly interprets $code as json format and fails to parse.

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [X] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. make LLM answer with text and code  (in   \`\`\` $code \`\`\`   format)
2. The output_parser will parse the $code  instead of  \`\`\` json \`\`\` 
3. FAIL

### Expected behavior

maybe use re.findall and get the last search result

EX:
```
match = re.findall(r"```(json)?(.*?)```", json_string, re.DOTALL)
```
DOC: Not working: <OpenAPI spec and create + execute valid requests against the API>
Labels: name: area: doc loader, name: auto:documentation

### Issue with current documentation:

Link not working: [OpenAPI](https://python.langchain.com/docs/use_cases/apis/openapi_openai)
where is the link? https://python.langchain.com/docs/modules/chains/how_to/openai_functions

### Idea or request for content:

_No response_
PydanticOutputParser not called by chain
Labels: name: auto:bug, name: area: models

### System Info

langchain 0.0.268
python 3.9
Windows 10

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [X] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
class Actor(BaseModel):
    name: str = Field(description="name of an actor")
    film_names: List[str] = Field(description="list of names of films they starred in")

parser = PydanticOutputParser(pydantic_object=Actor)

prompt = PromptTemplate(
    template="Answer the user query.\n{format_instructions}\n{query}\n",
    input_variables=["query"],
    partial_variables={"format_instructions": parser.get_format_instructions()}
)

chain = LLMChain(llm=ChatModel.get_model(), prompt=prompt, verbose=True)

output = chain.run(query="Generate the filmography for a random actor.", output_parser=parser)
```

Problem: _output_ is a JSON string, and not an Actor object.

### Expected behavior

The method _parse()_ from the OutputParser passed to the chain should be automatically called and return the parsed object, which is not the case. One has to call explicitly `parser.parse(output)` to retrieve an Actor object.

For custom parsers, it seems to work, but not for PydanticOutputParser. I expect the chain's behavior to be consistent across all parsers.
Issue: How to stop extra LLM call in ConversationalRetrievalChain for question rephrasing
Labels: name: auto:question, name: area: models

### Issue you'd like to raise.

I am using ConversationalRetrievalChain for the RAG question-answer bot.
There is one LLM call that I have not configured and it is reducing the quality of responses and increasing the time.

The prompt in the LLM call is:

> Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question, in its original language.
It is taking around 1 second to complete this call, and it is reducing the quality of the response as well.

How do I stop this call?

### Suggestion:

_No response_
Load a jpg using UnstructuredImageLoader from Langchain.
Labels: name: area: doc loader, name: auto:bug

### System Info

win10

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. Load a jpg using UnstructuredImageLoader from Langchain.
2. An error occurred.

### Expected behavior

got the error here：
```
loader:<langchain.document_loaders.image.UnstructuredImageLoader object at 0x000002926EA8EFB0>
Exception in thread Thread-3 (_handle_results):
Traceback (most recent call last):
  File "D:\ProgramData\anaconda3\envs\3.10.11\lib\threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "D:\ProgramData\anaconda3\envs\3.10.11\lib\threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "D:\ProgramData\anaconda3\envs\3.10.11\lib\multiprocessing\pool.py", line 579, in _handle_results
    task = get()
  File "D:\ProgramData\anaconda3\envs\3.10.11\lib\multiprocessing\connection.py", line 251, in recv     
    return _ForkingPickler.loads(buf.getbuffer())
TypeError: TesseractNotFoundError.__init__() takes 1 positional argument but 2 were given

```
docs: fix link url
Labels: name: area: agent, name: auto:documentation

This pull request corrects the URL links in the Async API documentation to align with the updated project layout. The links had not been updated despite the changes in layout.

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->
Issue: I am working with SQLChain and  initialize_agent  it is not answering from the connected database.
Labels: name: area: memory, name: area: agent, name: auto:bug, name: area: models

### Issue you'd like to raise.

def get_connection():
    user=os.environ.get('DB_USERNAME')       
    database= os.environ.get('DB_DATABASE')
    password=os.environ.get('DB_PASSWORD')
    host= os.environ.get('DB_HOST')
    port= os.environ.get('DB_PORT')
    return create_engine(
        url="mysql+pymysql://{0}:{1}@{2}:{3}/{4}".format(
            user, password, host, port, database
        )
)

ef get_whole_conversation(question):
    try:

        llm = ChatOpenAI(temperature=0,openai_api_key=env('OPENAI_API_KEY'),model='gpt-3.5-turbo')
        engine = get_connection()
        input_db = SQLDatabase(engine)
    
        db_chain = SQLDatabaseChain.from_llm(llm, input_db, verbose=True)
        prompt="""

"""
        tools = [Tool(name="Foo-Bar-db",func=db_chain.run,description=prompt)]
        
         agent_kwargs = {"extra_prompt_messages": [MessagesPlaceholder(variable_name="memory")],}
        memory = ConversationBufferWindowMemory(memory_key="memory",k=4,return_messages=True)
        agent = initialize_agent(tools,llm,agent=AgentType.OPENAI_FUNCTIONS,verbose=True,agent_kwargs=agent_kwargs,memory=memory)
agent.run(question)

### Suggestion:

_No response_
When using Langchain to upload a .docx file to the Milvus database, an error occurs.
Labels: name: area: embeddings, name: area: vector store, name: auto:bug

### System Info

Win10

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```python
    embeddings = HuggingFaceEmbeddings( model_name='sentence-transformers/LaBSE')
    vectordb = Milvus(embedding_function=embeddings, connection_args=milvus_settings)
    vectordb.add_documents(documents=spilt_results)
```
I got this Error:

Traceback (most recent call last):
  File "E:\文件\programs\test_jieba\ingest.py", line 44, in <module>
    main()
  File "E:\文件\programs\test_jieba\ingest.py", line 39, in main
    vectordb.add_documents(documents=spilt_results)
  File "E:\文件\programs\test_jieba\.venv\lib\site-packages\langchain\vectorstores\base.py", line 92, in add_documents
    return self.add_texts(texts, metadatas, **kwargs)
  File "E:\文件\programs\test_jieba\.venv\lib\site-packages\langchain\vectorstores\milvus.py", line 454, in add_texts
    insert_list = [insert_dict[x][i:end] for x in self.fields]
  File "E:\文件\programs\test_jieba\.venv\lib\site-packages\langchain\vectorstores\milvus.py", line 454, in <listcomp>
    insert_list = [insert_dict[x][i:end] for x in self.fields]
KeyError: 'file_path'

### Expected behavior

why is this happen?
DOC: Broken notebook link
Labels: name: auto:documentation

### Issue with current documentation:

There is a notebook link in [QA using a Retriever
](https://python.langchain.com/docs/use_cases/question_answering/how_to/vector_db_qa)
that should open a [notebook](https://python.langchain.com/docs/modules/chains/additional/question_answering.html)

But link is boken. Page does not exist.
Can you fix it?

Thanks



### Idea or request for content:

_No response_
adapt ChatGLM using api(chatglm.py)
Labels: name: auto:improvement, name: area: models

using sample:
```
endpoint_url = API URL
ChatGLM_llm = ChatGLM(
    endpoint_url=endpoint_url,
    api_key=Your API Key by ChatGLM
)
print(ChatGLM_llm("hello"))
```

Replace this entire comment with:
  - Description: The call of ChatGLM has been adapted.
  - Issue: The call of ChatGLM has been adapted.
  - Dependencies: Need python package `zhipuai`
  - Tag maintainer: @baskaryan, @hwchase17
  - Twitter handle: None