delayedkarma/langchain-issues huggingface dataset:
Use evaluator config in run_on_dataset
Labels: name: auto:improvement, name: auto:refactor

Use `EvalConfig` n the `run_on_dataset` function, plus:
- Add proactive validation for compatibility with the evaluators ( e.g., check can be converted to prompt or messages for LLM or check example input keys against the chain)
- Improve error messaging for the dataset <-> model you're testing
- Integration tests for the combinations of dataset formats and llm, chat models, and chains

TODO: - delete old evaluator loading #7563

Deltas off [#7508](https://github.com/hwchase17/langchain/pull/7508) (split the criteria evalutaor into the reference free and labeled classes) which builds off [#7388](https://github.com/hwchase17/langchain/pull/7388) which migrates from langchainplus_sdk to langsmith package

<details> <summary>Dataset/ model setup</summary><pre><code>
#  """Evaluation chain for a single QA evaluator."""
from uuid import uuid4
import pandas as pd
from langchain.client.runner_utils import run_on_dataset

from langsmith import Client

client = Client()

dataset_name = f"Testing - {str(uuid4())[-8:]}"
df = pd.DataFrame(
    {
        "some_input": [
            "What's the capital of California?",
            "What's the capital of Nevada?",
            "What's the capital of Oregon?",
            "What's the capital of Washington?",
        ],
        "some_output": ["Sacramento", "Carson City", "Salem", "Olympia"],
    }
)
ds = client.upload_dataframe(
    df, dataset_name, input_keys=["some_input"], output_keys=["some_output"]
)


from langchain.chat_models import ChatOpenAI
from langchain.chains import LLMChain


def chain_constructor() -> None:
    """Evaluate a chain on a dataset."""
    llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)
    chain = LLMChain.from_string(llm, "What's the capital of {input}?")
    return chain
</code></pre></details>



**Relevant snippet**
```
from langchain.evaluation.run_evaluators.config import (
    RunEvalConfig,
)

evaluation_config = RunEvalConfig(
    evaluator_configs=[
        RunEvalConfig.Criteria(criteria="helpfulness"),
        RunEvalConfig.Criteria(
            criteria={"my-criterion": "Is the answer fewer than 10 words?"}
        ),
        "qa", # Or could do RunEvalConfig.ContextQA(), etc.
        "context_qa",
       "embedding_distance",       
    ]
)


run_on_dataset(
    dataset_name,
    llm_or_chain_factory=chain_constructor,
    run_evaluator_config=evaluation_config,
)

``
Fix AttributeError _client_settings
Labels: name: auto:bug

Closes #7482 

@rlancemartin  @nb-programmer 
<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Add unit tests for StructuredChatOutputParser and handle variant action format
Labels: name: auto:improvement

  - Description: I added unit tests for StructuredChatOutput Parser (there were none previously). I also noticed in testing with chat-bison that it sometimes produced action output in a slightly different format, so I added an additional regular expression to correctly handle that output format.
  - Issue: N/A
  - Dependencies: None
  - Tag maintainer: @hinthornw
  - Twitter handle: N/A
Get Llama from LlamaCpp as LlamaCppEmbeddings base, without model reloading
Labels: name: auto:improvement

When we use LlamaCpp and we need embeddings, earlier we load model in LlamaCpp in memory, then we create LlamaCppEmbeddings and load the same model again. After this modification we can create LlamaCppEmbeddings with existing Llama model and use as always. Reduce needed memory, reduce time for loading.
1.    get_llama() to get Llama  from LlamaCpp 
2.   Create LlamaCppEmbeddings class with llama parametr: embeddings =  LlamaCppEmbeddings(llama = llm.get_llama())
MRKL Agent OutputParser Exception.
Labels: name: auto:bug, name: auto:question

### Issue you'd like to raise.

I keep getting OutputParserException: Could not parse LLM output.

I have tried setting handle_parsing_errors=True as well as handle_parsing_errors="Check your output and make sure it conforms!", and yet most of the times I find myself getting the OutputParserException.

Here is an example of the error:

```
> Entering new  chain...
Thought: The question is asking for a detailed explanation of a use example of chain of thought prompting. I should first check if there is a clear answer in the database.

Action: Lookup from database
Action Input: "use example of chain of thought prompting"

Observation: Sure! Here's an example of chain-of-thought prompting:

Let's say we have a language model that needs to solve a math word problem. The problem is: "John has 5 apples. He gives 2 apples to Mary. How many apples does John have now?"

With chain-of-thought prompting, we provide the model with a prompt that consists of triples: input, chain of thought, output. In this case, the prompt could be:

Input: "John has 5 apples. He gives 2 apples to Mary."
Chain of Thought: "To solve this problem, we need to subtract the number of apples John gave to Mary from the total number of apples John had."
Output: "John now has 3 apples."

By providing the model with this chain of thought, we guide it through the reasoning process step-by-step. The model can then generate the correct answer by following the provided chain of thought.

This approach of chain-of-thought prompting helps the language model to decompose multi-step problems into intermediate steps, allowing for better reasoning and problem-solving abilities.
Thought:
---------------------------------------------------------------------------
OutputParserException                     Traceback (most recent call last)
[<ipython-input-76-951eb95eb01c>](https://localhost:8080/#) in <cell line: 2>()
      1 query = "Can you explain a use example of chain of thought prompting in detail?"
----> 2 res = agent_chain(query)

6 frames
[/usr/local/lib/python3.10/dist-packages/langchain/agents/mrkl/output_parser.py](https://localhost:8080/#) in parse(self, text)
     40 
     41         if not re.search(r"Action\s*\d*\s*:[\s]*(.*?)", text, re.DOTALL):
---> 42             raise OutputParserException(
     43                 f"Could not parse LLM output: `{text}`",
     44                 observation="Invalid Format: Missing 'Action:' after 'Thought:'",

OutputParserException: Could not parse LLM output: `I have found a clear answer in the database that explains a use example of chain of thought prompting.`
```

Is there any other way in which I can mitigate this problem to get consistent outputs? 

### Suggestion:

Is there a way to use Retry Parser for this agent, if yes how?
retrievalqa callback handler
Labels: name: auto:improvement

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
TransportQueryError when using GraphQL tool
Labels: name: auto:bug

### System Info

When running the following code:

```
from langchain import OpenAI
from langchain.agents import load_tools, initialize_agent, AgentType
from langchain.utilities import GraphQLAPIWrapper
from langchain.memory import ConversationBufferMemory


llm = OpenAI(temperature=0, openai_api_key=openai_api_key)
token = "..."
tools = load_tools(
    ["graphql"],
    custom_headers={"Authorization": token, "Content-Type": "application/json"},
    graphql_endpoint="...",
    llm=llm
)

memory = ConversationBufferMemory(memory_key="chat_history")
agent = initialize_agent(
    tools, llm, agent=AgentType.CONVERSATIONAL_REACT_DESCRIPTION, verbose=True, memory=memory
)

graphql_fields = """query getCompanies {get_companies}"""

suffix = "Call the API with schema "

agent.run(f"{suffix} {graphql_fields}")
```

Im getting the error:
TransportQueryError: Error while fetching schema: {'errorType': 'UnauthorizedException', 'message': 'You are not authorized to make this call.'}
If you don't need the schema, you can try with: "fetch_schema_from_transport=False"

It doesn't matter what value is provided under custom_headers, or if it is passed as a parameter at all. The error is always the same. Playground code from https://python.langchain.com/docs/modules/agents/tools/integrations/graphql worked as intended.

Any idea of what the problem is?

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
from langchain import OpenAI
from langchain.agents import load_tools, initialize_agent, AgentType
from langchain.utilities import GraphQLAPIWrapper
from langchain.memory import ConversationBufferMemory


llm = OpenAI(temperature=0, openai_api_key=openai_api_key)
token = "..."
tools = load_tools(
    ["graphql"],
    custom_headers={"Authorization": token, "Content-Type": "application/json"},
    graphql_endpoint="...",
    llm=llm
)

memory = ConversationBufferMemory(memory_key="chat_history")
agent = initialize_agent(
    tools, llm, agent=AgentType.CONVERSATIONAL_REACT_DESCRIPTION, verbose=True, memory=memory
)

graphql_fields = """query getCompanies {get_companies}"""

suffix = "Call the API with schema "

agent.run(f"{suffix} {graphql_fields}")

TransportQueryError: Error while fetching schema: {'errorType': 'UnauthorizedException', 'message': 'You are not authorized to make this call.'}
If you don't need the schema, you can try with: "fetch_schema_from_transport=False"
```

### Expected behavior

An allowed API call that doesn't cause authentication issues
RecursiveCharacterTextSplitter strange behavior after v0.0.226
Labels: name: auto:bug

### System Info

After v0.0.226, the RecursiveCharacterTextSplitter seems to no longer separate properly at the end of sentences and now cuts many sentences mid-word.

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```python
splitter = RecursiveCharacterTextSplitter(
    chunk_size=450,
    chunk_overlap=20,
    length_function=len,
    #separators=["\n\n", "\n", ".", " ", ""], # tried with and without this
)
```

### Expected behavior

Would like to split at newlines or period marks.
AttributeError: 'Chroma' object has no attribute '_client_settings'
Labels: name: auto:bug

### System Info



on Python 3.10.10

with requirements.txt 
```
pandas==2.0.1
beautifulsoup4==4.12.2
langchain==0.0.229
chromadb==0.3.26
tiktoken==0.4.0
gradio==3.36.1
Flask==2.3.2
torch==2.0.1
sentence-transformers==2.2.2
``` 


### Who can help?

@hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I'm getting `AttributeError: 'Chroma' object has no attribute '_client_settings'` when running

```python
from langchain.vectorstores import Chroma
import chromadb
from chromadb.config import Settings
from langchain.embeddings import HuggingFaceEmbeddings
from constants.model_constants import HF_EMBEDDING_MODEL

chroma_client = chromadb.Client(Settings(chroma_api_impl="rest", chroma_server_host="xxxxx", chroma_server_http_port="443", chroma_server_ssl_enabled=True))


embedder = HuggingFaceEmbeddings(
                model_name=HF_EMBEDDING_MODEL,
                model_kwargs={"device": "cpu"},
                encode_kwargs={'normalize_embeddings': False}
            )

chroma_vector_store = Chroma(
            collection_name="test",
            embedding_function=embedder,
            client=chroma_client)
``` 


the traceback is 

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/rubelagu/.pyenv/versions/3.10.10/envs/xxxxTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/rubelagu/.pyenv/versions/3.10.10/envs/oraklet_chatbot/lib/python3.10/site-packages/langchain/vectorstores/chroma.py", line 94, in __init__
    self._client_settings.persist_directory or persist_directory
AttributeError: 'Chroma' object has no attribute '_client_settings'/lib/python3.10/site-packages/langchain/vectorstores/chroma.py", line 94, in __init__
    self._client_settings.persist_directory or persist_directory
AttributeError: 'Chroma' object has no attribute '_client_settings'
```
### Expected behavior

It should not raise an exception,


It seems to me that 
https://github.com/hwchase17/langchain/blob/5eec74d9a5435c671382e69412072a8725b2ec60/langchain/vectorstores/chroma.py#L93-L95

was introduced by commit https://github.com/hwchase17/langchain/commit/a2830e3056e4e616160b150bf5ea212a97df2dc4
from @nb-programmer   and @rlancemartin 

that commit assumes that self._client_settings exists always when in reality that won't be created if a client is passed
0.0.229 breaks existing code that works with 0.0.228 for ConverstaionalRetrievalChain
Labels: name: auto:bug

### System Info

Works in 0.0.228 but breaks in 0.0.229

### Who can help?

@hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

The latest version of Langchain (0.229) seems to break working code in 0.0.228.

e.g this code works in 0.228

```python
def qna(question: str, vector_name: str, chat_history=[]):

    logging.debug("Calling qna")

    llm, embeddings, llm_chat = pick_llm(vector_name)

    vectorstore = pick_vectorstore(vector_name, embeddings=embeddings)

    retriever = vectorstore.as_retriever(search_kwargs=dict(k=3))

    prompt = pick_prompt(vector_name)

    logging.basicConfig(level=logging.DEBUG)
    logging.debug(f"Chat history: {chat_history}")
    qa = ConversationalRetrievalChain.from_llm(ChatOpenAI(model="gpt-4", temperature=0.2, max_tokens=5000),
                                               retriever=retriever, 
                                               return_source_documents=True,
                                               verbose=True,
                                               output_key='answer',
                                               combine_docs_chain_kwargs={'prompt': prompt},
                                               condense_question_llm=OpenAI(model="gpt-3.5-turbo", temperature=0))

    try:
        result = qa({"question": question, "chat_history": chat_history})
    except Exception as err:
        error_message = traceback.format_exc()
        result = {"answer": f"An error occurred while asking: {question}: {str(err)} - {error_message}"}
    
    logging.basicConfig(level=logging.INFO)
    return result
```

But in 0.229 it errors like this:

```
INFO:openai:error_code=None error_message='This is a chat model and not supported in the v1/completions endpoint. Did you mean to use v1/chat/completions?' error_param=model error_type=invalid_request_error message='OpenAI API error received' stream_error=False
```

### Expected behavior

Same output
langchain.schema.OutputParserException: Could not parse LLM output: `Thought: Do I need to use a tool? No
Labels: name: auto:bug

### System Info

langchain==0.0.219
python 3.9

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
import os
from llama_index import LLMPredictor,ServiceContext,LangchainEmbedding
from langchain.embeddings.huggingface import HuggingFaceEmbeddings
from langchain.agents import Tool
from langchain.chains.conversation.memory import ConversationBufferMemory
from langchain.chat_models import AzureChatOpenAI

BASE_URL = "url"
API_KEY = "key"
DEPLOYMENT_NAME = "deployment_name"

model = AzureChatOpenAI(
    openai_api_base=BASE_URL,
    openai_api_version="version",
    deployment_name=DEPLOYMENT_NAME,
    openai_api_key=API_KEY,
    openai_api_type="azure",
)

from langchain.agents import initialize_agent

from llama_index import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("/Data").load_data()
llm_predictor = LLMPredictor(llm=model)
embed_model = LangchainEmbedding(HuggingFaceEmbeddings(model_name='huggingface model'))


service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor,embed_model=embed_model)

index = VectorStoreIndex.from_documents(documents=documents,service_context=service_context)

tools = [
    Tool(
        name="LlamaIndex",
        func=lambda q: str(index.as_query_engine().query(q)),
        description="useful for when you want to answer questions about the author. The input to this tool should be a complete english sentence.",
        return_direct=True,
    ),
]

memory = ConversationBufferMemory(memory_key="chat_history")
agent_executor = initialize_agent(
    tools, model, agent="conversational-react-description", memory=memory
)

while True:
    query = input("Enter query\n")
    print(agent_executor.run(input=query))

```
Trying the above code, but when i ask queries, it shows the error - '**langchain.schema.OutputParserException: Could not parse LLM output: `Thought: Do I need to use a tool? No**'

### Expected behavior

The error should not occur
Add LLM for Alibaba's Damo Academy's Tongyi Qwen API
Labels: name: auto:enhancement, name: auto:documentation

- Add langchain.llms.Tonyi for text completion, in examples into the Tonyi Text API,
- Add system tests.

Note async completion for the Text API is not yet supported and will be included in a future PR.

Dependencies: dashscope. It will be installed manually cause it is not need by  everyone.

Happy for feedback on any aspect of this PR @hwchase17 @baskaryan.
gpt4all+langchain_chain(RetrievalQAWithSourcesChain)
Labels: name: auto:question

### Issue you'd like to raise.

`def generate_answer(vector_store, question):
    chain = load_chain("qna/configs/chains/qa_with_sources_gpt4all.json")   
    # print(chain)
    # qa = VectorDBQAWithSourcesChain(combine_document_chain=chain, vectorstore=vector_store)
    qa = RetrievalQAWithSourcesChain(combine_document_chain=chain, retriever= vector_store.as_retriever() )
    result = send_prompt(qa, question)
    return result`

Im experimenting chain module , so i executed above code using openai model when coming to gpt4all- groovy model. it is throwing error
![Screenshot from 2023-07-10 16-37-40](https://github.com/hwchase17/langchain/assets/69832170/f2a5a8ef-382d-4922-a78a-541a66f494d1)


### Suggestion:

Can you suggest me whether Im doing right or wrong. Does gpt4all model supported or not?