delayedkarma/langchain-issues huggingface dataset:
VectaraRetriever does not use the parameters passed in by `Vectara.as_retriever()`
Labels: name: auto:bug

`class VectaraRetriever` does not use the parameters passed in by `Vectara.as_retriever()`. The class definition sets up `vectorstore` and `search_kwargs` but then there is no check for values coming in from `Vectara.as_retriever()`.

https://github.com/hwchase17/langchain/blob/e3b7effc8f39333076c2bedd7306de81ce988de6/langchain/vectorstores/vectara.py#L307
DOC: "Edit this page" returns 404
Labels: name: auto:bug, name: auto:documentation

### Issue with current documentation:

The links in the documentation to make contributions to pages point to the wrong file location within the repository. For example, the link on the "Quickstart" page points to the filepath `docs/docs/get_started/quickstart.mdx`, when it should point to `docs/docs_skeleton/docs/get_started/quickstart.mdx`. Presumably this is a result of #6300 and the links not getting updated.

### Idea or request for content:

If someone more familiar with the structure of the docs could tell me where to find the base template where the link templates are specified, and preferably a bit of guidance on how the documentation is structured now to make it easier to figure out the correct links, then I can make the updates.
llma Embeddings error
Labels: name: auto:bug

### System Info

Traceback (most recent call last):
  File "/Users/apple/Desktop/LLM/gpt4all_langchain_chatbots/teacher/lib/python3.10/site-packages/langchain/embeddings/llamacpp.py", line 85, in validate_environment
    from llama_cpp import Llama
ModuleNotFoundError: No module named 'llama_cpp'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/apple/Desktop/LLM/gpt4all_langchain_chatbots/mine.py", line 3, in <module>
    llama = LlamaCppEmbeddings(model_path="./models/ggml-gpt4all-j.bin")
  File "pydantic/main.py", line 339, in pydantic.main.BaseModel.__init__
  File "pydantic/main.py", line 1102, in pydantic.main.validate_model
  File "/Users/apple/Desktop/LLM/gpt4all_langchain_chatbots/teacher/lib/python3.10/site-packages/langchain/embeddings/llamacpp.py", line 89, in validate_environment
    raise ModuleNotFoundError(
ModuleNotFoundError: Could not import llama-cpp-python library. Please install the llama-cpp-python library to use this embedding model: pip install llama-cpp-python




### Who can help?

@sirrrik

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

pip install the latest langChain package from pypy on mac

### Expected behavior

Traceback (most recent call last):
  File "/Users/apple/Desktop/LLM/gpt4all_langchain_chatbots/teacher/lib/python3.10/site-packages/langchain/embeddings/llamacpp.py", line 85, in validate_environment
    from llama_cpp import Llama
ModuleNotFoundError: No module named 'llama_cpp'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/apple/Desktop/LLM/gpt4all_langchain_chatbots/mine.py", line 3, in <module>
    llama = LlamaCppEmbeddings(model_path="./models/ggml-gpt4all-j.bin")
  File "pydantic/main.py", line 339, in pydantic.main.BaseModel.__init__
  File "pydantic/main.py", line 1102, in pydantic.main.validate_model
  File "/Users/apple/Desktop/LLM/gpt4all_langchain_chatbots/teacher/lib/python3.10/site-packages/langchain/embeddings/llamacpp.py", line 89, in validate_environment
    raise ModuleNotFoundError(
ModuleNotFoundError: Could not import llama-cpp-python library. Please install the llama-cpp-python library to use this embedding model: pip install llama-cpp-python
Rm pytz dep
Labels: name: auto:improvement

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @dev2049
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @dev2049
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @vowelparrot
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
feat: enable `UnstructuredEmailLoader` to process attachments
Labels: name: auto:enhancement, name: auto:documentation

### Summary

Updates `UnstructuredEmailLoader` so that it can process attachments in addition to the e-mail content. The loader will process attachments if the `process_attachments` kwarg is passed when the loader is instantiated.

### Testing

```python

file_path = "fake-email-attachment.eml"
loader = UnstructuredEmailLoader(
    file_path, mode="elements", process_attachments=True
)
docs = loader.load()
docs[-1]
```

### Reviewers

-  @rlancemartin 
-  @eyurtsev
- @hwchase17
Error 422 when using BraveSearch
Labels: name: auto:bug

### System Info

langchain v0.0.220
Python 3.11.3

### Who can help?

@hwchase17 

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Using `BraveSearch` with an agent always returns an error 422

[I've created a collab notebook with an example displaying this error.](https://colab.research.google.com/drive/1mUr6KWND4ZYmvFnPywbJevZLzssJMvBR?usp=sharing)

I've tried many times with both `ZERO_SHOT_REACT_DESCRIPTION` and `OPENAI_FUNCTIONS` agents

### Expected behavior

Agents should be able to use this tool
Using JSONLinesLoader in Python
Labels: name: auto:enhancement

### Feature request

It would be great if the [JSONLinesLoader](https://js.langchain.com/docs/modules/indexes/document_loaders/examples/file_loaders/jsonlines) that's available in the JS version of Langchain could be ported to the Python version.

### Motivation

I find working with jsonl files to be frequently easier than json files.

### Your contribution

Not sure---I'm quite new to Python and so don't how to implement this.
HuggingFacePipeline not working
Labels: name: auto:bug

### System Info

langchain==0.0.219


### Who can help?


    @hwchase17
    @agola11



### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

>>> from langchain import HuggingFacePipeline
>>> mod = HuggingFacePipeline(model_id="tiiuae/falcon-7b-instruct", task="text-generation")




Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/guillem.garcia/miniconda3/envs/rene/lib/python3.10/site-packages/langchain/load/serializable.py", line 74, in __init__
    super().__init__(**kwargs)
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for HuggingFacePipeline
task
  extra fields not permitted (type=value_error.extra)


### Expected behavior

I expect it to work. When I instance a Pipeline with falcon using hugginface it works.


Also, Is there a way to deactivate pydantic? It makes it impossible to modify anything or make langchain compatible with our code
Add SpacyEmbeddings class
Labels: name: needs documentation, name: area: embeddings, name: auto:enhancement

- Description: Added a new SpacyEmbeddings class for generating embeddings using the Spacy library.
- Issue: Sentencebert/Bert/Spacy/Doc2vec embedding support #6952
- Dependencies: This change requires the Spacy library and the 'en_core_web_sm' Spacy model.
- Tag maintainer: @dev2049
- Twitter handle: N/A

This change includes a new SpacyEmbeddings class, but does not include a test or an example notebook.
Adding support for URL Parameters in RequestsWrapper
Labels: name: auto:improvement

### Feature request

I'd love to be able to extend the request URL with parameters.

Currently I can only provide request headers. This does not cover my use case.

### Motivation

Some APIs don't authenticate via headers and instead use URL parameters to provide API keys and Tokens.

Example: [Trello API](https://developer.atlassian.com/cloud/trello/rest/api-group-actions/#api-group-actions)

View first code snippet there for the action and you'll see.

curl --request GET \
  --url 'https://api.trello.com/1/actions/{id}?key=APIKey&token=APIToken'



### Your contribution

I would need some handholding to create my first contribution here due to the testing suite but I've got some code I've tested locally.

```python
class Requests(BaseModel):
    """Wrapper around requests to handle auth and async.

    The main purpose of this wrapper is to handle authentication (by saving
    headers) and enable easy async methods on the same base object.
    """

    headers: Optional[Dict[str, str]] = None
    aiosession: Optional[aiohttp.ClientSession] = None
    url_params: Optional[Dict[str, str]] = None

    class Config:
        """Configuration for this pydantic object."""

        extra = Extra.forbid
        arbitrary_types_allowed = True

    def get(self, url: str, **kwargs: Any) -> requests.Response:
        """GET the URL and return the text."""
        if self.url_params:
            url = url + '&'.join([f'{k}={v}' for k, v in self.url_params.items()])
        return requests.get(url, headers=self.headers, **kwargs)

    def post(self, url: str, data: Dict[str, Any], **kwargs: Any) -> requests.Response:
        """POST to the URL and return the text."""
        if self.url_params:
            url = url + '&'.join([f'{k}={v}' for k, v in self.url_params.items()])
        return requests.post(url, json=data, headers=self.headers, **kwargs)

    def patch(self, url: str, data: Dict[str, Any], **kwargs: Any) -> requests.Response:
        """PATCH the URL and return the text."""
        if self.url_params:
            url = url + '&'.join([f'{k}={v}' for k, v in self.url_params.items()])
        return requests.patch(url, json=data, headers=self.headers, **kwargs)

    def put(self, url: str, data: Dict[str, Any], **kwargs: Any) -> requests.Response:
        """PUT the URL and return the text."""
        if self.url_params:
            url = url + '&'.join([f'{k}={v}' for k, v in self.url_params.items()])
        return requests.put(url, json=data, headers=self.headers, **kwargs)

    def delete(self, url: str, **kwargs: Any) -> requests.Response:
        """DELETE the URL and return the text."""
        if self.url_params:
            url = url + '&'.join([f'{k}={v}' for k, v in self.url_params.items()])
        return requests.delete(url, headers=self.headers, **kwargs)
```
Enable CSVLoader to take list of columns as source_column (or metadata)
Labels: name: auto:improvement

### Feature request

Please update the CSVLoader to take list of columns as source_column.

### Motivation

This update will be be helpful, if the documents are embedded and used with vectordbs like Qdrant. As vectordbs like Qdrant offers filtering option, if the metadata is of Document object is used as payload then it will provide a great enhancement.

### Your contribution

No
i try to connect mysql database,but it give me a error about llm value is not a valid dict(type=type_error.dict),how to  solve the problem?
Labels: name: auto:bug

### Issue you'd like to raise.

from langchain.agents.agent_toolkits import SQLDatabaseToolkit
from langchain.sql_database import SQLDatabase
from transformers import AutoModel, AutoTokenizer
model_name = ".\\langchain-models\\THUDM\\chatglm2-6b"
local_model = AutoModel.from_pretrained(model_name, trust_remote_code=True)
local_tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
db = SQLDatabase.from_uri("mysql+pymysql://root:root@localhost/magic")
toolkit = SQLDatabaseToolkit(db=db, llm=local_model, tokenizer=local_tokenizer)
tables = toolkit.list_tables_sql_db()
print(tables)

Traceback (most recent call last):
  File "D:\chat\langchain-ChatGLM\test_sql.py", line 8, in <module>
    toolkit = SQLDatabaseToolkit(db=db, llm=local_model)
  File "pydantic\main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for SQLDatabaseToolkit
llm
  value is not a valid dict (type=type_error.dict)


### Suggestion:

_No response_
CharacterTextSplitter constanly generate chunks longer than given chunk_size 
Labels: name: auto:improvement

### Feature request

CharacterTextSplitter split a size of 1GB code base with warnings exceed the log buffer, like
```
Created a chunk of size 2140, which is longer than the specified 900
Created a chunk of size 1269, which is longer than the specified 900
Created a chunk of size 1955, which is longer than the specified 900
Created a chunk of size 3410, which is longer than the specified 900
Created a chunk of size 1192, which is longer than the specified 900
Created a chunk of size 1392, which is longer than the specified 900
Created a chunk of size 1540, which is longer than the specified 900
- very very long...
```
Walking through the relvent code
```
def _merge_splits(self, splits: Iterable[str], separator: str) -> List[str]:
        # We now want to combine these smaller pieces into medium size
        # chunks to send to the LLM.
        separator_len = self._length_function(separator)

        docs = []
        current_doc: List[str] = []
        total = 0
        for d in splits:
            _len = self._length_function(d)
            if (
                total + _len + (separator_len if len(current_doc) > 0 else 0)
                > self._chunk_size
            ):
                if total > self._chunk_size:
                    logger.warning(
                        f"Created a chunk of size {total}, "
                        f"which is longer than the specified {self._chunk_size}"
                    )
                if len(current_doc) > 0:
                    doc = self._join_docs(current_doc, separator)
                    if doc is not None:
                        docs.append(doc)
                    # Keep on popping if:
                    # - we have a larger chunk than in the chunk overlap
                    # - or if we still have any chunks and the length is long
                    while total > self._chunk_overlap or (
                        total + _len + (separator_len if len(current_doc) > 0 else 0)
                        > self._chunk_size
                        and total > 0
                    ):
                        total -= self._length_function(current_doc[0]) + (
                            separator_len if len(current_doc) > 1 else 0
                        )
                        current_doc = current_doc[1:]
            current_doc.append(d)
            total += _len + (separator_len if len(current_doc) > 1 else 0)
        doc = self._join_docs(current_doc, separator)
        if doc is not None:
            docs.append(doc)
        return docs
```
I know langchain try to keep semantic integrity of code language with language specific separtor.

### Motivation

Can we just pop the last split once the `total > self._chunk_size` in `def _merge_splits(self, splits: Iterable[str], separator: str) -> List[str]:`

### Your contribution

Maybe later I'll propose a PR
ValueError: `run` not supported when there is not exactly one output key. Got ['output', 'intermediate_steps']
Labels: name: auto:bug

### Issue you'd like to raise.

```
agent = initialize_agent(tools, llm, agent="zero-shot-react-description", return_intermediate_steps=True, verbose=True)
agent = initialize_agent(tools, llm, agent="zero-shot-react-description", return_intermediate_steps=True, memory=ConversationBufferMemory(memory_key="chat_history", input_key='input', output_key="output"), verbose=True)

```

https://python.langchain.com/docs/modules/agents/how_to/intermediate_steps

<img width="857" alt="image" src="https://github.com/hwchase17/langchain/assets/42615243/9ea12e4b-1775-4114-a750-0625e1cf8302">




### Suggestion:

I don't know how to solve it!
Conversational Retrieval QA with sources cannot return source
Labels: name: auto:bug

### System Info

ConversationalRetrievalChain with Question Answering with sources
```python
llm = OpenAI(temperature=0)
question_generator = LLMChain(llm=llm, prompt=CONDENSE_QUESTION_PROMPT)
doc_chain = load_qa_with_sources_chain(llm, chain_type="map_reduce")

chain = ConversationalRetrievalChain(
    retriever=vectorstore.as_retriever(),
    question_generator=question_generator,
    combine_docs_chain=doc_chain,
)
```
if i set the chain_type to "refine". it will not return sources.

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

just like the turtial code.

### Expected behavior

i want the 'refine' chain return sources.
Update redis.py
Labels: name: auto:improvement

Provided an option to connect to redis cloud using password,port and db along with existing parameters

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: Provided an option to connect to redis cloud using password,port and db along with existing parameters 
  - Issue: N/A
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: @hwchase17
  - Twitter handle: @ghatiki

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @dev2049
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @dev2049
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @vowelparrot
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Sentencebert/Bert/Spacy/Doc2vec embedding support
Labels: name: auto:improvement

### Feature request

Is there any way to store the Sentencebert/Bert/Spacy/Doc2vec based embeddings in the vector database using langchain
```
pages= "page content"
embeddings = OpenAIEmbeddings()
persist_directory = 'db'
vectordb = Chroma.from_documents(documents=pages, embedding=embeddings, persist_directory=persist_directory)
```
shall use **Sentencebert/Bert/Spacy/Doc2vec embeddings** instead of OpenAIEmbeddings() in the above code?
if possible then what is the syntax for that?

### Motivation

For using native embedding formats likeSentencebert/Bert/Spacy/Doc2vec in langchain

### Your contribution

For using native embedding formats likeSentencebert/Bert/Spacy/Doc2vec in langchain
Issue: Get Confused with ConversationalRetrievalChain + HuggingFacePipeline always gen wizard "Helpful answer: "
Labels: name: auto:question

my code:
```
from langchain.llms import HuggingFacePipeline
  from langchain.chains import ConversationalRetrievalChain
  pipe = pipeline(
      "text-generation", # task type
      model=model,
      tokenizer=tokenizer,
      max_new_tokens=1024,
      device=0, # very trick, gpu rank
  )
  local_llm = HuggingFacePipeline(pipeline=pipe)
  qa = ConversationalRetrievalChain.from_llm(local_llm, retriever=retriever)
  questions = [
      "void Plan::ScheduleWork(map<Edge*, Want>::iterator want_e) {",
  ]
  chat_history = []
  for question in questions:
      result = qa({"question": question, "chat_history": chat_history})
  #     chat_history.append((question, result["answer"]))
      print("**chat_history**", chat_history)
      print(f"=>**Question**: {question} \n")
      print(f"=>**Answer**: {result['answer']} \n\n")
```

  which generate like this:
```
  **chat_history** []
  =>**Question**: void Plan::ScheduleWork(map<Edge*, Want>::iterator want_e) { 
  
  =>**Answer**:  void Plan::ScheduleWork(map<Edge*, Want>::iterator want_e) {
    if (want_e->second == kWantToFinish) {
      // This edge has already been scheduled.  We can get here again if an edge
      // and one of its dependencies share an order-only input, or if a node
      // duplicates an out edge (see https://github.question/: void Plan::ScheduleWork(map<Edge*, Want>::iterator want_e) {
  Helpful Answer: void Plan::ScheduleWork(map<Edge*, Want>::iterator want_e) {
    if (want_e->second == kWantToFinish) {
      // This edge has already been scheduled.  We can get here again if an edge
      // and one of its dependencies share an order-only input, or if a node
      // duplicates an out edge (see https://github.com/ninja-build/ninja/pull/519).
      // Avoid scheduling the work again.
      return;
    }
    assert(want_e->second == kWantToStart);
    want_e->second = kWantToFinish;
```


my `model` is locally fine-tuned starcode with perf merged, `retriever` is DeepLake vectorstore with some args like cos etc.
I do not get the "Helpful Answer" block, WHERE ON EARTH THIS COME FROM???

### Suggestion:

_No response_
LLMSummarizationCheckerFromSource
Labels: name: auto:enhancement

### Feature request

Just curious if there is a feature to check whether the summarized output from a source document hallucinates or not by extending the checker chain.

### Motivation

LLMSummarizationCheckerChain checks facts of summaries using LLM knowledge. The motivation is to develop a feature that checks for facts from an augmented source document. 

### Your contribution

Will submit a PR if this does not exist yet and if it is a nice to have feature