delayedkarma/langchain-issues huggingface dataset:
Infino integration for simplified logs, metrics & search across LLM data & token usage
Labels: name: enhancement, name: lgtm, name: callbacks

<!--
Thank you for contributing to Langchain! Your PR will appear in our release under the title you set. Please make sure it highlights your valuable contribution.

Replace this with a description of the change, the issue it fixes (if applicable), and relevant context. List any dependencies required for this change.

After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost.

Finally, we'd love to show appreciation for your contribution - if you'd like us to shout you out on Twitter, please also include your handle!
-->

### Integration of Infino with LangChain for Enhanced Observability

This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models.

#### Please refer to the following files related to integration:
- `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino.
- Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. 
- [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration.

By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging.

Co-authors: @vinaykakade @savannahar68

### Who can review?
I request @hwchase17, @agola11 and other community members for the review.
No Documentation for constructor parameter client

I just started a new project in langchain and when I try to create a OpenAIEmbeddings object, I'm asked for a client which has a type Any. This is not in documentation anywhere and it's hard to figure out what client is required.

It would be amazing if someone can clarify the use-case for this. I would be happy to raise a PR to document this.

## Error Screenshot
![Error Screenshot](https://github.com/hwchase17/langchain/assets/21296041/f992fe12-c541-4fc0-9722-895626a9ac4e)


## Version
![Versions](https://github.com/hwchase17/langchain/assets/21296041/b69b5de7-84bf-4a64-9d5e-208e95b705b6)
s/21296041/7155ffef-acdf-4a21-80d7-4b60484fab93)

https://github.com/hwchase17/langchain/blob/7ad13cdbdbd45b1348f199419da836bdbcbc02e2/langchain/embeddings/openai.py#L108
Integrate Rockset as Vectorstore
Labels: name: enhancement, name: lgtm

#### Description

This PR adds Rockset as a vectorstore for langchain. [Rockset](https://rockset.com/blog/introducing-vector-search-on-rockset/) is a real time OLAP database which provides a fast and efficient vector search functionality. Further since it is entirely schemaless, it can store metadata in separate columns thereby allowing fast metadata filters during vector similarity search (as opposed to storing the entire metadata in a single JSON column). It currently supports three distance functions: `COSINE_SIMILARITY`, `EUCLIDEAN_DISTANCE`, and `DOT_PRODUCT`. 

This PR adds `rockset` client as an optional dependency. 

We would love a twitter shoutout, our handle is https://twitter.com/RocksetCloud

#### Before submitting

1. Integration test: https://github.com/anubhav94N/langchain/blob/master/tests/integration_tests/vectorstores/test_rocksetdb.py
2. Example notebook: https://github.com/anubhav94N/langchain/blob/master/docs/modules/indexes/vectorstores/examples/rockset_vector_database.ipynb
3. Ran `make format` and `make lint` locally


#### Who can review?

@hwchase17, @dev2049 can you help review please?
Summary is run on pruned messages, not remaining messages

I just checked the token usage for the summay generation after I set the max_token_limit to 100 and wondered why I had a token usage over 1000 tokens. I think the call in the following line should pass the messages that are left in the buffer and not the pruned messages, 

https://github.com/hwchase17/langchain/blob/7ad13cdbdbd45b1348f199419da836bdbcbc02e2/langchain/memory/summary_buffer.py#L72

I think it should be

```
self.moving_summary_buffer = self.predict_new_summary(
    buffer, self.moving_summary_buffer
)
```

or am I missing something here?
The Agent is incompatible with the GPT-3.5-turbo-16k model.

### System Info

LangChain v0.0.198

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

After using the GPT-3.5-turbo-16k model for LLM, 
in Agent return
the expected output for "Action" was
"Action: python_repl_ast". 
However, the actual output received was 
"Action: Use the value_counts() function to count the number of occurrences of each user type in the 'user_type' column of the 'df1' dataframe."

### Expected behavior

"Action: python_repl_ast"
ConversationalRetrievalChain does not return source in result

mentioned in document but not implemented

https://python.langchain.com/en/latest/modules/chains/index_examples/chat_vector_db.html#conversationalretrievalchain-with-question-answering-with-sources

<!--
Thank you for contributing to LangChain! Your PR will appear in our release under the title you set. Please make sure it highlights your valuable contribution.

Replace this with a description of the change, the issue it fixes (if applicable), and relevant context. List any dependencies required for this change.

After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost.

Finally, we'd love to show appreciation for your contribution - if you'd like us to shout you out on Twitter, please also include your handle!
-->

<!-- Remove if not applicable -->

Fixes # (issue)

#### Before submitting

<!-- If you're adding a new integration, please include:

1. a test for the integration - favor unit tests that does not rely on network access.
2. an example notebook showing its use


See contribution guidelines for more information on how to write tests, lint
etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->

#### Who can review?

Tag maintainers/contributors who might be interested:

<!-- For a quicker response, figure out the right person to tag with @

  @hwchase17 - project lead

  Tracing / Callbacks
  - @agola11

  Async
  - @agola11

  DataLoaders
  - @eyurtsev

  Models
  - @hwchase17
  - @agola11

  Agents / Tools / Toolkits
  - @hwchase17

  VectorStores / Retrievers / Memory
  - @dev2049

 -->
Fix output final text for HuggingFaceTextGenInference when streaming

The LLM integration [HuggingFaceTextGenInference](https://github.com/hwchase17/langchain/blob/master/langchain/llms/huggingface_text_gen_inference.py) already has streaming support.

However, when streaming is enabled, it always returns an empty string as the final output text when the LLM is finished. This is because `text` is instantiated with an empty string and never updated.

This PR fixes the collection of the final output text by concatenating new tokens.
ValueError: invalid literal for int() with base 10: '0
'  use  it will throw an exception

### System Info

macos

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [x] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [x] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction



llm = AzureOpenAI(deployment_name=deployment, model_name="text-davinci-003", temperature=0, max_tokens=500)
llm_chain = load_qa_chain(llm, verbose=True,chain_type="map_rerank")

and when i run 
ch = llm_chain.run(input_documents=context, question=question)

it will throw an exception
    zip(typed_results, docs), key=lambda x: -int(x[0][self.rank_key])
ValueError: invalid literal for int() with base 10: '0
'




### Expected behavior

this is my llm function

llm = AzureOpenAI(deployment_name=deployment, model_name="text-davinci-003", temperature=0, max_tokens=500)
llm_chain = load_qa_chain(llm, verbose=True,chain_type="map_rerank")

and when i run 
ch = llm_chain.run(input_documents=context, question=question)

it will throw an exception
    zip(typed_results, docs), key=lambda x: -int(x[0][self.rank_key])
ValueError: invalid literal for int() with base 10: '0
'


and when i change the chain_type ="stuff" it worked. but currently, i want to use map_rerank. any one can help me ?
OPENAI_FUNCTIONS agent doesn't accept python_repl or google-search or wolfram-alpha tool

### System Info

Langchain 0.0.200
Python 3.11

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [x] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Here is my [gist](https://gist.github.com/alonsosilvaallende/17968208a14994e8285d91abdd79efab) to reproduce the error and my current workaround.
The description of my gist is basically:
1. I followed the [example notebook](https://github.com/hwchase17/langchain/blob/master/docs/modules/agents/agents/examples/openai_functions_agent.ipynb) for OPENAI_FUNCTIONS Agent
2. In the tools, add `python_repl` as follows: `load_tools(['python_repl'])[0]` so that the Tools are modified as follows:
```python
tools = [
    Tool(
        name = "Search",
        func=search.run,
        description="useful for when you need to answer questions about current events. You should ask targeted questions"
    ),
    Tool(
        name="Calculator",
        func=llm_math_chain.run,
        description="useful for when you need to answer questions about math"
    ),
    Tool(
        name="FooBar-DB",
        func=db_chain.run,
        description="useful for when you need to answer questions about FooBar. Input should be in the form of a question containing full context"
    ),
    load_tools(['python_repl'])[0]
]
```

### Expected behavior

I expect the agent to run normally, however it gives me the error:
'Python REPL' does not match '^[a-zA-Z0-9_-]{1,64}$' - 'functions.4.name'
Here is my [gist](https://gist.github.com/alonsosilvaallende/17968208a14994e8285d91abdd79efab) to see the error and my current workaround.
If I understand correctly, the agent doesn't like that the name has a space. Indeed, if I redefine `python_repl` as:
```python
Tool(name='Python',
        func=python_repl.run,
        description='A Python shell. Use this to execute python commands. Input should be a valid python command. If you want to see the output of a value, you should print it out with `print(...)`.'
    ),
```
the problem disappears.  It's a workaround but I expect that either this agent not to have problems with names with spaces or to rename Python REPL as Python_REPL and similarly the other tools as Wolfram_Alpha, etc.
ClickHouse ERROR: Distance function argument of Annoy index must be of type String.

### System Info

langchain: 0.0.200
platform: macOS
python:  3.10.11
clickhouse: version 23.5.2.7 (official build)


### Who can help?

@hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

<img width="1351" alt="image" src="https://github.com/hwchase17/langchain/assets/12210038/187c268c-7eca-40f1-80a6-596ba3885be9">


### Expected behavior

success write data to clickhouse
perf(openai): changed base model to gpt-3.5-turbo 

I've changed the base model in OpenAI Class to gpt-3.5-turbo. It was already changed in ChatOpenAI class.

Let me know if there was any reason why you kept text-davinci-003 which is 10 times more costly and not that smart in OpenAI Class and not in ChatOpenAI :D 


@hwchase17
@agola11
Azure OpenAPI sometimes returns message without text, but also missing "content" key. Langchain should handle it gracefuly.

### System Info
langchain: 0.0.188 (but same would happen in the latest master too)
python 3.10
Linux


Apparently, when model generates no text (it will depend on a specific set of prompts, messages, NO stopwords is used), Azure OpenAI API responds with something like this:
```

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "role": "assistant"
      }
    }
  ],

```
As you can see, "message" doesn't have "content" key at all. And LangChain expects that key to always be there and at one point there will be KeyError that "content" is not available. Not the best implementation of API from AOAI side, but we should handle it and raise some dedicated exception. The good place for it seems to be here: https://github.com/hwchase17/langchain/blob/master/langchain/chat_models/openai.py#L368 , (def _create_chat_result)

I've "fixed" it by doing this:

```
    def _create_chat_result(self, response: Mapping[str, Any]) -> ChatResult:
        generations = []
        for res in response["choices"]:
            if "content" not in res["message"]: #<--- checking it it's missing
                raise EmptyResponseFromModel()# <---- raising custom exception that I can intercept in the main code and react appropriately
```


### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Can't share exact prompts and messages

### Expected behavior

We should handle this broken response from the AOAI gracefully.
On API base URL change, underlying connection doesn't always change.

### System Info

Version: 0.0.188 (but I don't see any change in the latest master that would fix that)
Python: 3.10
Linux
Problem description:
In one of our environments we have multiple Azure OpenAPI base URLs and have to change them on a fly. We do it by changing LLM within the Chain, basically recreating LLM with the new Base URL, Deployment name and so on, something like this:
   
 ```
llm = AzureChatOpenAI(
        openai_api_base=api_base,
        openai_api_version=api_version,
        deployment_name=deployment_name,
        openai_api_key=api_key,
        openai_api_type="azure",)
```

Unfortunately it doesn't always have an effect, as, apparently most of those parameters are not passed correctly to the OpenAI API, here: https://github.com/hwchase17/langchain/blob/master/langchain/chat_models/openai.py#L294 (def completion_with_retry and def acompletion_with_retry) kwargs don't contain things like base endpoint and so on. It seems to be from some legacy openai implementation when it had to be declared as environmental variable. And basically leads to a situation (time to time, depending on how fast the change is, if concurrently different users use different models, etc.) that base URL will remain the same, while deployment name changes.


To fix this,  I simply added:
```
...
     retry_decorator = _create_retry_decorator(llm)
    try:
        kwargs['api_base'] = llm.openai_api_base
        kwargs['api_key'] = llm.openai_api_key
        kwargs['api_type'] = llm.openai_api_type
        kwargs['api_version'] = llm.openai_api_version
        kwargs['organization'] = llm.openai_organization if llm.openai_organization else None
    except:
        pass
    @retry_decorator
...
```

As it's inside chat_models and openai supports those arguments for chat models, haven't had any problems with it.

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Steps to reproduce:
1. Have 2 different API Base URLs with different deployment names
2. Create many CoversationChain with AzureChatOpenAI (one per "user" from step#4)
3. Have multiple threads/asyncio (in my specific case, it's behind FastAPI)
4. emulate multiple, parallel calls of CoversationChain.run and then start randomly changing for some user AzureChatOpenAI model pointing to different Base URL and deployment name
5. You will see that some calls will fail, cause deployment name would change but base URL won't (easy to observe by setting logging level to DEBUG)

### Expected behavior

Change of base URL and deployment name should be propagated to the "caller" deterministically and reliably.
AzureChatOpenAI.get_num_tokens_from_messages  does not work

### System Info

langchain==0.0.198 and current HEAD

AzureChat inherits from OpenAIChat
![image](https://github.com/hwchase17/langchain/assets/877883/11b94aef-6dd0-4f75-882a-9558b6550c1b)

Which throws on Azure's model name
![image](https://github.com/hwchase17/langchain/assets/877883/fa995f5c-ba8b-4a3b-a877-9981484893dd)

Azure's model name is gpt-35-turbo, not 3.5
![image](https://github.com/hwchase17/langchain/assets/877883/72d716e6-3402-47a3-a3cc-262b3720d8ed)




### Who can help?

@hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. deploy private GPT3.5 on Azure
2. Initialize an AzureChatOpenAI object
3. call get_num_tokens_from_messages
4. observe the exception

### Expected behavior

no exception
Make so explicit class constructor is called in ElasticVectorSearch from_texts
Labels: name: lgtm

Fixes #6198 

ElasticKnnSearch.from_texts is actually ElasticVectorSearch.from_texts and throws because it calls ElasticKnnSearch constructor with the wrong arguments. 

Now ElasticKnnSearch has its own from_texts, which constructs a proper ElasticKnnSearch.

#### Who can review?

@CodeDevNinja @dev2049
ValueError: Too many arguments to single-input tool Calculator. Args: ['age ^ 0.43', {'age': 26}]

### System Info

0.200

### Who can help?

_No response_

### Information

- [x] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [x] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

run *docs/modules/agents/agents/examples/openai_functions_agent.ipynb*
at line `mrkl.run("Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?")`

> 
> > Entering new  chain...
> 
> Invoking: `Search` with `{'query': 'Leo DiCaprio girlfriend'}`
> 
> 
> Leonardo DiCaprio and Gigi Hadid were recently spotted at a pre-Oscars party, sparking interest once again in their rumored romance. The Revenant actor and the model first made headlines when they were spotted together at a New York Fashion Week afterparty in September 2022.
> Invoking: `Calculator` with `{'expression': 'age ^ 0.43', 'variables': {'age': 26}}`
> 

then throw err: `ValueError: Too many arguments to single-input tool Calculator. Args: ['age ^ 0.43', {'age': 26}]`



### Expected behavior

should output value
OpenAI: Function calling and other API updates

### Feature request

https://openai.com/blog/function-calling-and-other-api-updates

I think we should update the ChatOpenAi models behavior with tools so that it used the native API.

### Motivation

Their model is likely trained to handle functions this way, and will have a lot better support. It also supposedly guarantees json matching the json schema, which can be hard to achieve otherwise.

### Your contribution

I may be able to help. I’m working on OpenAI at work, but I’m just learning langchains API.
count tokens for new OpenAI model versions

Trying to call `ChatOpenAI.get_num_tokens_from_messages` returns the following error for the newly announced models `gpt-3.5-turbo-0613` and `gpt-4-0613`:

```
NotImplementedError: get_num_tokens_from_messages() is not presently implemented for model gpt-3.5-turbo-0613.See https://github.com/openai/openai-python/blob/main/chatml.md for information on how messages are converted to tokens.
```

This adds support for counting tokens for those models, by counting tokens the same way they're counted for the previous versions of `gpt-3.5-turbo` and `gpt-4`.

#### reviewers

  - @hwchase17
  - @agola11