Token Usage Management System
Labels: name: auto:enhancement, name: area: models

### Feature request

The idea is simple. When using various models users have rate limits, both in requests per min/hour and Tokens. When using the raw OpenAI SDK I used the code below. It just uses the "usage" metrics provided in the response (as of now I dont know how to retrieve a 'raw response' when utilizing langchain queries'


### Motivation

This was inspired by the experience of trying to use the GPT api to generate documentaion for a long list of functions of varying length. While trying to run these requests en mass I repeatedly had my process error out because of RATE LIMIT Errors. I dont think the idea of needing to run queries asynchronously against OpenAI API is a common enough use case (though strongely enough not natively supported by the Python SDK) 

### Your contribution

I would be Honored!!!! To contribute this to the project but I honestly would need some help figuring out how to integrate it. This is the base Idea though, Feel free to reachout to collab! codeblackwell@gmail.com or @codeblackwell anywhere. (apologies for the formatting. It just wouldnt work with me).

`
 class TokenManager:
    def __init__(self, max_tokens_per_minute=10000):
        self.max_tokens_per_minute = max_tokens_per_minute
        self.tokens_used_in_last_minute = 0
        self.last_request_time = time.time()

    def tokens_available(self):
        current_time = time.time()
        time_since_last_request = current_time - self.last_request_time

        if time_since_last_request >= 60:
            self.tokens_used_in_last_minute = 0
            self.last_request_time = current_time

        return self.max_tokens_per_minute - self.tokens_used_in_last_minute

    def update_tokens_used(self, tokens_used):
        if not isinstance(tokens_used, int) or tokens_used < 0:
            raise ValueError("Invalid token count")
        self.tokens_used_in_last_minute += tokens_used

    def get_usage_stats(self):
        current_time = time.time()
        time_since_last_request = current_time - self.last_request_time

        if time_since_last_request >= 60:
            self.tokens_used_in_last_minute = 0
            self.last_request_time = current_time

        absolute_usage = self.tokens_used_in_last_minute
        percent_usage = (self.tokens_used_in_last_minute / self.max_tokens_per_minute) * 100

        return absolute_usage, percent_usage`

token_manager = TokenManager()

`def fetch_response(code_snippet, system_prompt, token_manager):
    """
    Fetches a response using the OpenAI API.

    Parameters:
        code_snippet (str): The code snippet to be processed.
        system_prompt (str): The system prompt.

    Returns:
        dict: The response from the API.
    """
    while token_manager.tokens_available() < 2500:  # Assume a minimum of 10 tokens per request
        time.sleep(1)

    try:
        response = openai.ChatCompletion.create(
            model="gpt-4",
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": code_snippet}
            ]
        )
        tokens_used = response['usage']['total_tokens']
        token_manager.update_tokens_used(tokens_used)
        return response
    except Exception as e:
        print(f"An error occurred: {str(e)}")`

`def main(df, system_prompt):
    """
    Processes a DataFrame of code snippets.

    Parameters:
        df (pd.DataFrame): The DataFrame containing code snippets.
        system_prompt (str): The system prompt.

    Returns:
        pd.DataFrame: The DataFrame with the added output column.
    """
    responses = [fetch_response(row['code'], system_prompt) for _, row in df.iterrows()]
    df['output'] = responses
    return df`

# Run the main function
`new_df = asyncio.run(main(filtered_df, system_prompt_1))`