get_openai_callback() does not show the cost taken from new OpenAI's model ("gpt-4-1106-preview" and  "gpt-3.5-turbo-1106")
Labels: name: auto:bug, name: area: models

When we employ the new models introduced during DevDay, gpt-3.5-turbo-1106 and gpt-4-1106-preview, get_openai_callback() does not accurately display their total cost, which includes both the prompt and completion token cost. The subsequent minimal working example illustrates this issue. The costs should not be $0.0, but rather $0.01 * 30 + $0.03 * 598 = $18.24 for gpt-4-1106-preview and $0.0010 * 30 + $0.0020 * 160 = $0.35 for gpt-3.5-turbo-1106.

## MWE

```python
from langchain.schema.messages import HumanMessage, SystemMessage
from langchain.chat_models import ChatOpenAI
from langchain.callbacks import get_openai_callback
# export OPENAI_API_KEY="..."

replies = []
for model in [
    "gpt-4-1106-preview",
    "gpt-3.5-turbo-1106",
]:
    chat = ChatOpenAI(model=model)
    messages = [
        SystemMessage(content="You're a helpful assistant"),
        HumanMessage(content="What is the purpose of model regularization? Explain comprehensibly in English."),
    ]
    with get_openai_callback() as cb:
        reply = chat.invoke(messages).content
    print(cb, reply, sep="\n------\n")
    print("="*16)
    replies.append(reply)

replies
```

## Output of MWE

```
Tokens Used: 628
	Prompt Tokens: 30
	Completion Tokens: 598
Successful Requests: 1
Total Cost (USD): $0.0
------
Model regularization is a technique used in machine learning to prevent a model from overfitting the training data and to improve its generalization to unseen data. Overfitting occurs when a model learns the training data too well, including its noise and outliers, which often results in poor performance on new, unseen data because the model is too tailored to the specifics of the training set.

Here is a comprehensible explanation of the purpose of regularization:

1. **Prevent Overfitting**: Regularization helps to keep the model simple by introducing a penalty for more complex models. By doing so, it discourages the learning of a model that is too complex for the underlying pattern in the data. A simpler model may not perform as well on the training data, but it can perform better on new data because it captures the general trend rather than the specific noise.

2. **Improves Generalization**: The main goal of a machine learning model is to make accurate predictions on new, unseen data. Regularization helps by ensuring that the model's performance during training is more reflective of how it will perform on new data. This is achieved by penalizing the complexity of the model and thereby encouraging the model to be more robust to variations in data.

3. **Controls Model Complexity**: Regularization techniques typically introduce a hyperparameter that controls the trade-off between fitting the training data well and keeping the model complexity low. By adjusting this hyperparameter, one can find a good balance where the model is complex enough to capture the underlying patterns but not so complex that it starts fitting the noise.

4. **Introduces Bias to Reduce Variance**: In statistical terms, regularization introduces a small amount of bias to the model to significantly reduce its variance. This trade-off is beneficial because high variance models are those that overfit the data, while a little bias can make the model more stable and accurate in its predictions on new data.

5. **Handles Multicollinearity**: In cases where the features (input variables) are highly correlated, it can be difficult for the model to estimate the relationship of each feature with the output variable. Regularization techniques can help reduce the impact of multicollinearity by penalizing the coefficients of the correlated features, leading to more stable estimates.

Common regularization techniques include:

- **L1 Regularization (Lasso)**: This adds a penalty equal to the absolute value of the magnitude of coefficients. It can lead to some coefficients being shrunk to zero, effectively performing feature selection.
- **L2 Regularization (Ridge)**: This adds a penalty equal to the square of the magnitude of coefficients. This generally shrinks coefficients evenly but does not set them to zero.
- **Elastic Net**: This is a combination of L1 and L2 regularization and balances the properties of both methods.

In summary, regularization is a crucial step in building machine learning models that are effective at making predictions on new, unseen data by keeping the models simpler and more generalizable.
================
Tokens Used: 194
	Prompt Tokens: 30
	Completion Tokens: 164
Successful Requests: 1
Total Cost (USD): $0.0
------
Model regularization is a technique used in machine learning to prevent overfitting. Overfitting occurs when a model learns the training data too well and performs poorly on new, unseen data. Regularization helps to address this issue by adding a penalty to the model's learning process, discouraging it from becoming too complex and fitting noise in the data.

There are different types of regularization, such as L1 and L2 regularization, which add a penalty based on the magnitude of the model's coefficients. By using regularization, the model is encouraged to focus on the most important features in the data and avoid being overly sensitive to small fluctuations.

In simple terms, model regularization helps to keep the model in check and prevent it from becoming too specialized to the training data, improving its ability to make accurate predictions on new, unseen data.
================
```

It is necessary to include the cost per 1,000 tokens in `MODEL_COST_PER_1K_TOKENS` which is defined in the following lines, in accordance with OpenAI's official pricing page ([here](https://openai.com/pricing#gpt-4-turbo) for gpt-4-turbo; [here](https://openai.com/pricing#gpt-4-turbo:~:text=%C2%A0/%201K%20tokens-,GPT%2D3.5%20Turbo,-GPT%2D3.5%20Turbo) for gpt-3.5-turbo).

https://github.com/langchain-ai/langchain/blob/ff87f4b4f90c1d13ddb79120c6ded6c0af2959b7/libs/langchain/langchain/callbacks/openai_info.py#L7C1-L35