Repository: microsoft/autogen
Pull request: FAQ, working with LLM endpoints and explaining why config-list is a list
Making the docs clearer about how LLM API endpoint config works.

## Related issue number

Closes #749 and #97

## Checks

- [x] I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
- [N/A] I've added tests (if relevant) corresponding to the changes introduced in this PR.
- [x] I've made sure all auto checks have passed.


Problem statement:
Why is config list a list?
Hi, it seems that the basics of the config list structure is missing or not obvious from the documentation.

Why is config list a list of models in the first place? What happens with this list of models when you pass it to an agent? Does it just use the first one or choose one with some criteria? The structure and use of the config list seems like something that should be at the very top of the documentation.

Hints from issue discussion:
Yes, this is critical information to surface. It's also complicated. The fullest explanation is in [oai_openai_utils.ipynb](https://github.com/microsoft/autogen/blob/main/notebook/oai_openai_utils.ipynb). The Quickstart on the main readme refers to OAI_CONFIG_LIST_sample and the ./notebook folder. Suppose that we expanded item 2 in the Quickstart to say:

**Copy OAI_CONFIG_LIST_sample to ./notebook folder, name to OAI_CONFIG_LIST, and set the correct configuration. (See [oai_openai_utils.ipynb](https://github.com/microsoft/autogen/blob/main/notebook/oai_openai_utils.ipynb) for details on model configurations.)**

Would that fill in the documentation gap, or would you suggest doing more?
Hi, thanks for your quick reply. A couple comments:

The notebook example goes into a lot of detail about how to generate a config list, but lacks sufficient detail about how the config list is actually used by autogen. It mentions, "config_list aids in dynamically selecting the appropriate model configuration, managing API keys, endpoints, and versions for efficient operation of the intelligent assistant", but this is too vague in my opinion. It doesn't give me any actionable information on how I should best define a config list for a specific use case. Additionally, If I was concerned about cost or latency I'd still be very unsure how to estimate how often each model may be used.

Also the notebook is obscured a bit by its title alone. I'd really recommend a notebook that's titled, `config_list_guide.ipynb` or something like that. Especially considering users will likely want to know how to set configurations across not only OpenAI models, but local models, and other LLM APIs (Anthropic, Bedrock, etc.).

Thanks for the recommendations! 
@AaronWard FYI about the notebook title suggestion.
@austinmw the missing info you are looking for is at https://microsoft.github.io/autogen/docs/Use-Cases/enhanced_inference#error-handling
It'll be good to fill the gap by making a link to that page in the notebook or a more prominent place if desired.
I've seen more than one people asking this question. We should definitely improve the documentation. Any volunteer?
@sonichi I asked a similar question on Discord yesterday (which you and @rickyloynd-microsoft have very kindly responded to). 

I would love to help, but as a newbie , I'd probably need some guidance. 😬
Thanks. The easiest thing to do is to add the link at the place where you thought you were supposed to find that info. An outsider could actually be good at this task because it is a blind spot of the insiders.
> @AaronWard FYI about the notebook title suggestion.

Just seeing this issue now. Good call out.

The reason for the name is because of it being in `autogen/oai/openai_utils.py`. Considering config lists are provider agnostic the functions not directly related to openai could be broken out into a separate file, and keep `retrieve_assistants_by_name` in `openai_utils.py`? 

I can PR this if it sounds good.
There is no need to break out now because the `openai` refers to the `openai-python` package and not necessarily openai endpoints. The configs indeed follow the openai convention.
Hi @sonichi, added some lines to the documentation. Please have a look.

Commit message: FAQ and notebook extended

Patch to review:
diff --git a/notebook/oai_openai_utils.ipynb b/notebook/oai_openai_utils.ipynb
index 2040f570c5ed..02d54909430b 100644
--- a/notebook/oai_openai_utils.ipynb
+++ b/notebook/oai_openai_utils.ipynb
@@ -24,7 +24,9 @@
     "- `config_list_openai_aoai`: Constructs a list of configurations using both Azure OpenAI and OpenAI endpoints, sourcing API keys from environment variables or local files.\n",
     "- `config_list_from_json`: Loads configurations from a JSON structure, either from an environment variable or a local JSON file, with the flexibility of filtering configurations based on given criteria.\n",
     "- `config_list_from_models`: Creates configurations based on a provided list of models, useful when targeting specific models without manually specifying each configuration.\n",
-    "- `config_list_from_dotenv`: Constructs a configuration list from a `.env` file, offering a consolidated way to manage multiple API configurations and keys from a single file."
+    "- `config_list_from_dotenv`: Constructs a configuration list from a `.env` file, offering a consolidated way to manage multiple API configurations and keys from a single file.\n",
+    "\n",
+    "If mutiple models are provided, the Autogen client (`OpenAIWrapper`) and agents don't choose the \"best model\" on any criteria - inference is done through the very first model and the next one is used only if the current model fails (e.g. API throtling by the provider)."
    ]
   },
   {
diff --git a/website/docs/FAQ.md b/website/docs/FAQ.md
index 96110e67c898..e560092f54a2 100644
--- a/website/docs/FAQ.md
+++ b/website/docs/FAQ.md
@@ -3,6 +3,7 @@
 - [Set your API endpoints](#set-your-api-endpoints)
   - [Use the constructed configuration list in agents](#use-the-constructed-configuration-list-in-agents)
   - [Unexpected keyword argument 'base_url'](#unexpected-keyword-argument-base_url)
+  - [How does an agent decide which model to pick out of the list?](#how-does-an-agent-decide-which-model-to-pick-out-of-the-list)
   - [Can I use non-OpenAI models?](#can-i-use-non-openai-models)
 - [Handle Rate Limit Error and Timeout Error](#handle-rate-limit-error-and-timeout-error)
 - [How to continue a finished conversation](#how-to-continue-a-finished-conversation)
@@ -20,7 +21,35 @@
 
 ## Set your API endpoints
 
-There are multiple ways to construct configurations for LLM inference in the `oai` utilities:
+Autogen relies on 3rd party API endpoints for LLM inference. It works great with OpenAI and Azure OpenAI models, and can work with any model (self-hosted or local) that is accessible through an inference server compatible with OpenAI Chat Completions API.
+
+An agent requires a list of configuration dictionaries for setting up model endpoints. Each configuration dictionary can contain the following keys:
+- `model` (str): Required. The identifier of the model to be used, such as 'gpt-4', 'gpt-3.5-turbo', or 'llama-7B'.
+- `api_key` (str): Optional. The API key required for authenticating requests to the model's API endpoint.
+- `api_type` (str): Optional. The type of API service being used. This could be 'azure' for Azure Cognitive Services, 'openai' for OpenAI, or other custom types.
+- `base_url` (str): Optional. The base URL of the API endpoint. This is the root address where API calls are directed.
+- `api_version` (str): Optional. The version of the Azure API you wish to use
+
+For example:
+```python
+config_list = [
+{
+"model": "gpt-4",
+"api_key": os.environ.get("AZURE_OPENAI_API_KEY"),
+"api_type": "azure",
+"base_url": os.environ.get("AZURE_OPENAI_API_BASE"),
+"api_version": "2023-03-15-preview",
+},
+
+{
+"model": "llama-7B",
+"base_url": "http://127.0.0.1:8080",
+"api_type": "open_ai",
+}
+]
+```
+
+In `autogen` module there are [multiple helper functions](/docs/reference/oai/openai_utils) allowing to construct configurations using different sources:
 
 - `get_config_list`: Generates configurations for API calls, primarily from provided API keys.
 - `config_list_openai_aoai`: Constructs a list of configurations using both Azure OpenAI and OpenAI endpoints, sourcing API keys from environment variables or local files.
@@ -47,13 +76,21 @@ You can also explicitly specify that by:
 assistant = autogen.AssistantAgent(name="assistant", llm_config={"api_key": ...})
 ```
 
+### How does an agent decide which model to pick out of the list?
+
+An agent uses the very first model available in the "config_list" and makes LLM calls against this model. If the model fail (e.g. API throtling) the agent will retry the request against the 2nd model and so on until  prompt completion is received (or throws an error if none of the models successfully completes the request). There's no implicit/hidden logic inside agents that is used to pick "the best model for the task". It is developers responsibility to pick the right models and use them with agents.
+
+Besides throtling/rotating models the 'config_list' can be useful for:
+- Having a single global list of models and [filtering it](/docs/reference/oai/openai_utils#filter_config) based on certain keys (e.g. name, tag) in order to pass select models into a certain agent (e.g. use cheaper GPT 3.5 for agents solving easier tasks)
+- Using more advanced features not directly related to Agents, such as `filter_func` with [`OpenAIWrapper`](/docs/reference/oai/client#create) or [inference optimization](/docs/Examples#enhanced-inferences)
+
 ### Unexpected keyword argument 'base_url'
 
 In version >=1, OpenAI renamed their `api_base` parameter to `base_url`. So for older versions, use `api_base` but for newer versions use `base_url`.
 
 ### Can I use non-OpenAI models?
 
-Yes. Please check https://microsoft.github.io/autogen/blog/2023/07/14/Local-LLMs for an example.
+Yes. Autogen can work with any API endpoint which complies with OpenAI-compatible RESTful APIs - e.g. serving local LLM via FastChat or LM Studio. Please check https://microsoft.github.io/autogen/blog/2023/07/14/Local-LLMs for an example.
 
 ## Handle Rate Limit Error and Timeout Error
 


Code review comments:
diff_hunk: @@ -47,13 +76,21 @@ You can also explicitly specify that by:
 assistant = autogen.AssistantAgent(name="assistant", llm_config={"api_key": ...})
 ```
 
+### How does an agent decide which model to pick out of the list?
+
+An agent uses the very first model available in the "config_list" and makes LLM calls against this model. If the model fail (e.g. API throtling) the agent will retry the request against the 2nd model and so on until  prompt completion is received (or throws an error if none of the models successfully completes the request). There's no implicit/hidden logic inside agents that is used to pick "the best model for the task". It is developers responsibility to pick the right models and use them with agents.
+
+Besides throtling/rotating models the 'config_list' can be useful for:
+- Having a single global list of models and [filtering it](/docs/reference/oai/openai_utils#filter_config) based on certain keys (e.g. name, tag) in order to pass select models into a certain agent (e.g. use cheaper GPT 3.5 for agents solving easier tasks)
path: website/docs/FAQ.md
text: @user1:
<img width="895" alt="Screenshot 2024-01-29 at 23 06 23" src="https://github.com/microsoft/autogen/assets/22074246/b0adf7d9-e9ab-4b5f-bd14-2cd355697a9b">

Getting a 404 for this link

diff_hunk: @@ -20,7 +21,35 @@
 
 ## Set your API endpoints
 
-There are multiple ways to construct configurations for LLM inference in the `oai` utilities:
+Autogen relies on 3rd party API endpoints for LLM inference. It works great with OpenAI and Azure OpenAI models, and can work with any model (self-hosted or local) that is accessible through an inference server compatible with OpenAI Chat Completions API.
+
+An agent requires a list of configuration dictionaries for setting up model endpoints. Each configuration dictionary can contain the following keys:
+- `model` (str): Required. The identifier of the model to be used, such as 'gpt-4', 'gpt-3.5-turbo', or 'llama-7B'.
+- `api_key` (str): Optional. The API key required for authenticating requests to the model's API endpoint.
+- `api_type` (str): Optional. The type of API service being used. This could be 'azure' for Azure Cognitive Services, 'openai' for OpenAI, or other custom types.
+- `base_url` (str): Optional. The base URL of the API endpoint. This is the root address where API calls are directed.
+- `api_version` (str): Optional. The version of the Azure API you wish to use
+
+For example:
+```python
+config_list = [
+{
+"model": "gpt-4",
+"api_key": os.environ.get("AZURE_OPENAI_API_KEY"),
+"api_type": "azure",
+"base_url": os.environ.get("AZURE_OPENAI_API_BASE"),
+"api_version": "2023-03-15-preview",
path: website/docs/FAQ.md
text: @user1:
```suggestion
"api_version": "2023-12-01-preview",
```