Repository: opea-project/GenAIExamples
Pull request #1900: AgentQnA - add support for remote server
Author: login: alexsin368
Created: 2025-05-02T00:47:29Z | Merged: 2025-05-14T03:12:58Z

## Description

Add support for usage of a remote server inference endpoint. Supports Enterprise Inference.
Clean up README with fixed typos and additional instructions for setting up the OpenAI endpoint on the WebUI server.

Related dependent PR: https://github.com/opea-project/GenAIComps/pull/1644

## Issues

N/A

## Type of change

List the type of change like below. Please delete options that are not relevant.

- [ ] Bug fix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds new functionality)
- [ ] Breaking change (fix or feature that would break existing design and interface)
- [ ] Others (enhancement, documentation, validation, etc.)

## Dependencies

None

## Tests

Tested AgentQnA on the UI and verified the chat completions are working.



Diff:
diff --git a/AgentQnA/README.md b/AgentQnA/README.md
index 6844f716e7..c78703d6fb 100644
--- a/AgentQnA/README.md
+++ b/AgentQnA/README.md
@@ -99,7 +99,7 @@ flowchart LR
 
 #### First, clone the `GenAIExamples` repo.
 
-```
+```bash
 export WORKDIR=<your-work-directory>
 cd $WORKDIR
 git clone https://github.com/opea-project/GenAIExamples.git
@@ -109,7 +109,7 @@ git clone https://github.com/opea-project/GenAIExamples.git
 
 ##### For proxy environments only
 
-```
+```bash
 export http_proxy="Your_HTTP_Proxy"
 export https_proxy="Your_HTTPs_Proxy"
 # Example: no_proxy="localhost, 127.0.0.1, 192.168.1.1"
@@ -118,14 +118,24 @@ export no_proxy="Your_No_Proxy"
 
 ##### For using open-source llms
 
-```
+Set up a [HuggingFace](https://huggingface.co/) account and generate a [user access token](https://huggingface.co/docs/transformers.js/en/guides/private#step-1-generating-a-user-access-token).
+
+Then set an environment variable with the token and another for a directory to download the models:
+
+```bash
 export HUGGINGFACEHUB_API_TOKEN=<your-HF-token>
-export HF_CACHE_DIR=<directory-where-llms-are-downloaded> #so that no need to redownload every time
+export HF_CACHE_DIR=<directory-where-llms-are-downloaded> #  to avoid redownloading models
 ```
 
-##### [Optional] OPANAI_API_KEY to use OpenAI models
+##### [Optional] OPENAI_API_KEY to use OpenAI models or Intel® AI for Enterprise Inference
 
-```
+To use OpenAI models, generate a key following these [instructions](https://platform.openai.com/api-keys).
+
+To use a remote server running Intel® AI for Enterprise Inference, contact the cloud service provider or owner of the on-prem machine for a key to access the desired model on the server.
+
+Then set the environment variable `OPENAI_API_KEY` with the key contents:
+
+```bash
 export OPENAI_API_KEY=<your-openai-key>
 ```
 
@@ -133,16 +143,18 @@ export OPENAI_API_KEY=<your-openai-key>
 
 ##### Gaudi
 
-```
+```bash
 source $WORKDIR/GenAIExamples/AgentQnA/docker_compose/intel/hpu/gaudi/set_env.sh
 ```
 
 ##### Xeon
 
-```
+```bash
 source $WORKDIR/GenAIExamples/AgentQnA/docker_compose/intel/cpu/xeon/set_env.sh
 ```
 
+For running
+
 ### 2. Launch the multi-agent system. </br>
 
 We make it convenient to launch the whole system with docker compose, which includes microservices for LLM, agents, UI, retrieval tool, vector database, dataprep, and telemetry. There are 3 docker compose files, which make it easy for users to pick and choose. Users can choose a different retrieval tool other than the `DocIndexRetriever` example provided in our GenAIExamples repo. Users can choose not to launch the telemetry containers.
@@ -184,14 +196,37 @@ docker compose -f $WORKDIR/GenAIExamples/DocIndexRetriever/docker_compose/intel/
 
 #### Launch on Xeon
 
-On Xeon, only OpenAI models are supported. The command below will launch the multi-agent system with the `DocIndexRetriever` as the retrieval tool for the Worker RAG agent.
+On Xeon, OpenAI models and models deployed on a remote server are supported. Both methods require an API key.
 
 ```bash
 export OPENAI_API_KEY=<your-openai-key>
 cd $WORKDIR/GenAIExamples/AgentQnA/docker_compose/intel/cpu/xeon
+```
+
+##### OpenAI Models
+
+The command below will launch the multi-agent system with the `DocIndexRetriever` as the retrieval tool for the Worker RAG agent.
+
+```bash
 docker compose -f $WORKDIR/GenAIExamples/DocIndexRetriever/docker_compose/intel/cpu/xeon/compose.yaml -f compose_openai.yaml up -d
 ```
 
+##### Models on Remote Server
+
+When models are deployed on a remote server with Intel® AI for Enterprise Inference, a base URL and an API key are required to access them. To run the Agent microservice on Xeon while using models deployed on a remote server, add `compose_remote.yaml` to the `docker compose` command and set additional environment variables.
+
+###### Notes
+
+- `OPENAI_API_KEY` is already set in a previous step.
+- `model` is used to overwrite the value set for this environment variable in `set_env.sh`.
+- `LLM_ENDPOINT_URL` is the base URL given from the owner of the on-prem machine or cloud service provider. It will follow this format: "https://<DNS>". Here is an example: "https://api.inference.example.com".
+
+```bash
+export model=<name-of-model-card>
+export LLM_ENDPOINT_URL=<http-endpoint-of-remote-server>
+docker compose -f $WORKDIR/GenAIExamples/DocIndexRetriever/docker_compose/intel/cpu/xeon/compose.yaml -f compose_openai.yaml -f compose_remote.yaml up -d
+```
+
 ### 3. Ingest Data into the vector database
 
 The `run_ingest_data.sh` script will use an example jsonl file to ingest example documents into a vector database. Other ways to ingest data and other types of documents supported can be found in the OPEA dataprep microservice located in the opea-project/GenAIComps repo.
@@ -208,12 +243,18 @@ bash run_ingest_data.sh
 The UI microservice is launched in the previous step with the other microservices.
 To see the UI, open a web browser to `http://${ip_address}:5173` to access the UI. Note the `ip_address` here is the host IP of the UI microservice.
 
-1. `create Admin Account` with a random value
-2. add opea agent endpoint `http://$ip_address:9090/v1` which is a openai compatible api
+1. Click on the arrow above `Get started`. Create an admin account with a name, email, and password.
+2. Add an OpenAI-compatible API endpoint. In the upper right, click on the circle button with the user's initial, go to `Admin Settings`->`Connections`. Under `Manage OpenAI API Connections`, click on the `+` to add a connection. Fill in these fields:
+
+- **URL**: `http://${ip_address}:9090/v1`, do not forget the `v1`
+- **Key**: any value
+- **Model IDs**: any name i.e. `opea-agent`, then press `+` to add it
+
+Click "Save".
 
 ![opea-agent-setting](assets/img/opea-agent-setting.png)
 
-3. test opea agent with ui
+3. Test OPEA agent with UI. Return to `New Chat` and ensure the model (i.e. `opea-agent`) is selected near the upper left. Enter in any prompt to interact with the agent.
 
 ![opea-agent-test](assets/img/opea-agent-test.png)
 
diff --git a/AgentQnA/docker_compose/intel/cpu/xeon/compose_remote.yaml b/AgentQnA/docker_compose/intel/cpu/xeon/compose_remote.yaml
new file mode 100644
index 0000000000..24536435a3
--- /dev/null
+++ b/AgentQnA/docker_compose/intel/cpu/xeon/compose_remote.yaml
@@ -0,0 +1,18 @@
+# Copyright (C) 2025 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+services:
+  worker-rag-agent:
+    environment:
+      llm_endpoint_url: ${LLM_ENDPOINT_URL}
+      api_key: ${OPENAI_API_KEY}
+
+  worker-sql-agent:
+    environment:
+      llm_endpoint_url: ${LLM_ENDPOINT_URL}
+      api_key: ${OPENAI_API_KEY}
+
+  supervisor-react-agent:
+    environment:
+      llm_endpoint_url: ${LLM_ENDPOINT_URL}
+      api_key: ${OPENAI_API_KEY}


Reviews:
user: login: louie-tsai
body: Could you just create a yaml file to overwrite original compose.yaml instead of creating a new one?   it increase a maintenance efforts, and you probably just change couple values for vllm.  please check compose.telemetry.yaml as an example.

if all new features create a new compose.yaml, we will be overwhelmed with many compose.yaml files soon.

state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-02T01:36:14Z
user: login: louie-tsai
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-02T02:08:42Z
user: login: alexsin368
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-03T00:05:29Z
user: login: alexsin368
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-03T00:17:42Z
user: login: alexsin368
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-03T00:21:36Z
user: login: alexsin368
body: addressed comments
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-03T00:34:53Z
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

A pull request to add support for remote server inference, including configuration updates and documentation enhancements.
- Introduces a new docker-compose file with environment variables for remote inference.
- Updates the README with corrected typos and detailed instructions for setting up both OpenAI and remote server inference.

### Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

| File                                                                 | Description                                                                          |
|----------------------------------------------------------------------|--------------------------------------------------------------------------------------|
| AgentQnA/docker_compose/intel/cpu/xeon/compose_remote.yaml             | New configuration file for setting environment variables for remote inference.       |
| AgentQnA/README.md                                                   | Updated documentation with instructions for configuring remote inference and fixes.  |


<details>
<summary>Comments suppressed due to low confidence (1)</summary>

**AgentQnA/README.md:225**
* [nitpick] The variable name 'model' is generic and may lead to confusion; consider renaming it to something more descriptive, such as MODEL_ID, to clearly indicate its purpose.
```
export model=<name-of-model-card>
```
</details>


state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T08:35:17Z
user: login: minmin-intel
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-13T20:34:01Z
user: login: louie-tsai
body: looks good to me. thanks
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-14T00:52:42Z
user: login: yinghu5
body: LGTM, thank you!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-14T02:14:09Z

Review comments:
diff_hunk: @@ -184,14 +196,31 @@ docker compose -f $WORKDIR/GenAIExamples/DocIndexRetriever/docker_compose/intel/
 
 #### Launch on Xeon
 
-On Xeon, only OpenAI models are supported. The command below will launch the multi-agent system with the `DocIndexRetriever` as the retrieval tool for the Worker RAG agent.
+On Xeon, OpenAI models and models deployed on a remote server are supported. Both methods require an API key.
 
 ```bash
 export OPENAI_API_KEY=<your-openai-key>
 cd $WORKDIR/GenAIExamples/AgentQnA/docker_compose/intel/cpu/xeon
+```
+
+##### OpenAI Models
+
+The command below will launch the multi-agent system with the `DocIndexRetriever` as the retrieval tool for the Worker RAG agent.
+
+```bash
 docker compose -f $WORKDIR/GenAIExamples/DocIndexRetriever/docker_compose/intel/cpu/xeon/compose.yaml -f compose_openai.yaml up -d
 ```
 
+##### Models on Remote Server
+
+To run on Xeon with models deployed on a remote server, run with the `compose_remote.yaml` instead. Additional environment variables also need to be set.
+
+```bash
+export model=<name-of-model-card>
path: AgentQnA/README.md
user: login: louie-tsai
body: what is the difference between this model variable and MODEL_ID in set_env.sh. why users need to set model name in two difference places?
created_at: 2025-05-02T02:04:04Z
author_association: COLLABORATOR
diff_hunk: @@ -184,14 +196,31 @@ docker compose -f $WORKDIR/GenAIExamples/DocIndexRetriever/docker_compose/intel/
 
 #### Launch on Xeon
 
-On Xeon, only OpenAI models are supported. The command below will launch the multi-agent system with the `DocIndexRetriever` as the retrieval tool for the Worker RAG agent.
+On Xeon, OpenAI models and models deployed on a remote server are supported. Both methods require an API key.
 
 ```bash
 export OPENAI_API_KEY=<your-openai-key>
 cd $WORKDIR/GenAIExamples/AgentQnA/docker_compose/intel/cpu/xeon
+```
+
+##### OpenAI Models
+
+The command below will launch the multi-agent system with the `DocIndexRetriever` as the retrieval tool for the Worker RAG agent.
+
+```bash
 docker compose -f $WORKDIR/GenAIExamples/DocIndexRetriever/docker_compose/intel/cpu/xeon/compose.yaml -f compose_openai.yaml up -d
 ```
 
+##### Models on Remote Server
+
+To run on Xeon with models deployed on a remote server, run with the `compose_remote.yaml` instead. Additional environment variables also need to be set.
+
+```bash
+export model=<name-of-model-card>
+export LLM_ENDPOINT_URL=<http-endpoint-of-remote-server>
path: AgentQnA/README.md
user: login: louie-tsai
body: you probably need to give an example to explain how it works for Denvr.
created_at: 2025-05-02T02:07:04Z
author_association: COLLABORATOR
diff_hunk: @@ -184,14 +196,31 @@ docker compose -f $WORKDIR/GenAIExamples/DocIndexRetriever/docker_compose/intel/
 
 #### Launch on Xeon
 
-On Xeon, only OpenAI models are supported. The command below will launch the multi-agent system with the `DocIndexRetriever` as the retrieval tool for the Worker RAG agent.
+On Xeon, OpenAI models and models deployed on a remote server are supported. Both methods require an API key.
 
 ```bash
 export OPENAI_API_KEY=<your-openai-key>
 cd $WORKDIR/GenAIExamples/AgentQnA/docker_compose/intel/cpu/xeon
+```
+
+##### OpenAI Models
+
+The command below will launch the multi-agent system with the `DocIndexRetriever` as the retrieval tool for the Worker RAG agent.
+
+```bash
 docker compose -f $WORKDIR/GenAIExamples/DocIndexRetriever/docker_compose/intel/cpu/xeon/compose.yaml -f compose_openai.yaml up -d
 ```
 
+##### Models on Remote Server
+
+To run on Xeon with models deployed on a remote server, run with the `compose_remote.yaml` instead. Additional environment variables also need to be set.
+
+```bash
+export model=<name-of-model-card>
+export LLM_ENDPOINT_URL=<http-endpoint-of-remote-server>
+docker compose -f $WORKDIR/GenAIExamples/DocIndexRetriever/docker_compose/intel/cpu/xeon/compose.yaml -f compose_remote.yaml up -d
path: AgentQnA/README.md
user: login: louie-tsai
body: suggest to keep use the original compose.yaml but keep differences in compose.remote.yaml , so users could enable the feature by "docker compose -f compose.yaml -f compose.remote.yaml"  it avoids duplication and save maintenace effofrt
created_at: 2025-05-02T02:08:37Z
author_association: COLLABORATOR
diff_hunk: @@ -184,14 +196,31 @@ docker compose -f $WORKDIR/GenAIExamples/DocIndexRetriever/docker_compose/intel/
 
 #### Launch on Xeon
 
-On Xeon, only OpenAI models are supported. The command below will launch the multi-agent system with the `DocIndexRetriever` as the retrieval tool for the Worker RAG agent.
+On Xeon, OpenAI models and models deployed on a remote server are supported. Both methods require an API key.
 
 ```bash
 export OPENAI_API_KEY=<your-openai-key>
 cd $WORKDIR/GenAIExamples/AgentQnA/docker_compose/intel/cpu/xeon
+```
+
+##### OpenAI Models
+
+The command below will launch the multi-agent system with the `DocIndexRetriever` as the retrieval tool for the Worker RAG agent.
+
+```bash
 docker compose -f $WORKDIR/GenAIExamples/DocIndexRetriever/docker_compose/intel/cpu/xeon/compose.yaml -f compose_openai.yaml up -d
 ```
 
+##### Models on Remote Server
+
+To run on Xeon with models deployed on a remote server, run with the `compose_remote.yaml` instead. Additional environment variables also need to be set.
+
+```bash
+export model=<name-of-model-card>
path: AgentQnA/README.md
user: login: alexsin368
body: For the agent, "model" is the env variable used. There is no LLM_MODEL_ID. EMBEDDING_MODEL_ID and RERANK_MODEL_ID are used by the retriever.

We need to set `model` here to overwrite the original value (gpt-4o-mini-2024-07-18" set in set_env.sh. Added a note on this.
created_at: 2025-05-03T00:05:29Z
author_association: COLLABORATOR
diff_hunk: @@ -184,14 +196,31 @@ docker compose -f $WORKDIR/GenAIExamples/DocIndexRetriever/docker_compose/intel/
 
 #### Launch on Xeon
 
-On Xeon, only OpenAI models are supported. The command below will launch the multi-agent system with the `DocIndexRetriever` as the retrieval tool for the Worker RAG agent.
+On Xeon, OpenAI models and models deployed on a remote server are supported. Both methods require an API key.
 
 ```bash
 export OPENAI_API_KEY=<your-openai-key>
 cd $WORKDIR/GenAIExamples/AgentQnA/docker_compose/intel/cpu/xeon
+```
+
+##### OpenAI Models
+
+The command below will launch the multi-agent system with the `DocIndexRetriever` as the retrieval tool for the Worker RAG agent.
+
+```bash
 docker compose -f $WORKDIR/GenAIExamples/DocIndexRetriever/docker_compose/intel/cpu/xeon/compose.yaml -f compose_openai.yaml up -d
 ```
 
+##### Models on Remote Server
+
+To run on Xeon with models deployed on a remote server, run with the `compose_remote.yaml` instead. Additional environment variables also need to be set.
+
+```bash
+export model=<name-of-model-card>
+export LLM_ENDPOINT_URL=<http-endpoint-of-remote-server>
path: AgentQnA/README.md
user: login: alexsin368
body: I added some notes
created_at: 2025-05-03T00:17:42Z
author_association: COLLABORATOR
diff_hunk: @@ -184,14 +196,31 @@ docker compose -f $WORKDIR/GenAIExamples/DocIndexRetriever/docker_compose/intel/
 
 #### Launch on Xeon
 
-On Xeon, only OpenAI models are supported. The command below will launch the multi-agent system with the `DocIndexRetriever` as the retrieval tool for the Worker RAG agent.
+On Xeon, OpenAI models and models deployed on a remote server are supported. Both methods require an API key.
 
 ```bash
 export OPENAI_API_KEY=<your-openai-key>
 cd $WORKDIR/GenAIExamples/AgentQnA/docker_compose/intel/cpu/xeon
+```
+
+##### OpenAI Models
+
+The command below will launch the multi-agent system with the `DocIndexRetriever` as the retrieval tool for the Worker RAG agent.
+
+```bash
 docker compose -f $WORKDIR/GenAIExamples/DocIndexRetriever/docker_compose/intel/cpu/xeon/compose.yaml -f compose_openai.yaml up -d
 ```
 
+##### Models on Remote Server
+
+To run on Xeon with models deployed on a remote server, run with the `compose_remote.yaml` instead. Additional environment variables also need to be set.
+
+```bash
+export model=<name-of-model-card>
+export LLM_ENDPOINT_URL=<http-endpoint-of-remote-server>
+docker compose -f $WORKDIR/GenAIExamples/DocIndexRetriever/docker_compose/intel/cpu/xeon/compose.yaml -f compose_remote.yaml up -d
path: AgentQnA/README.md
user: login: alexsin368
body: good idea, implemented your suggestion
created_at: 2025-05-03T00:21:36Z
author_association: COLLABORATOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-05-02T00:47:41Z
body: <h1>Dependency Review</h1>
✅ No vulnerabilities or license issues found.<h2>Scanned Files</h2>
None

<!-- dependency-review-pr-comment-marker -->
author_association: NONE
user: login: alexsin368
created_at: 2025-05-03T00:40:50Z
body: The hyperlink check is failing with https://platform.openai.com/api-keys. I checked that it is a valid link, just requires the user to log in to their OpenAI account. Can we bypass this error?
author_association: COLLABORATOR
user: login: yinghu5
created_at: 2025-05-07T09:00:35Z
body: Hi @vikramts @NeoZhangJianyu  Here the Hyperlink check CI report error, could you please help to resolve it?
author_association: COLLABORATOR
user: login: vikramts
created_at: 2025-05-08T06:09:14Z
body: @alexsin368 - If the  https://platform.openai.com/api-keys URL is valid, then I don't see why we cannot bypass the error detail. Technically, this is not an error. And we can expect the user to understand that they need to log into their OpenAI account. So I do not see an error here really.
author_association: COLLABORATOR