Repository: microsoft/NLWeb
Pull request #101: fixing anthropic provider
Author: login: chelseacarter29
Labels: name: bug
description: Something isn't working
Created: 2025-05-19T20:29:22Z | Merged: 2025-05-23T15:48:46Z

updated anthropic llm provider; removed anthropic embedding model option (they don't have one; not needed)


Diff:
diff --git a/code/.env.template b/code/.env.template
index 95357f0c5..b50fa2e7f 100644
--- a/code/.env.template
+++ b/code/.env.template
@@ -14,6 +14,8 @@ AZURE_VECTOR_SEARCH_API_KEY="<TODO>"
 AZURE_OPENAI_ENDPOINT="https://TODO.openai.azure.com/"
 AZURE_OPENAI_API_KEY="<TODO>"
 
+ANTHROPIC_API_KEY="<TODO>"
+
 INCEPTION_ENDPOINT="https://api.inceptionlabs.ai/v1/chat/completions"
 INCEPTION_API_KEY="<TODO>"
 
diff --git a/code/config/config_embedding.yaml b/code/config/config_embedding.yaml
index 7df5b86e7..dae78d67c 100644
--- a/code/config/config_embedding.yaml
+++ b/code/config/config_embedding.yaml
@@ -6,10 +6,6 @@ providers:
     api_endpoint_env: OPENAI_ENDPOINT
     model: text-embedding-3-small
 
-  anthropic:
-    api_key_env: ANTHROPIC_API_KEY
-    model: ANTHROPIC_EMBEDDING_MODEL
-
   gemini:
     api_key_env: GCP_PROJECT
     model: GCP_EMBEDDING_MODEL
diff --git a/code/config/config_llm.yaml b/code/config/config_llm.yaml
index a3e24ada0..c5001e677 100644
--- a/code/config/config_llm.yaml
+++ b/code/config/config_llm.yaml
@@ -18,8 +18,8 @@ providers:
   anthropic:
     api_key_env: ANTHROPIC_API_KEY
     models:
-      high: claude-2
-      low: claude-2
+      high: claude-3-5-sonnet-20241022
+      low: claude-3-haiku-20240307
 
   gemini:
     api_key_env: GCP_PROJECT
diff --git a/code/embedding/anthropic_embedding.py b/code/embedding/anthropic_embedding.py
deleted file mode 100644
index 586146976..000000000
--- a/code/embedding/anthropic_embedding.py
+++ /dev/null
@@ -1,119 +0,0 @@
-# Copyright (c) 2025 Microsoft Corporation.
-# Licensed under the MIT License
-
-"""
-Anthropic embedding implementation.
-
-WARNING: This code is under development and may undergo changes in future releases.
-Backwards compatibility is not guaranteed at this time.
-"""
-
-import os
-import asyncio
-import threading
-from typing import List, Optional
-
-from anthropic import AsyncAnthropic
-from config.config import CONFIG
-
-from utils.logging_config_helper import get_configured_logger, LogLevel
-logger = get_configured_logger("anthropic_embedding")
-
-# Add lock for thread-safe client access
-_client_lock = threading.Lock()
-anthropic_client = None
-
-def get_anthropic_api_key() -> str:
-    """
-    Retrieve the Anthropic API key from configuration.
-    """
-    # Get the API key from the embedding provider config
-    provider_config = CONFIG.get_embedding_provider("anthropic")
-    if provider_config and provider_config.api_key:
-        api_key = provider_config.api_key
-        if api_key:
-            return api_key
-    
-    # Fallback to environment variable
-    api_key = os.getenv("ANTHROPIC_API_KEY")
-    if not api_key:
-        error_msg = "Anthropic API key not found in configuration or environment"
-        logger.error(error_msg)
-        raise ValueError(error_msg)
-    
-    return api_key
-
-def get_async_client() -> AsyncAnthropic:
-    """
-    Configure and return an asynchronous Anthropic client.
-    """
-    global anthropic_client
-    with _client_lock:  # Thread-safe client initialization
-        if anthropic_client is None:
-            try:
-                api_key = get_anthropic_api_key()
-                anthropic_client = AsyncAnthropic(api_key=api_key)
-                logger.debug("Anthropic client initialized successfully")
-            except Exception as e:
-                logger.exception("Failed to initialize Anthropic client")
-                raise
-    
-    return anthropic_client
-
-async def get_anthropic_embeddings(
-    text: str,
-    model: Optional[str] = None,
-    timeout: float = 30.0
-) -> List[float]:
-    """
-    Generate an embedding for a single text using Anthropic API.
-    
-    Args:
-        text: The text to embed
-        model: Optional model ID to use, defaults to provider's configured model
-        timeout: Maximum time to wait for the embedding response in seconds
-        
-    Returns:
-        List of floats representing the embedding vector
-    """
-    # If model not provided, get it from config
-    if model is None:
-        provider_config = CONFIG.get_embedding_provider("anthropic")
-        if provider_config and provider_config.model:
-            model = provider_config.model
-        else:
-            # Default to a common Anthropic embedding model
-            model = "claude-3-embedding-v1"
-    
-    logger.debug(f"Generating Anthropic embedding with model: {model}")
-    logger.debug(f"Text length: {len(text)} chars")
-    
-    client = get_async_client()
-
-    try:
-        # Anthropic's embedding API usage
-        response = await client.embeddings.create(
-            model=model,
-            input=text,
-            dimensions=1536  # Optional: specify dimensions if the model supports it
-        )
-        
-        embedding = response.embedding
-        logger.debug(f"Anthropic embedding generated, dimension: {len(embedding)}")
-        return embedding
-    except Exception as e:
-        logger.exception("Error generating Anthropic embedding")
-        logger.log_with_context(
-            LogLevel.ERROR,
-            "Anthropic embedding generation failed",
-            {
-                "model": model,
-                "text_length": len(text),
-                "error_type": type(e).__name__,
-                "error_message": str(e)
-            }
-        )
-        raise
-
-# Note: Anthropic might not support native batch embedding as of now,
-# so the main wrapper will handle batching by making multiple single embedding calls.
\ No newline at end of file
diff --git a/code/embedding/embedding.py b/code/embedding/embedding.py
index 08b83a177..bc8a4354d 100644
--- a/code/embedding/embedding.py
+++ b/code/embedding/embedding.py
@@ -20,7 +20,6 @@
 # Add locks for thread-safe provider access
 _provider_locks = {
     "openai": threading.Lock(),
-    "anthropic": threading.Lock(),
     "gemini": threading.Lock(),
     "azure_openai": threading.Lock(),
     "snowflake": threading.Lock()
@@ -82,17 +81,6 @@ async def get_embedding(
             logger.debug(f"OpenAI embeddings received, dimension: {len(result)}")
             return result
 
-        if provider == "anthropic":
-            logger.debug("Getting Anthropic embeddings")
-            # Import here to avoid circular imports
-            from embedding.anthropic_embedding import get_anthropic_embeddings
-            result = await asyncio.wait_for(
-                get_anthropic_embeddings(text, model=model_id),
-                timeout=timeout
-            )
-            logger.debug(f"Anthropic embeddings received, dimension: {len(result)}")
-            return result
-
         if provider == "gemini":
             logger.debug("Getting Gemini embeddings")
             # Import here to avoid potential circular imports
@@ -219,21 +207,6 @@ async def batch_get_embeddings(
             logger.debug(f"Snowflake batch embeddings received, count: {len(result)}")
             return result
             
-        if provider == "anthropic":
-            # Anthropic might not have a native batch API, so process one by one
-            logger.debug("Getting Anthropic batch embeddings (sequential)")
-            from embedding.anthropic_embedding import get_anthropic_embeddings
-            # Process texts one by one with individual timeouts
-            results = []
-            for text in texts:
-                embedding = await asyncio.wait_for(
-                    get_anthropic_embeddings(text, model=model_id),
-                    timeout=30  # Individual timeout per text
-                )
-                results.append(embedding)
-            logger.debug(f"Anthropic batch embeddings received, count: {len(results)}")
-            return results
-            
         if provider == "gemini":
             # Gemini might not have a native batch API, so process one by one
             logger.debug("Getting Gemini batch embeddings (sequential)")
diff --git a/code/llm/anthropic.py b/code/llm/anthropic.py
index 7802f7479..2d042e6e1 100644
--- a/code/llm/anthropic.py
+++ b/code/llm/anthropic.py
@@ -40,12 +40,13 @@ def get_api_key(cls) -> str:
         """Retrieve the Anthropic API key from the environment or raise an error."""
         # Get the API key from the preferred provider config
         provider_config = CONFIG.llm_providers["anthropic"]
-        api_key_env_var = provider_config.api_key_env
-        
-        key = os.getenv(api_key_env_var)
-        if not key:
-            raise ConfigurationError(f"{api_key_env_var} is not set")
-        return key
+        if provider_config and provider_config.api_key:
+            api_key = provider_config.api_key
+            if api_key:
+                api_key = api_key.strip('"')  # Remove quotes if present
+                return api_key
+        # If we didn't find a key, the environment variable is not set properly
+        raise ConfigurationError("Environment variable ANTHROPIC_API_KEY is not set")
 
     @classmethod
     def get_client(cls) -> AsyncAnthropic:
diff --git a/docs/database-load-tool.md b/docs/database-load-tool.md
index 0feb928e7..6ce71b301 100644
--- a/docs/database-load-tool.md
+++ b/docs/database-load-tool.md
@@ -30,7 +30,7 @@ With this command, three things happen in the following order:
 The command structure for this is as follows - run this while in your `myenv` virtual environment from the 'code' folder:
 
 ```sh
-python db_load.py <file-path> <site-name>
+python -m tools.db_load <file-path> <site-name>
 ```
 
 The 'file-path' can be a URL or local path.  The 'site-name' is what the data source is called if you want to scope your search (in code/config_nlweb.yaml) or need to remove the site and/or entries.


Reviews:
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

Updates the Anthropic LLM provider configuration and removes the unused Anthropic embedding model option.

- Switched `db_load` invocation to use module syntax.
- Adjusted Anthropic API key retrieval to pull from `provider_config.api_key` and strip quotes.
- Removed Anthropic embedding provider entry and updated LLM model versions.

### Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

<details>
<summary>Show a summary per file</summary>

| File                          | Description                                                  |
| ----------------------------- | ------------------------------------------------------------ |
| docs/db_load.md               | Change from `python db_load.py` to `python -m tools.db_load`. |
| code/llm/anthropic.py         | Modified `get_api_key` to use `provider_config.api_key` and return `None`. |
| code/embedding/anthropic_embedding.py | Removed embedding model config, updated API key retrieval.       |
| code/config/config_llm.yaml   | Updated Anthropic high/low model versions.                   |
| code/config/config_embedding.yaml | Removed Anthropic embedding provider block.                     |
</details>



<details>
<summary>Comments suppressed due to low confidence (2)</summary>

**code/embedding/anthropic_embedding.py:39**
* The `os` module is used here without importing it, which will cause a `NameError` at runtime. Please add `import os` at the top of the file.
```
api_key = os.getenv("ANTHROPIC_API_KEY")
```
**code/llm/anthropic.py:48**
* The method now returns None instead of raising an error when no API key is found, which may lead to unexpected None values downstream. Consider raising `ConfigurationError` to maintain previous behavior.
```
return None
```
</details>


state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T16:35:15Z
user: login: jennifermarsman
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-21T17:12:47Z
user: login: chelseacarter29
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-23T00:42:07Z
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR fixes the Anthropic LLM provider by updating the API key retrieval mechanism and removing support for Anthropic embeddings (since Anthropic does not provide one).  
- Update Anthropic API key retrieval in the LLM provider  
- Remove Anthropic embedding support from code, configuration, and documentation changes  
- Update configuration models and .env template to reflect the latest Anthropic provider settings

### Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

<details>
<summary>Show a summary per file</summary>

| File                                | Description                                                        |
| ----------------------------------- | ------------------------------------------------------------------ |
| docs/database-load-tool.md          | Updated command usage to run the database load tool via the tools module |
| code/llm/anthropic.py               | Updated API key retrieval mechanism and error messaging              |
| code/embedding/embedding.py          | Removed Anthropic embedding logic from embedding functions           |
| code/embedding/anthropic_embedding.py | Removed unused Anthropic embedding implementation                   |
| code/config/config_llm.yaml         | Updated Anthropic model configuration                               |
| code/config/config_embedding.yaml   | Removed Anthropic embedding configuration                           |
| code/.env.template                  | Added Anthropic API key placeholder                                  |
</details>



<details>
<summary>Comments suppressed due to low confidence (1)</summary>

**code/config/config_embedding.yaml:8**
* Since the Anthropic embedding configuration has been removed, please verify that any related documentation or references have been updated accordingly to prevent confusion.
```
-  anthropic:
```
</details>


state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-23T14:35:49Z
user: login: jennifermarsman
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-23T14:42:01Z
user: login: rvguha
body: lgtm
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-23T15:48:45Z

Review comments:
diff_hunk: @@ -12,7 +12,6 @@
 import asyncio
path: code/embedding/anthropic_embedding.py
user: login: jennifermarsman
body: Can this whole file be deleted if there is no embedding support for Anthropic?  
created_at: 2025-05-21T17:12:47Z
author_association: COLLABORATOR
diff_hunk: @@ -12,7 +12,6 @@
 import asyncio
path: code/embedding/anthropic_embedding.py
user: login: chelseacarter29
body: Yes, i think so - we should probably make a note in documentation about that potentiality (can update in another PR with other documentation).
created_at: 2025-05-23T00:42:07Z
author_association: COLLABORATOR
diff_hunk: @@ -40,12 +40,13 @@ def get_api_key(cls) -> str:
         """Retrieve the Anthropic API key from the environment or raise an error."""
         # Get the API key from the preferred provider config
         provider_config = CONFIG.llm_providers["anthropic"]
-        api_key_env_var = provider_config.api_key_env
-        
-        key = os.getenv(api_key_env_var)
-        if not key:
-            raise ConfigurationError(f"{api_key_env_var} is not set")
-        return key
+        if provider_config and provider_config.api_key:
+            api_key = provider_config.api_key
+            if api_key:
+                api_key = api_key.strip('"')  # Remove quotes if present
+                return api_key
+        # If we didn't find a key, the environment variable is not set properly
+        raise ConfigurationError("Environment variable ANTHROPIC_API_KEY is not set")
path: code/llm/anthropic.py
user: login: Copilot
body: The error message still references an environment variable even though the API key is now retrieved from the provider configuration. Consider updating the message to indicate that the Anthropic API key is missing from the configuration.
```suggestion
        # If we didn't find a key, the API key is missing from the provider configuration
        raise ConfigurationError("Anthropic API key is missing from the provider configuration (CONFIG.llm_providers['anthropic'].api_key)")
```
created_at: 2025-05-23T14:35:49Z
author_association: CONTRIBUTOR

PR comments:
user: login: coffee-mug
created_at: 2025-05-21T08:21:34Z
body: Anthropic suggests using Voyage embedding models in their docs so did some changes in that sense yet I can't have it running locally, issue about dimensions reduction with qdrant. Were you able to run your code locally with Claude? 
author_association: NONE
user: login: jennifermarsman
created_at: 2025-05-23T14:39:24Z
body: > Anthropic suggests using Voyage embedding models in their docs so did some changes in that sense yet I can't have it running locally, issue about dimensions reduction with qdrant. Were you able to run your code locally with Claude?

@coffee-mug Agreed: https://docs.anthropic.com/en/docs/build-with-claude/embeddings.  I'll open a new issue to track adding support for Voyager embeddings: #128 .  Thanks!  
author_association: COLLABORATOR