Repository: citadel-ai/langcheck
Pull request #190: structured output for `GeminiEvalClient.get_float_score` in newer Gemini SDK
Author: login: taniokay
Created: 2025-03-25T09:44:26Z | Merged: 2025-03-27T09:44:45Z

Resolves #187 

As specified in the issue, the currently used Gemini SDK was deprecated. This PR migrates to the newer Gemini SDK `google-genai`, and uses structured output for `get_float_score`.

Linked issue #187: Gemini SDK is deprecated
The current Gemini SDK is marked as deprecated: https://github.com/google-gemini/deprecated-generative-ai-python, so we should migrate to new SDK: https://ai.google.dev/gemini-api/docs/migrate
Linked issue comments: user: login: liwii
created_at: 2025-03-24T08:52:01Z
body: Good to know 😢 
author_association: CONTRIBUTOR

Diff:
diff --git a/pyproject.toml b/pyproject.toml
index f383ecb8..614003ef 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -25,7 +25,7 @@ dependencies = [
     'plotly >= 5',
     'requests',
     'rouge-score >= 0.1.2',
-    'sentence-transformers >= 2',
+    'sentence-transformers >= 2, < 4',
     'sentencepiece>=0.1.95',
     'tomli; python_version < "3.11"',
     'tokenizers >= 0.13.2; python_version >= "3.11"',  # See https://github.com/citadel-ai/langcheck/pull/45
@@ -59,7 +59,7 @@ anthropic = [
     'anthropic'
 ]
 gemini = [
-    'google-generativeai'
+    'google-genai'
 ]
 local-llm = [
     'vllm'
diff --git a/src/langcheck/metrics/eval_clients/_gemini.py b/src/langcheck/metrics/eval_clients/_gemini.py
index c2ed5400..2af584d1 100644
--- a/src/langcheck/metrics/eval_clients/_gemini.py
+++ b/src/langcheck/metrics/eval_clients/_gemini.py
@@ -1,14 +1,11 @@
 from __future__ import annotations
 
-import os
-import warnings
-from typing import Any
+from typing import Any, Literal
 
-import google.ai.generativelanguage as glm
 import torch
-from google.generativeai.client import configure
-from google.generativeai.embedding import embed_content
-from google.generativeai.generative_models import GenerativeModel
+from google import genai
+from google.genai import types
+from pydantic import BaseModel
 
 from langcheck.utils.progress_bar import tqdm_wrapper
 
@@ -22,8 +19,7 @@ class GeminiEvalClient(EvalClient):
 
     def __init__(
         self,
-        model: GenerativeModel | None = None,
-        model_args: dict[str, Any] | None = None,
+        model_name: str = "gemini-1.5-flash",
         generate_content_args: dict[str, Any] | None = None,
         embed_model_name: str | None = None,
         *,
@@ -34,47 +30,44 @@ def __init__(
         information is automatically read from the environment variables,
         so please make sure GOOGLE_API_KEY is set.
 
-        TODO: Allow the user to specify the use of async. There currently
-        seems to be an issue with `generate_content_async()` that is blocking
-        this: https://github.com/google-gemini/generative-ai-python/issues/207
+        TODO: Allow the user to specify the use of async.
+        https://github.com/citadel-ai/langcheck/issues/191
 
         Ref:
             https://ai.google.dev/api/python/google/generativeai/GenerativeModel
 
         Args:
-            model: (Optional) The Gemini model to use. If not provided, the
-                model will be created using the model_args.
-            model_args: (Optional) Dict of args to create the Gemini model.
+            model_name: (Optional) The Gemini model to use. Defaults to
+                "gemini-1.5-flash".
             generate_content_args: (Optional) Dict of args to pass in to the
-                ``generate_content`` function.
+                ``generate_content`` function. The keys should be the same as
+                the keys in the ``genai.types.GenerateContentConfig`` type.
             embed_model_name: (Optional) The name of the embedding model to use.
-                If not provided, the models/embedding-001 model will be used.
-            system_prompt: (Optional) The system prompt to use. If not provided,
-                no system prompt will be used.
+                If not provided, the models/text-embedding-004 model will be used.
+            system_prompt: (Optional) The system prompt for ``generate_content``
+                in ``get_text_responses`` function. If not provided, no system
+                prompt will be used.
         """
-        if model:
-            self._text_response_model = model
-            self._structured_assessment_model = model
-        else:
-            configure(api_key=os.getenv("GOOGLE_API_KEY"))
-            model_args = model_args or {}
-            self._structured_assessment_model = GenerativeModel(**model_args)
-            # Only add system prompt to the text response model if it is provided
-            if system_prompt:
-                if "system_instruction" in model_args:
-                    warnings.warn(
-                        '"system_instruction" of model_args will be ignored because '
-                        "system_prompt is provided."
-                    )
-                model_args["system_instruction"] = system_prompt
-            self._text_response_model = GenerativeModel(**model_args)
-
+        self.client = genai.Client()
+        self._model_name = model_name
         self._generate_content_args = generate_content_args or {}
         self._embed_model_name = embed_model_name
+        self._system_instruction = system_prompt
+
+        self._validate_generate_content_config()
+
+    def _validate_generate_content_config(self) -> None:
+        try:
+            _ = types.GenerateContentConfig(**self._generate_content_args)
+        except (TypeError, ValueError) as e:
+            raise ValueError(
+                f"Invalid generate_content_args: {self._generate_content_args}"
+                f"Error: {e}"
+            )
 
     def _call_api(
         self,
-        model: GenerativeModel,
+        model: str,
         prompts: list[str] | list[str | None],
         config: dict[str, Any],
         *,
@@ -84,7 +77,11 @@ def _call_api(
         # of exception handling with the async version.
         def _call_api_with_exception_filter(prompt: str) -> Any:
             try:
-                return model.generate_content(prompt, **config)
+                return self.client.models.generate_content(
+                    model=model,
+                    contents=prompt,
+                    config=types.GenerateContentConfig(**config),
+                )
             except Exception as e:
                 return e
 
@@ -128,11 +125,15 @@ def get_text_responses(
                 f"prompts must be a list, not a {type(prompts).__name__}"
             )
 
-        config = {"generation_config": {"temperature": 0.0}}
+        config: dict[str, Any] = {
+            "temperature": 0.0,
+            "system_instruction": self._system_instruction,
+        }
         config.update(self._generate_content_args or {})
+
         tqdm_description = tqdm_description or "Intermediate assessments (1/2)"
         responses = self._call_api(
-            model=self._text_response_model,
+            model=self._model_name,
             prompts=prompts,
             config=config,
             tqdm_description=tqdm_description,
@@ -154,13 +155,12 @@ def get_float_score(
     ) -> list[float | None]:
         """The function that transforms the unstructured assessments (i.e. long
         texts that describe the evaluation results) into scores. We leverage the
-        function calling API to extract the short assessment results from the
+        structured output API to extract the short assessment results from the
         unstructured assessments, so please make sure that the model you use
-        supports function calling
-        (https://ai.google.dev/gemini-api/docs/function-calling#supported-models).
+        supports structured output (See the References for more details).
 
-        Ref:
-            https://ai.google.dev/gemini-api/docs/function-calling
+        References:
+            https://ai.google.dev/gemini-api/docs/structured-output
 
         Args:
             metric_name: The name of the metric to be used. (e.g. "toxicity")
@@ -178,13 +178,24 @@ def get_float_score(
         if language not in ["en", "ja", "de"]:
             raise ValueError(f"Unsupported language: {language}")
 
-        fn_call_template = get_template(
-            f"{language}/get_score/function_calling.j2"
+        structured_output_template = get_template(
+            f"{language}/get_score/structured_output.j2"
         )
 
         options = list(score_map.keys())
-        fn_call_messages = [
-            fn_call_template.render(
+
+        class Response(BaseModel):
+            score: Literal[tuple(options)]  # type: ignore
+
+        config = {
+            "temperature": 0.0,
+            "response_mime_type": "application/json",
+            "response_schema": Response,
+        }
+        config.update(self._generate_content_args or {})
+
+        prompts = [
+            structured_output_template.render(
                 {
                     "metric": metric_name,
                     "unstructured_assessment": unstructured_assessment,
@@ -196,47 +207,18 @@ def get_float_score(
             for unstructured_assessment in unstructured_assessment_result
         ]
 
-        assessment_schema = glm.Schema(type=glm.Type.STRING, enum=options)
-
-        save_assessment = glm.FunctionDeclaration(
-            name="save_assessment",
-            description=f"Save the assessment of {metric_name}.",
-            parameters=glm.Schema(
-                type=glm.Type.OBJECT,
-                properties={"assessment": assessment_schema},
-            ),
-        )
-        config_structured_assessments = {
-            "tools": [save_assessment],
-            "tool_config": {"function_calling_config": "ANY"},
-            "generation_config": {"temperature": 0.0},
-        }
-        config_structured_assessments.update(self._generate_content_args or {})
-
         tqdm_description = tqdm_description or "Scores (2/2)"
         responses = self._call_api(
-            model=self._structured_assessment_model,
-            prompts=fn_call_messages,
-            config=config_structured_assessments,
+            model=self._model_name,
+            prompts=prompts,
+            config=config,
             tqdm_description=tqdm_description,
         )
-        assessments = []
-        for response in responses:
-            if response is None:
-                assessments.append(None)
-                continue
-            function_call = (
-                response.candidates[0].content.parts[0].function_call
-            )
-            fc_dict = type(function_call).to_dict(function_call)
-            assessments.append(fc_dict.get("args", {}).get("assessment"))
-        # Check if any of the assessments are not recognized.
-        for assessment in assessments:
-            if (assessment is None) or (assessment in options):
-                continue
-            # By leveraging the function calling API, this should be pretty
-            # rare, but we're dealing with LLMs here so nothing is absolute!
-            print(f'Gemini returned an unrecognized assessment: "{assessment}"')
+
+        assessments = [
+            response.parsed.score if response else None
+            for response in responses
+        ]
 
         return [
             score_map[assessment]
@@ -246,7 +228,10 @@ def get_float_score(
         ]
 
     def similarity_scorer(self) -> GeminiSimilarityScorer:
-        return GeminiSimilarityScorer(embed_model_name=self._embed_model_name)
+        return GeminiSimilarityScorer(
+            embed_model_name=self._embed_model_name,
+            client=self.client,
+        )
 
 
 class GeminiSimilarityScorer(BaseSimilarityScorer):
@@ -255,16 +240,25 @@ class GeminiSimilarityScorer(BaseSimilarityScorer):
     EvalClients.
     """
 
-    def __init__(self, embed_model_name: str | None):
+    def __init__(
+        self,
+        embed_model_name: str | None,
+        client: genai.Client,
+    ):
         super().__init__()
 
-        self.embed_model_name = embed_model_name or "models/embedding-001"
+        self._embed_model_name = embed_model_name or "models/text-embedding-004"
+        self._client = client
 
     def _embed(self, inputs: list[str]) -> torch.Tensor:
         """Embed the inputs using the Gemini API."""
         # Embed the inputs
-        embed_response = embed_content(
-            model=self.embed_model_name, content=inputs
+        embed_response = self._client.models.embed_content(
+            model=self._embed_model_name,
+            contents=inputs,  # type: ignore
         )
 
-        return torch.Tensor([item for item in embed_response["embedding"]])
+        assert embed_response.embeddings is not None
+        return torch.Tensor(
+            [embed.values for embed in embed_response.embeddings]
+        )
diff --git a/src/langcheck/metrics/eval_clients/_openai.py b/src/langcheck/metrics/eval_clients/_openai.py
index 6a5a6e1e..4bf73a58 100644
--- a/src/langcheck/metrics/eval_clients/_openai.py
+++ b/src/langcheck/metrics/eval_clients/_openai.py
@@ -269,7 +269,7 @@ class Response(BaseModel):
             score: Literal[tuple(options)]  # type: ignore
 
         structured_output_template = get_template(
-            f"{language}/get_score/function_calling.j2"
+            f"{language}/get_score/structured_output.j2"
         )
 
         config = {"model": "gpt-4o-mini"}
diff --git a/src/langcheck/metrics/prompts/de/get_score/function_calling.j2 b/src/langcheck/metrics/prompts/de/get_score/structured_output.j2
similarity index 100%
rename from src/langcheck/metrics/prompts/de/get_score/function_calling.j2
rename to src/langcheck/metrics/prompts/de/get_score/structured_output.j2
diff --git a/src/langcheck/metrics/prompts/en/get_score/function_calling.j2 b/src/langcheck/metrics/prompts/en/get_score/structured_output.j2
similarity index 100%
rename from src/langcheck/metrics/prompts/en/get_score/function_calling.j2
rename to src/langcheck/metrics/prompts/en/get_score/structured_output.j2
diff --git a/src/langcheck/metrics/prompts/ja/get_score/function_calling.j2 b/src/langcheck/metrics/prompts/ja/get_score/structured_output.j2
similarity index 100%
rename from src/langcheck/metrics/prompts/ja/get_score/function_calling.j2
rename to src/langcheck/metrics/prompts/ja/get_score/structured_output.j2
diff --git a/src/langcheck/metrics/scorer/detoxify_models.py b/src/langcheck/metrics/scorer/detoxify_models.py
index c0f659cc..da8b087c 100644
--- a/src/langcheck/metrics/scorer/detoxify_models.py
+++ b/src/langcheck/metrics/scorer/detoxify_models.py
@@ -36,7 +36,7 @@ def load_checkpoint(
     loaded = torch.hub.load_state_dict_from_url(
         checkpoint_url, map_location=device
     )
-    class_names = loaded["config"]["dataset"]["args"]["classes"]
+    class_names: list[str] = loaded["config"]["dataset"]["args"]["classes"]
     change_names = {
         "toxic": "toxicity",
         "identity_hate": "identity_attack",
diff --git a/tests/metrics/eval_clients/test_gemini.py b/tests/metrics/eval_clients/test_gemini.py
index c68993a2..4b12695b 100644
--- a/tests/metrics/eval_clients/test_gemini.py
+++ b/tests/metrics/eval_clients/test_gemini.py
@@ -1,10 +1,12 @@
 from __future__ import annotations
 
 import os
+from typing import Literal
 from unittest.mock import Mock, patch
 
 import pytest
-from google.generativeai.types import generation_types
+from google.genai import types
+from pydantic import BaseModel
 
 from langcheck.metrics.eval_clients import GeminiEvalClient
 
@@ -13,13 +15,13 @@
 def test_get_text_response_gemini(system_prompt):
     prompts = ["Assess the factual consistency of the generated output..."] * 2
     answer = "The output is fully factually consistent."
-    mock_response = Mock(spec=generation_types.GenerateContentResponse)
+    mock_response = Mock(spec=types.GenerateContentResponse)
     mock_response.text = answer
     mock_response.candidates = [Mock(finish_reason=1)]
-    # Calling the google.generativeai.GenerativeModel.generate_content method
+    # Calling the google.genai.models.Models.generate_content method
     # requires a Google API key, so we mock the return value instead
     with patch(
-        "google.generativeai.GenerativeModel.generate_content",
+        "google.genai.models.Models.generate_content",
         return_value=mock_response,
     ):
         # Set the necessary env vars for the GeminiEvalClient
@@ -40,25 +42,17 @@ def test_get_float_score_gemini(system_prompt, language):
     short_assessment_result = "Fully Consistent"
     score_map = {short_assessment_result: 1.0}
 
-    mock_response = Mock(spec=generation_types.GenerateContentResponse)
-    mock_response.text = short_assessment_result
+    class Response(BaseModel):
+        score: Literal[tuple(score_map.keys())]  # type: ignore
 
-    class FunctionCallMock(Mock):
-        @classmethod
-        def to_dict(cls, instance):
-            return {"args": {"assessment": short_assessment_result}}
-
-    mock_response.candidates = [
-        Mock(
-            finish_reason=1,
-            content=Mock(parts=[Mock(function_call=FunctionCallMock())]),
-        )
-    ]
+    mock_response = Mock(spec=types.GenerateContentResponse)
+    mock_response.parsed = Response(score=short_assessment_result)
+    mock_response.candidates = [Mock(finish_reason=1)]
 
-    # Calling the google.generativeai.GenerativeModel.generate_content method
+    # Calling the google.genai.models.Models.generate_content method
     # requires a Google API key, so we mock the return value instead
     with patch(
-        "google.generativeai.GenerativeModel.generate_content",
+        "google.genai.models.Models.generate_content",
         return_value=mock_response,
     ):
         # Set the necessary env vars for the GeminiEvalClient
@@ -74,13 +68,19 @@ def to_dict(cls, instance):
 
 
 def test_similarity_scorer_gemini():
-    mock_embedding_response = {"embedding": [[0.1, 0.2, 0.3]]}
+    mock_embedding_response = [0.1, 0.2, 0.3]
 
-    # Calling the google.generativeai.embed_content method requires a Google
+    # Calling the google.genai.models.Models.embed_content method requires a Google
     # API key, so we mock the return value instead
     with patch(
-        "langcheck.metrics.eval_clients._gemini.embed_content",
-        Mock(return_value=mock_embedding_response),
+        "google.genai.models.Models.embed_content",
+        Mock(
+            return_value=types.EmbedContentResponse(
+                embeddings=[
+                    types.ContentEmbedding(values=mock_embedding_response)
+                ]
+            )
+        ),
     ):
         # Set the necessary env vars for the GeminiEvalClient
         os.environ["GOOGLE_API_KEY"] = "dummy_key"


Reviews:
user: login: liwii
body: Thanks for the change!! As far as I tested it is working pretty well!!

Made minor comments, mainly about
- Additional arguments to support VertexAI properly
- Consistency of the argument names with previous versions & other eval clients
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T07:34:49Z
user: login: taniokay
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T14:14:44Z
user: login: taniokay
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T14:14:54Z
user: login: taniokay
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T14:15:02Z
user: login: taniokay
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T14:15:10Z
user: login: taniokay
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T14:19:19Z
user: login: liwii
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T07:30:39Z
user: login: liwii
body: LGTM, thanks!

There seemed to be type errors not related to this PR because of the recent update of pyright & sentence-transformer, so I directly pushed fix.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T08:14:34Z

Review comments:
diff_hunk: @@ -47,35 +44,34 @@ def __init__(
                 model will be created using the model_args.
path: src/langcheck/metrics/eval_clients/_gemini.py
user: login: liwii
body: Could you update the docstring about `model` and `model_args` as well?
created_at: 2025-03-26T05:39:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -23,12 +21,11 @@ class GeminiEvalClient(EvalClient):
 
     def __init__(
         self,
-        model: GenerativeModel | None = None,
-        model_args: dict[str, Any] | None = None,
+        model_name: str = "gemini-1.5-flash",
         generate_content_args: dict[str, Any] | None = None,
         embed_model_name: str | None = None,
         *,
-        system_prompt: str | None = None,
+        system_instruction: str | None = None,
path: src/langcheck/metrics/eval_clients/_gemini.py
user: login: liwii
body: I used `system_prompt` as the argument name for the consistency with other eval clients such as `OpenAI` or `Anthropic.

I know there are some parameter names already inconsistent, but I want to keep the named parameters after `*` as consistent as possible...

WDYT? If you think `system_instruction` is still better it is fine, but in that case could you insert a specific error message for `system_prompt` to notify the interface change?
created_at: 2025-03-26T05:46:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -23,12 +21,11 @@ class GeminiEvalClient(EvalClient):
 
     def __init__(
         self,
-        model: GenerativeModel | None = None,
-        model_args: dict[str, Any] | None = None,
+        model_name: str = "gemini-1.5-flash",
         generate_content_args: dict[str, Any] | None = None,
         embed_model_name: str | None = None,
         *,
-        system_prompt: str | None = None,
+        system_instruction: str | None = None,
     ):
         """
         Initialize the Gemini evaluation client. The authentication
path: src/langcheck/metrics/eval_clients/_gemini.py
user: login: liwii
body: The TODO below mentions the issue of `google-generativeai`'s async API.

Can you check if the problem persists in `google-genai`? It's okay not to support `use_async` right now, but could you at least replace the link and make a GitHub issue?
created_at: 2025-03-26T05:48:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -47,35 +44,34 @@ def __init__(
                 model will be created using the model_args.
             model_args: (Optional) Dict of args to create the Gemini model.
             generate_content_args: (Optional) Dict of args to pass in to the
-                ``generate_content`` function.
+                ``generate_content`` function. The keys should be the same as
+                the keys in the ``genai.types.GenerateContentConfig`` type.
             embed_model_name: (Optional) The name of the embedding model to use.
                 If not provided, the models/embedding-001 model will be used.
-            system_prompt: (Optional) The system prompt to use. If not provided,
-                no system prompt will be used.
+            system_instruction: (Optional) The system instruction to use. If not
+                provided, no system instruction will be used.
         """
-        if model:
-            self._text_response_model = model
-            self._structured_assessment_model = model
-        else:
-            configure(api_key=os.getenv("GOOGLE_API_KEY"))
-            model_args = model_args or {}
-            self._structured_assessment_model = GenerativeModel(**model_args)
-            # Only add system prompt to the text response model if it is provided
-            if system_prompt:
-                if "system_instruction" in model_args:
-                    warnings.warn(
-                        '"system_instruction" of model_args will be ignored because '
-                        "system_prompt is provided."
-                    )
-                model_args["system_instruction"] = system_prompt
-            self._text_response_model = GenerativeModel(**model_args)
-
+        self.client = genai.Client(api_key=os.getenv("GOOGLE_API_KEY"))
path: src/langcheck/metrics/eval_clients/_gemini.py
user: login: liwii
body: Seemingly the `genai.Client` object supports some parameters specific to VertexAI.
https://googleapis.github.io/python-genai/genai.html#genai.client.Client.vertexai

Is there any reason that you removed the `model_args` parameter? If not, shall we restore that to support the VertexAI API straightforwardly?
created_at: 2025-03-26T05:53:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -47,35 +44,34 @@ def __init__(
                 model will be created using the model_args.
             model_args: (Optional) Dict of args to create the Gemini model.
             generate_content_args: (Optional) Dict of args to pass in to the
-                ``generate_content`` function.
+                ``generate_content`` function. The keys should be the same as
+                the keys in the ``genai.types.GenerateContentConfig`` type.
             embed_model_name: (Optional) The name of the embedding model to use.
                 If not provided, the models/embedding-001 model will be used.
-            system_prompt: (Optional) The system prompt to use. If not provided,
-                no system prompt will be used.
+            system_instruction: (Optional) The system instruction to use. If not
+                provided, no system instruction will be used.
         """
-        if model:
-            self._text_response_model = model
-            self._structured_assessment_model = model
-        else:
-            configure(api_key=os.getenv("GOOGLE_API_KEY"))
-            model_args = model_args or {}
-            self._structured_assessment_model = GenerativeModel(**model_args)
-            # Only add system prompt to the text response model if it is provided
-            if system_prompt:
-                if "system_instruction" in model_args:
-                    warnings.warn(
-                        '"system_instruction" of model_args will be ignored because '
-                        "system_prompt is provided."
-                    )
-                model_args["system_instruction"] = system_prompt
-            self._text_response_model = GenerativeModel(**model_args)
-
+        self.client = genai.Client(api_key=os.getenv("GOOGLE_API_KEY"))
+        self._model_name = model_name
         self._generate_content_args = generate_content_args or {}
         self._embed_model_name = embed_model_name
+        self._system_instruction = system_instruction
+
+        self._validate_generate_content_config()
+
+    def _validate_generate_content_config(self) -> None:
+        if self._system_instruction:
+            try:
path: src/langcheck/metrics/eval_clients/_gemini.py
user: login: liwii
body: Maybe it is helpful to always validate the generate content config, no matter whether the `system_instruction` is given?
created_at: 2025-03-26T05:54:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -47,35 +44,34 @@ def __init__(
                 model will be created using the model_args.
             model_args: (Optional) Dict of args to create the Gemini model.
             generate_content_args: (Optional) Dict of args to pass in to the
-                ``generate_content`` function.
+                ``generate_content`` function. The keys should be the same as
+                the keys in the ``genai.types.GenerateContentConfig`` type.
             embed_model_name: (Optional) The name of the embedding model to use.
                 If not provided, the models/embedding-001 model will be used.
-            system_prompt: (Optional) The system prompt to use. If not provided,
-                no system prompt will be used.
+            system_instruction: (Optional) The system instruction to use. If not
+                provided, no system instruction will be used.
         """
-        if model:
-            self._text_response_model = model
-            self._structured_assessment_model = model
-        else:
-            configure(api_key=os.getenv("GOOGLE_API_KEY"))
-            model_args = model_args or {}
-            self._structured_assessment_model = GenerativeModel(**model_args)
-            # Only add system prompt to the text response model if it is provided
-            if system_prompt:
-                if "system_instruction" in model_args:
-                    warnings.warn(
-                        '"system_instruction" of model_args will be ignored because '
-                        "system_prompt is provided."
-                    )
-                model_args["system_instruction"] = system_prompt
-            self._text_response_model = GenerativeModel(**model_args)
-
+        self.client = genai.Client(api_key=os.getenv("GOOGLE_API_KEY"))
+        self._model_name = model_name
         self._generate_content_args = generate_content_args or {}
         self._embed_model_name = embed_model_name
+        self._system_instruction = system_instruction
+
+        self._validate_generate_content_config()
+
+    def _validate_generate_content_config(self) -> None:
+        if self._system_instruction:
+            try:
path: src/langcheck/metrics/eval_clients/_gemini.py
user: login: taniokay
body: True, fixed! [21b5c5d](https://github.com/citadel-ai/langcheck/pull/190/commits/21b5c5d28e1a2b4e63d3d56f88cd963cf00e7235)
created_at: 2025-03-26T14:14:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -23,12 +21,11 @@ class GeminiEvalClient(EvalClient):
 
     def __init__(
         self,
-        model: GenerativeModel | None = None,
-        model_args: dict[str, Any] | None = None,
+        model_name: str = "gemini-1.5-flash",
         generate_content_args: dict[str, Any] | None = None,
         embed_model_name: str | None = None,
         *,
-        system_prompt: str | None = None,
+        system_instruction: str | None = None,
     ):
         """
         Initialize the Gemini evaluation client. The authentication
path: src/langcheck/metrics/eval_clients/_gemini.py
user: login: taniokay
body: Created an issue #191 
The problem is gone now with `google-genai` so I just updated the comment here. I'll probably work on this too.
fixed in [21b5c5d](https://github.com/citadel-ai/langcheck/pull/190/commits/21b5c5d28e1a2b4e63d3d56f88cd963cf00e7235)
created_at: 2025-03-26T14:14:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -23,12 +21,11 @@ class GeminiEvalClient(EvalClient):
 
     def __init__(
         self,
-        model: GenerativeModel | None = None,
-        model_args: dict[str, Any] | None = None,
+        model_name: str = "gemini-1.5-flash",
         generate_content_args: dict[str, Any] | None = None,
         embed_model_name: str | None = None,
         *,
-        system_prompt: str | None = None,
+        system_instruction: str | None = None,
path: src/langcheck/metrics/eval_clients/_gemini.py
user: login: taniokay
body: I briefly took a look on several LLM SDKs and the namings across them aren't consistent, so now I agree with your opinion and I think it's more important keep consistency within langcheck. Let me revert the argument to `system_prompt`, but I left the private member as `self._system_instruction` following Gemini SDK. Fixed in [21b5c5d](https://github.com/citadel-ai/langcheck/pull/190/commits/21b5c5d28e1a2b4e63d3d56f88cd963cf00e7235)
created_at: 2025-03-26T14:15:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -47,35 +44,34 @@ def __init__(
                 model will be created using the model_args.
path: src/langcheck/metrics/eval_clients/_gemini.py
user: login: taniokay
body: updated! [21b5c5d](https://github.com/citadel-ai/langcheck/pull/190/commits/21b5c5d28e1a2b4e63d3d56f88cd963cf00e7235)
created_at: 2025-03-26T14:15:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -47,35 +44,34 @@ def __init__(
                 model will be created using the model_args.
             model_args: (Optional) Dict of args to create the Gemini model.
             generate_content_args: (Optional) Dict of args to pass in to the
-                ``generate_content`` function.
+                ``generate_content`` function. The keys should be the same as
+                the keys in the ``genai.types.GenerateContentConfig`` type.
             embed_model_name: (Optional) The name of the embedding model to use.
                 If not provided, the models/embedding-001 model will be used.
-            system_prompt: (Optional) The system prompt to use. If not provided,
-                no system prompt will be used.
+            system_instruction: (Optional) The system instruction to use. If not
+                provided, no system instruction will be used.
         """
-        if model:
-            self._text_response_model = model
-            self._structured_assessment_model = model
-        else:
-            configure(api_key=os.getenv("GOOGLE_API_KEY"))
-            model_args = model_args or {}
-            self._structured_assessment_model = GenerativeModel(**model_args)
-            # Only add system prompt to the text response model if it is provided
-            if system_prompt:
-                if "system_instruction" in model_args:
-                    warnings.warn(
-                        '"system_instruction" of model_args will be ignored because '
-                        "system_prompt is provided."
-                    )
-                model_args["system_instruction"] = system_prompt
-            self._text_response_model = GenerativeModel(**model_args)
-
+        self.client = genai.Client(api_key=os.getenv("GOOGLE_API_KEY"))
path: src/langcheck/metrics/eval_clients/_gemini.py
user: login: taniokay
body: We can use Vertex AI via env variables like the following:
```bash
# from https://cloud.google.com/vertex-ai/generative-ai/docs/start/quickstarts/quickstart-multimodal#gen-ai-sdk-for-python
# Replace the `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION` values
# with appropriate values for your project.
export GOOGLE_CLOUD_PROJECT=GOOGLE_CLOUD_PROJECT
export GOOGLE_CLOUD_LOCATION=us-central1
export GOOGLE_GENAI_USE_VERTEXAI=True
```
(I found that we didn't even need to explicitly pass API key to use Gemini Developer API so I removed it)

But is it better to support Vertex AI via explicit arguments in `GeminiEvalClient.__init__()`?
created_at: 2025-03-26T14:19:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -47,35 +44,34 @@ def __init__(
                 model will be created using the model_args.
             model_args: (Optional) Dict of args to create the Gemini model.
             generate_content_args: (Optional) Dict of args to pass in to the
-                ``generate_content`` function.
+                ``generate_content`` function. The keys should be the same as
+                the keys in the ``genai.types.GenerateContentConfig`` type.
             embed_model_name: (Optional) The name of the embedding model to use.
                 If not provided, the models/embedding-001 model will be used.
-            system_prompt: (Optional) The system prompt to use. If not provided,
-                no system prompt will be used.
+            system_instruction: (Optional) The system instruction to use. If not
+                provided, no system instruction will be used.
         """
-        if model:
-            self._text_response_model = model
-            self._structured_assessment_model = model
-        else:
-            configure(api_key=os.getenv("GOOGLE_API_KEY"))
-            model_args = model_args or {}
-            self._structured_assessment_model = GenerativeModel(**model_args)
-            # Only add system prompt to the text response model if it is provided
-            if system_prompt:
-                if "system_instruction" in model_args:
-                    warnings.warn(
-                        '"system_instruction" of model_args will be ignored because '
-                        "system_prompt is provided."
-                    )
-                model_args["system_instruction"] = system_prompt
-            self._text_response_model = GenerativeModel(**model_args)
-
+        self.client = genai.Client(api_key=os.getenv("GOOGLE_API_KEY"))
path: src/langcheck/metrics/eval_clients/_gemini.py
user: login: liwii
body: Okay I see, then let's go with the current settings for now!

Actually I haven't been able to call VertexAI through `genai` properly yet, so I'll make another PR later if I find out something works better with explicit named arguments!
created_at: 2025-03-27T07:30:39Z
author_association: CONTRIBUTOR

PR comments:
user: login: taniokay
created_at: 2025-03-25T09:58:26Z
body: @liwii This is the follow-up of #186, could you take a look?

author_association: CONTRIBUTOR
user: login: taniokay
created_at: 2025-03-25T12:51:32Z
body: Seemingly Gemini Developer API (current impl in `_gemini.py`) and Gemini API on Vertex AI are integrated into the same SDK so I think this PR can include Vertex AI too: https://github.com/googleapis/python-genai
author_association: CONTRIBUTOR
user: login: liwii
created_at: 2025-03-26T05:38:03Z
body: Let me take a look now 👀 
author_association: CONTRIBUTOR
user: login: taniokay
created_at: 2025-03-27T09:43:51Z
body: > There seemed to be type errors not related to this PR because of the recent update of pyright & sentence-transformer, so I directly pushed fix.

Thank you for your review and fix!
author_association: CONTRIBUTOR