Repository: zoedsoupe/mentor
Pull request #9: feat: Gemini adapter
Author: login: yoschihirokaimoto
Created: 2025-04-15T19:16:23Z | Merged: 2025-04-16T13:20:37Z

## Problem

Allow different models to be used in the framework.

## Solution

Creating an adapter for google gemini models.

## Rationale

Following the already existing adapter structure as possible.


Diff:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3177e21..2cf1e7b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,22 @@
 
 All notable changes to this project will be documented in this file.
 
+## [Unreleased]
+
+### Added
+- Added Google Gemini adapter for structured output support
+
+## [0.2.2] - 2025-04-14
+
+### Changes
+- Minor bug fixes and improvements
+
+## [0.2.1] - 2025-02-15
+
+### Fixed
+- Fixed minor bugs in OpenAI adapter
+- Improved error handling
+
 ## [0.2.0] - 2025-01-28
 
 ### Added
diff --git a/README.md b/README.md
index f99aed7..235af4d 100644
--- a/README.md
+++ b/README.md
@@ -25,12 +25,12 @@ end
 ```
 
 ## Usage
- 
+
 The `mentor` library is useful for coaxing an LLM to return JSON that maps to a schema that you provide, rather than the default unstructured text output. If you define your own validation logic, `mentor` can automatically retry prompts when validation fails (returning natural language error messages to the LLM, to guide it when making corrections).
 
 `mentor` is designed to be used with a variaty of LLM providers like [OpenAI API](https://platform.openai.com/docs/api-reference/chat-completions/create), [llama.cpp](https://github.com/ggerganov/llama.cpp), [Bumblebee](https://github.com/elixir-nx/bumblebee) and so on (check the [LLM.Adapters](#llm-adapters) section) by using an extendable adapter behavior.
 
-At its simplest, usage with `Ecto` is pretty straightforward: 
+At its simplest, usage with `Ecto` is pretty straightforward:
 
 1. Create an `Ecto` schema, with a native `@moduledoc` string that explains the schema definition to the LLM or define a `llm_description/0` callback to return it.
 2. Use the `Mentor.Ecto.Schema` macro to validate and fetch the documentation prompt and enforce callbacks.
@@ -103,6 +103,7 @@ Check out our [Quickstart Guide](https://hexdocs.pm/mentor/quickstart.html) for
 - [OpenAI structured outputs docs](https://platform.openai.com/docs/guides/structured-outputs)
 - [OpenAI input formatting cookbook](https://cookbook.openai.com/examples/how_to_format_inputs_to_chatgpt_models)
 - [Anthropic Message format](https://docs.anthropic.com/en/api/messages#body-messages)
+- [Google Gemini structured outputs docs](https://ai.google.dev/gemini-api/docs/structured-output?lang=rest)
 
 ## Spiritual inspirations
 - [instructor_ex](https://hexdocs.pm/instructor)
diff --git a/lib/mentor.ex b/lib/mentor.ex
index 9e0cdb7..eb4baf2 100644
--- a/lib/mentor.ex
+++ b/lib/mentor.ex
@@ -40,6 +40,7 @@ defmodule Mentor do
   alias Mentor.Ecto, as: MentorEcto
   alias Mentor.HTTPClient.Finch
   alias Mentor.LLM.Adapter
+  alias Mentor.LLM.Adapters.Gemini
   alias Mentor.LLM.Adapters.OpenAI
 
   @type message :: %{role: String.t(), content: term}
@@ -93,7 +94,7 @@ defmodule Mentor do
     ]
   ]
 
-  defguard is_llm_adapter(llm) when llm in [OpenAI] or is_atom(llm)
+  defguard is_llm_adapter(llm) when llm in [OpenAI, Gemini] or is_atom(llm)
 
   @initial_prompt """
   You are a highly intelligent and skilled assistant. Your task is to analyze and understand the content provided, then generate well-structured outputs that adhere to the constraints and requirements specified in the subsequent instructions. Your responses must be accurate, concise, and match the intended structure or purpose.
diff --git a/lib/mentor/llm/adapters/gemini.ex b/lib/mentor/llm/adapters/gemini.ex
new file mode 100644
index 0000000..8cc79eb
--- /dev/null
+++ b/lib/mentor/llm/adapters/gemini.ex
@@ -0,0 +1,204 @@
+defmodule Mentor.LLM.Adapters.Gemini do
+  use Mentor.LLM.Adapter
+
+  @known_models ~w[gemini-2.0-pro gemini-2.0-pro-latest gemini-2.0-pro-vision gemini-2.0-flash gemini-2.0-flash-lite gemini-2.0-flash-latest gemini-2.0-flash-vision gemini-1.5-pro gemini-1.5-pro-latest gemini-1.5-flash]
+
+  @options NimbleOptions.new!(
+             url: [
+               type: :string,
+               default: "https://generativelanguage.googleapis.com/v1beta/models",
+               doc: "Base API endpoint to use for sending requests"
+             ],
+             api_key: [
+               type: :string,
+               required: true,
+               doc: "Google Generative AI API key"
+             ],
+             model: [
+               type: {:or, [:string, {:in, @known_models}]},
+               required: true,
+               doc:
+                 "The Gemini model to query on, known models are: `#{inspect(@known_models, pretty: true)}`"
+             ],
+             temperature: [
+               type: :float,
+               default: 1.0,
+               doc:
+                 "What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic."
+             ],
+             http_options: [
+               type: :keyword_list,
+               default: [],
+               keys: [
+                 pool_timeout: [type: :integer],
+                 receive_timeout: [type: :integer],
+                 request_timeout: [type: :integer]
+               ]
+             ]
+           )
+
+  @moduledoc """
+  An adapter for integrating Google's Gemini language models with the Mentor framework.
+
+  This module implements the `Mentor.LLM.Adapter` behaviour, enabling communication between Mentor and Google Generative AI API. It facilitates sending prompts and receiving responses, ensuring compatibility with Mentor's expected data structures.
+
+  ## Options
+
+  #{NimbleOptions.docs(@options)}
+
+  ## Usage
+
+  To utilize this adapter, configure your `Mentor` instance with the appropriate options:
+
+      config = [
+        api_key: System.get_env("GEMINI_API_KEY"),
+        model: "gemini-2.0-pro"
+      ]
+
+      mentor = Mentor.start_chat_with!(Mentor.LLM.Adapters.Gemini, adapter_config: config)
+
+  ## Multimodal Support
+
+  The adapter supports multimodal inputs using the Gemini API format directly:
+
+      Mentor.append_message(%{
+        role: "user",
+        content: [
+          %{
+            type: "text",
+            text: "Extract information from this image."
+          },
+          %{
+            type: "image_base64",
+            data: "base64_encoded_data",
+            mime_type: "image/jpeg"
+          }
+        ]
+      })
+
+  The adapter passes these formats directly to Gemini with minimal transformation, allowing you to
+  use any content format supported by the Gemini API.
+
+  ## Considerations
+
+  - **API Key Security**: Ensure your Google API key is stored securely and not exposed in your codebase.
+  - **Model Availability**: Verify that the specified model is available and suitable for your use case. Refer to Google's official documentation for the most up-to-date list of models and their capabilities.
+  - **Vision Models**: For image processing, use vision-capable models.
+  - **Error Handling**: The `complete/1` function returns `{:ok, response}` on success or `{:error, reason}` on failure. Implement appropriate error handling in your application to manage these scenarios.
+  """
+
+  @keys Keyword.keys(@options.schema)
+
+  @impl true
+  def complete(%Mentor{config: config} = mentor) do
+    config = Keyword.take(config, @keys)
+
+    with {:ok, config} <- NimbleOptions.validate(config, @options),
+         {:ok, resp} <- make_gemini_request(mentor, config) do
+      if resp.status == 200 do
+        JSON.decode!(resp.body)
+        |> parse_response_body(mentor.__schema__)
+      else
+        {:error, resp}
+      end
+    end
+  end
+
+  defp make_gemini_request(%Mentor{} = mentor, config) do
+    body = make_gemini_body(mentor, config)
+    model = config[:model]
+    url = "#{config[:url]}/#{model}:generateContent?key=#{config[:api_key]}"
+
+    headers = [
+      {"content-type", "application/json"},
+      {"accept", "application/json"}
+    ]
+
+    mentor.http_client.request(url, body, headers, config[:http_options])
+  end
+
+  defp make_gemini_body(%Mentor{} = mentor, config) do
+    {system_instruction, regular_messages} = extract_system_instruction(mentor.messages)
+
+    contents =
+      Enum.map(regular_messages, fn message ->
+        %{
+          role: message.role,
+          parts: convert_message_content(message.content)
+        }
+      end)
+
+    schema_without_additional_props = Map.delete(mentor.json_schema, :additionalProperties)
+
+    body = %{
+      contents: contents,
+      generationConfig: %{
+        temperature: config[:temperature],
+        response_mime_type: "application/json",
+        response_schema: schema_without_additional_props
+      }
+    }
+
+    if system_instruction do
+      Map.put(body, :system_instruction, system_instruction)
+    else
+      body
+    end
+  end
+
+  defp extract_system_instruction(messages) do
+    system_message = Enum.find(messages, fn msg -> msg.role == "system" end)
+    regular_messages = Enum.reject(messages, fn msg -> msg.role == "system" end)
+
+    system_instruction =
+      case system_message do
+        %{content: content} when is_binary(content) ->
+          %{parts: [%{text: content}]}
+
+        %{content: content} when is_list(content) ->
+          %{parts: convert_message_content(content)}
+
+        nil ->
+          nil
+      end
+
+    {system_instruction, regular_messages}
+  end
+
+  defp convert_message_content(content) when is_binary(content) do
+    [%{text: content}]
+  end
+
+  defp convert_message_content(content) when is_list(content) do
+    Enum.map(content, fn
+      %{type: "text", text: text} ->
+        %{text: text}
+
+      text when is_binary(text) ->
+        %{text: text}
+
+      %{type: "image_base64", data: data, mime_type: mime_type} ->
+        %{inline_data: %{data: data, mime_type: mime_type}}
+
+      other ->
+        other
+    end)
+  end
+
+  defp parse_response_body(
+         %{"candidates" => [%{"content" => %{"parts" => [%{"text" => content} | _]}} | _]},
+         _schema
+       ) do
+    case JSON.decode(content) do
+      {:ok, parsed} when is_map(parsed) ->
+        {:ok, parsed}
+
+      {:error, _} ->
+        {:error, "Failed to extract JSON from Gemini response"}
+    end
+  end
+
+  defp parse_response_body(response, _schema) do
+    {:error, "Invalid response format from Gemini API: #{inspect(response)}"}
+  end
+end
diff --git a/test/mentor_test.exs b/test/mentor_test.exs
index a01a554..5e8d6af 100644
--- a/test/mentor_test.exs
+++ b/test/mentor_test.exs
@@ -4,6 +4,7 @@ defmodule MentorTest do
   import Mox
 
   alias Mentor.LLM.Adapters.OpenAI
+  alias Mentor.LLM.Adapters.Gemini
 
   Code.compile_file(Path.expand("./evaluation/schema.ex"))
 
@@ -18,10 +19,16 @@ defmodule MentorTest do
        |> Mentor.configure_adapter(api_key: "hehe", model: "gpt-4o-mini")
        |> Mentor.configure_http_client(@mock)
        |> Mentor.define_max_retries(0)
+       |> Mentor.configure_backoff(max_backoff: 1, base_backoff: 1),
+     gemini_mentor:
+       Mentor.start_chat_with!(Gemini, schema: Schema)
+       |> Mentor.configure_adapter(api_key: "test_key", model: "gemini-2.0-flash-lite")
+       |> Mentor.configure_http_client(@mock)
+       |> Mentor.define_max_retries(0)
        |> Mentor.configure_backoff(max_backoff: 1, base_backoff: 1)}
   end
 
-  test "start_chat_with!/2 initializes Mentor struct correctly" do
+  test "start_chat_with!/2 initializes Mentor struct correctly with OpenAI" do
     schema = Schema
     adapter = OpenAI
     config = [model: "gpt-4", api_key: "test_key"]
@@ -39,7 +46,25 @@ defmodule MentorTest do
            } = mentor
   end
 
-  test "append_message/2 adds a new message to the messages list", %{mentor: mentor} do
+  test "start_chat_with!/2 initializes Mentor struct correctly with Gemini" do
+    schema = Schema
+    adapter = Gemini
+    config = [model: "gemini-2.0-flash-lite", api_key: "test_key"]
+
+    mentor =
+      Mentor.start_chat_with!(adapter, schema: schema)
+      |> Mentor.configure_adapter(config)
+
+    assert %Mentor{
+             __schema__: ^schema,
+             adapter: ^adapter,
+             config: ^config,
+             max_retries: 3,
+             debug: false
+           } = mentor
+  end
+
+  test "append_message/2 adds a new message to the messages list with OpenAI", %{mentor: mentor} do
     message = %{role: "user", content: "Hello, assistant!"}
 
     updated_mentor = Mentor.append_message(mentor, message)
@@ -47,6 +72,44 @@ defmodule MentorTest do
     assert [%{role: "user", content: "Hello, assistant!"}] = updated_mentor.messages
   end
 
+  test "append_message/2 adds multimodal content to the messages list with Gemini", %{
+    gemini_mentor: mentor
+  } do
+    message = %{
+      role: "user",
+      content: [
+        %{
+          type: "text",
+          text: "Extract information from this image."
+        },
+        %{
+          type: "image_base64",
+          data: "base64_data",
+          mime_type: "image/jpeg"
+        }
+      ]
+    }
+
+    updated_mentor = Mentor.append_message(mentor, message)
+
+    assert [
+             %{
+               role: "user",
+               content: [
+                 %{
+                   type: "text",
+                   text: "Extract information from this image."
+                 },
+                 %{
+                   type: "image_base64",
+                   data: "base64_data",
+                   mime_type: "image/jpeg"
+                 }
+               ]
+             }
+           ] = updated_mentor.messages
+  end
+
   describe "complete/1" do
     test "processes successful response correctly", %{mentor: mentor} do
       expect(@mock, :request, fn _url, _body, _headers, _opts ->
@@ -70,6 +133,32 @@ defmodule MentorTest do
                |> Mentor.complete()
     end
 
+    test "processes successful Gemini response correctly", %{gemini_mentor: mentor} do
+      expect(@mock, :request, fn _url, _body, _headers, _opts ->
+        body =
+          JSON.encode!(%{
+            candidates: [
+              %{
+                content: %{
+                  parts: [
+                    %{
+                      text: ~s|{"name": "Gemini", "age": 25}|
+                    }
+                  ]
+                }
+              }
+            ]
+          })
+
+        {:ok, %Finch.Response{status: 200, headers: [], body: body}}
+      end)
+
+      assert {:ok, %Schema{name: "Gemini", age: 25}} =
+               mentor
+               |> Mentor.append_message(%{role: "user", content: "Generate a response."})
+               |> Mentor.complete()
+    end
+
     test "retries on validation error", %{mentor: mentor} do
       @mock
       |> expect(:request, fn _url, _body, _headers, _opts ->
@@ -120,5 +209,68 @@ defmodule MentorTest do
                |> Mentor.append_message(%{role: "user", content: "Generate a response."})
                |> Mentor.complete()
     end
+
+    test "retries on validation error with Gemini", %{gemini_mentor: mentor} do
+      @mock
+      |> expect(:request, fn _url, _body, _headers, _opts ->
+        body =
+          JSON.encode!(%{
+            candidates: [
+              %{
+                content: %{
+                  parts: [
+                    %{
+                      text: ~s|{"name": 123, "age": "invalid"}|
+                    }
+                  ]
+                }
+              }
+            ]
+          })
+
+        {:ok, %Finch.Response{status: 200, headers: [], body: body}}
+      end)
+      |> expect(:request, fn _url, _body, _headers, _opts ->
+        body =
+          JSON.encode!(%{
+            candidates: [
+              %{
+                content: %{
+                  parts: [
+                    %{
+                      text: ~s|{"name": "Gemini", "age": 101}|
+                    }
+                  ]
+                }
+              }
+            ]
+          })
+
+        {:ok, %Finch.Response{status: 200, headers: [], body: body}}
+      end)
+      |> expect(:request, fn _url, _body, _headers, _opts ->
+        body =
+          JSON.encode!(%{
+            candidates: [
+              %{
+                content: %{
+                  parts: [
+                    %{
+                      text: ~s|{"name": "Gemini", "age": 30}|
+                    }
+                  ]
+                }
+              }
+            ]
+          })
+
+        {:ok, %Finch.Response{status: 200, headers: [], body: body}}
+      end)
+
+      assert {:ok, %Schema{name: "Gemini", age: 30}} =
+               mentor
+               |> Mentor.append_message(%{role: "user", content: "Generate a response."})
+               |> Mentor.complete()
+    end
   end
 end


Reviews:
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR adds support for Google Gemini models by incorporating a Gemini adapter into the framework. Key changes include:
- Adding a quickstart example for the Gemini adapter in pages/quickstart.livemd.
- Updating the README.md with usage examples and documentation for both OpenAI and Gemini adapters.
- Modifying the CHANGELOG.md to document the new Gemini adapter features.

### Reviewed Changes

Copilot reviewed 3 out of 5 changed files in this pull request and generated no comments.

| File                        | Description                                             |
| --------------------------- | ------------------------------------------------------- |
| pages/quickstart.livemd     | Added a code snippet to demonstrate using the Gemini adapter. |
| README.md                   | Introduced a new section for LLM Adapters with examples for Gemini and OpenAI. |
| CHANGELOG.md                | Updated changelog to include new features for Gemini adapter. |

<details>
<summary>Files not reviewed (2)</summary>

* **lib/mentor.ex**: Language not supported
* **lib/mentor/llm/adapters/gemini.ex**: Language not supported
</details>

<details>
<summary>Comments suppressed due to low confidence (1)</summary>

**README.md:125**
* The API key environment variable for the Gemini adapter is inconsistent with the one used in the quickstart example (LB_GEMINI_API_KEY). Consider using a consistent naming convention across all examples.
```
  api_key: System.fetch_env!("GEMINI_API_KEY"),
```
</details>


state: COMMENTED
author_association: NONE
submitted_at: 2025-04-15T20:02:38Z
user: login: zoedsoupe
state: CHANGES_REQUESTED
author_association: OWNER
submitted_at: 2025-04-15T20:27:05Z
user: login: yoschihirokaimoto
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T20:30:25Z
user: login: yoschihirokaimoto
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T20:50:53Z
user: login: yoschihirokaimoto
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T23:34:50Z
user: login: yoschihirokaimoto
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T23:35:07Z
user: login: zoedsoupe
body: thanks for the contribution 💜
state: APPROVED
author_association: OWNER
submitted_at: 2025-04-16T13:10:31Z

Review comments:
diff_hunk: @@ -0,0 +1,280 @@
+defmodule Mentor.LLM.Adapters.Gemini do
+  use Mentor.LLM.Adapter
+
+  @known_models ~w[gemini-2.0-pro gemini-2.0-pro-latest gemini-2.0-pro-vision gemini-2.0-flash gemini-2.0-flash-lite gemini-2.0-flash-latest gemini-2.0-flash-vision gemini-1.5-pro gemini-1.5-pro-latest gemini-1.5-flash]
+
+  # Example base64 image used in documentation
+  @doc_b64_image "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="
+
+  @options NimbleOptions.new!(
+             url: [
+               type: :string,
+               default: "https://generativelanguage.googleapis.com/v1beta/models",
+               doc: "Base API endpoint to use for sending requests"
+             ],
+             api_key: [
+               type: :string,
+               required: true,
+               doc: "Google Generative AI API key"
+             ],
+             model: [
+               type: {:or, [:string, {:in, @known_models}]},
+               required: true,
+               doc:
+                 "The Gemini model to query on, known models are: `#{inspect(@known_models, pretty: true)}`"
+             ],
+             temperature: [
+               type: :float,
+               default: 1.0,
+               doc:
+                 "What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic."
+             ],
+             http_options: [
+               type: :keyword_list,
+               default: [],
+               keys: [
+                 pool_timeout: [type: :integer],
+                 receive_timeout: [type: :integer],
+                 request_timeout: [type: :integer]
+               ]
+             ]
+           )
+
+  @moduledoc """
+  An adapter for integrating Google's Gemini language models with the Mentor framework.
+
+  This module implements the `Mentor.LLM.Adapter` behaviour, enabling communication between Mentor and Google Generative AI API. It facilitates sending prompts and receiving responses, ensuring compatibility with Mentor's expected data structures.
+
+  ## Options
+
+  #{NimbleOptions.docs(@options)}
+
+  ## Usage
+
+  To utilize this adapter, configure your `Mentor` instance with the appropriate options:
+
+      config = [
+        api_key: System.get_env("GEMINI_API_KEY"),
+        model: "gemini-2.0-pro"
+      ]
+
+      mentor = Mentor.start_chat_with!(Mentor.LLM.Adapters.Gemini, adapter_config: config)
+
+  ## Multimodal Support
+
+  The adapter supports multimodal inputs like images. You can use various image format input styles:
+
+  ### Using Base64 Data URLs (OpenAI style):
+
+      # Using the sample base64 image (1x1 transparent pixel)
+      b64_encoded_image = @doc_b64_image
+
+      Mentor.append_message(%{
+        role: "user",
+        content: [
+          %{
+            type: "image_url",
+            image_url: %{
+              url: "data:image/jpeg;base64,#{@doc_b64_image}"
+            }
+          }
+        ]
+      })
+
+  ### Using a Direct URL:
+
+      Mentor.append_message(%{
+        role: "user",
+        content: [
+          %{
+            type: "image_url",
+            image_url: %{
+              url: "https://example.com/image.jpg"
+            }
+          }
+        ]
+      })
+
+  ### Gemini Direct Format:
+
+      Mentor.append_message(%{
+        role: "user",
+        content: [
+          %{
+            type: "image_base64",
+            data: "base64_encoded_data",
+            mime_type: "image/jpeg"
+          }
+        ]
+      })
+
+  ## Considerations
+
+  - **API Key Security**: Ensure your Google API key is stored securely and not exposed in your codebase.
+  - **Model Availability**: Verify that the specified model is available and suitable for your use case. Refer to Google's official documentation for the most up-to-date list of models and their capabilities.
+  - **Vision Models**: For image processing, use vision-capable models.
+  - **Error Handling**: The `complete/1` function returns `{:ok, response}` on success or `{:error, reason}` on failure. Implement appropriate error handling in your application to manage these scenarios.
+  """
+
+  @keys Keyword.keys(@options.schema)
+
+  @impl true
+  def complete(%Mentor{config: config} = mentor) do
+    config = Keyword.take(config, @keys)
+
+    with {:ok, config} <- NimbleOptions.validate(config, @options),
+         {:ok, resp} <- make_gemini_request(mentor, config) do
+      if resp.status == 200 do
+        JSON.decode!(resp.body)
+        |> parse_response_body(mentor.__schema__)
+      else
+        {:error, resp}
+      end
+    end
+  end
+
+  defp make_gemini_request(%Mentor{} = mentor, config) do
+    body = make_gemini_body(mentor, config)
+    model = config[:model]
+    url = "#{config[:url]}/#{model}:generateContent?key=#{config[:api_key]}"
+
+    headers = [
+      {"content-type", "application/json"},
+      {"accept", "application/json"}
+    ]
+
+    mentor.http_client.request(url, body, headers, config[:http_options])
+  end
+
+  defp make_gemini_body(%Mentor{} = mentor, config) do
+    {system_instruction, regular_messages} = extract_system_instruction(mentor.messages)
+
+    contents =
+      Enum.map(regular_messages, fn message ->
+        %{
+          role: convert_role(message.role),
+          parts: convert_message_content(message.content)
+        }
+      end)
+
+    schema_without_additional_props = Map.delete(mentor.json_schema, :additionalProperties)
+
+    body = %{
+      contents: contents,
+      generationConfig: %{
+        temperature: config[:temperature],
+        response_mime_type: "application/json",
+        response_schema: schema_without_additional_props
+      }
+    }
+
+    if system_instruction do
+      Map.put(body, :system_instruction, system_instruction)
+    else
+      body
+    end
+  end
+
+  defp extract_system_instruction(messages) do
+    system_message = Enum.find(messages, fn msg -> msg.role == "system" end)
+    regular_messages = Enum.reject(messages, fn msg -> msg.role == "system" end)
+
+    system_instruction =
+      case system_message do
+        %{content: content} when is_binary(content) ->
+          %{parts: [%{text: content}]}
+
+        %{content: content} when is_list(content) ->
+          %{parts: convert_message_content(content)}
+
+        nil ->
+          nil
+      end
+
+    {system_instruction, regular_messages}
+  end
+
+  defp convert_message_content(content) when is_binary(content) do
+    [%{text: content}]
+  end
+
+  defp convert_message_content(content) when is_list(content) do
+    Enum.map(content, fn
+      %{type: "text", text: text} ->
+        %{text: text}
+
+      text when is_binary(text) ->
+        %{text: text}
+
+      %{type: "image_url", image_url: %{url: url} = image_url} ->
+        handle_image_url(url, image_url)
+
+      %{type: "image", image: %{} = image} ->
+        image
+
+      %{type: "image_base64", data: data, mime_type: mime_type} ->
+        %{inline_data: %{data: data, mime_type: mime_type}}
+
+      other ->
+        %{text: "Unsupported content type: #{inspect(other)}"}
+    end)
+  end
+
+  defp handle_image_url("data:" <> rest = _url, _image_url) do
+    # Handle base64 data URLs
+    case String.split(rest, ";base64,", parts: 2) do
+      [mime_type, base64_data] ->
+        %{inline_data: %{data: base64_data, mime_type: mime_type}}
+
+      _ ->
+        %{text: "Invalid base64 image data"}
+    end
+  end
+
+  defp handle_image_url("http://" <> _rest = url, _image_url) do
+    %{file_data: %{file_uri: url, mime_type: guess_mime_type(url)}}
+  end
+
+  defp handle_image_url("https://" <> _rest = url, _image_url) do
+    %{file_data: %{file_uri: url, mime_type: guess_mime_type(url)}}
+  end
+
+  defp handle_image_url(_url, _image_url) do
+    %{text: "Unsupported image URL format"}
+  end
+
+  defp guess_mime_type(url) do
+    case Path.extname(url) |> String.downcase() do
+      ".jpg" -> "image/jpeg"
+      ".jpeg" -> "image/jpeg"
+      ".png" -> "image/png"
+      ".gif" -> "image/gif"
+      ".webp" -> "image/webp"
+      ".svg" -> "image/svg+xml"
+      # Default assumption
+      _ -> "image/jpeg"
+    end
+  end
+
+  defp convert_role("user"), do: "user"
+  defp convert_role("assistant"), do: "model"
+  defp convert_role("system"), do: "user"
path: lib/mentor/llm/adapters/gemini.ex
user: login: zoedsoupe
body: didn't understand, gemini doesn't support `system` messages?
created_at: 2025-04-15T20:07:55Z
author_association: OWNER
path: pages/quickstart.livemd
user: login: zoedsoupe
body: I don't think we need to add a single alternative example for each adapter since the only thing that changes is the module to be used as adapter, so we can remove that change
created_at: 2025-04-15T20:08:48Z
author_association: OWNER
diff_hunk: @@ -0,0 +1,280 @@
+defmodule Mentor.LLM.Adapters.Gemini do
+  use Mentor.LLM.Adapter
+
+  @known_models ~w[gemini-2.0-pro gemini-2.0-pro-latest gemini-2.0-pro-vision gemini-2.0-flash gemini-2.0-flash-lite gemini-2.0-flash-latest gemini-2.0-flash-vision gemini-1.5-pro gemini-1.5-pro-latest gemini-1.5-flash]
+
+  # Example base64 image used in documentation
+  @doc_b64_image "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="
+
+  @options NimbleOptions.new!(
+             url: [
+               type: :string,
+               default: "https://generativelanguage.googleapis.com/v1beta/models",
+               doc: "Base API endpoint to use for sending requests"
+             ],
+             api_key: [
+               type: :string,
+               required: true,
+               doc: "Google Generative AI API key"
+             ],
+             model: [
+               type: {:or, [:string, {:in, @known_models}]},
+               required: true,
+               doc:
+                 "The Gemini model to query on, known models are: `#{inspect(@known_models, pretty: true)}`"
+             ],
+             temperature: [
+               type: :float,
+               default: 1.0,
+               doc:
+                 "What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic."
+             ],
+             http_options: [
+               type: :keyword_list,
+               default: [],
+               keys: [
+                 pool_timeout: [type: :integer],
+                 receive_timeout: [type: :integer],
+                 request_timeout: [type: :integer]
+               ]
+             ]
+           )
+
+  @moduledoc """
+  An adapter for integrating Google's Gemini language models with the Mentor framework.
+
+  This module implements the `Mentor.LLM.Adapter` behaviour, enabling communication between Mentor and Google Generative AI API. It facilitates sending prompts and receiving responses, ensuring compatibility with Mentor's expected data structures.
+
+  ## Options
+
+  #{NimbleOptions.docs(@options)}
+
+  ## Usage
+
+  To utilize this adapter, configure your `Mentor` instance with the appropriate options:
+
+      config = [
+        api_key: System.get_env("GEMINI_API_KEY"),
+        model: "gemini-2.0-pro"
+      ]
+
+      mentor = Mentor.start_chat_with!(Mentor.LLM.Adapters.Gemini, adapter_config: config)
+
+  ## Multimodal Support
+
+  The adapter supports multimodal inputs like images. You can use various image format input styles:
+
+  ### Using Base64 Data URLs (OpenAI style):
+
+      # Using the sample base64 image (1x1 transparent pixel)
+      b64_encoded_image = @doc_b64_image
+
+      Mentor.append_message(%{
+        role: "user",
+        content: [
+          %{
+            type: "image_url",
+            image_url: %{
+              url: "data:image/jpeg;base64,#{@doc_b64_image}"
+            }
+          }
+        ]
+      })
+
+  ### Using a Direct URL:
+
+      Mentor.append_message(%{
+        role: "user",
+        content: [
+          %{
+            type: "image_url",
+            image_url: %{
+              url: "https://example.com/image.jpg"
+            }
+          }
+        ]
+      })
+
+  ### Gemini Direct Format:
+
+      Mentor.append_message(%{
+        role: "user",
+        content: [
+          %{
+            type: "image_base64",
+            data: "base64_encoded_data",
+            mime_type: "image/jpeg"
+          }
+        ]
+      })
+
+  ## Considerations
+
+  - **API Key Security**: Ensure your Google API key is stored securely and not exposed in your codebase.
+  - **Model Availability**: Verify that the specified model is available and suitable for your use case. Refer to Google's official documentation for the most up-to-date list of models and their capabilities.
+  - **Vision Models**: For image processing, use vision-capable models.
+  - **Error Handling**: The `complete/1` function returns `{:ok, response}` on success or `{:error, reason}` on failure. Implement appropriate error handling in your application to manage these scenarios.
+  """
+
+  @keys Keyword.keys(@options.schema)
+
+  @impl true
+  def complete(%Mentor{config: config} = mentor) do
+    config = Keyword.take(config, @keys)
+
+    with {:ok, config} <- NimbleOptions.validate(config, @options),
+         {:ok, resp} <- make_gemini_request(mentor, config) do
+      if resp.status == 200 do
+        JSON.decode!(resp.body)
+        |> parse_response_body(mentor.__schema__)
+      else
+        {:error, resp}
+      end
+    end
+  end
+
+  defp make_gemini_request(%Mentor{} = mentor, config) do
+    body = make_gemini_body(mentor, config)
+    model = config[:model]
+    url = "#{config[:url]}/#{model}:generateContent?key=#{config[:api_key]}"
+
+    headers = [
+      {"content-type", "application/json"},
+      {"accept", "application/json"}
+    ]
+
+    mentor.http_client.request(url, body, headers, config[:http_options])
+  end
+
+  defp make_gemini_body(%Mentor{} = mentor, config) do
+    {system_instruction, regular_messages} = extract_system_instruction(mentor.messages)
+
+    contents =
+      Enum.map(regular_messages, fn message ->
+        %{
+          role: convert_role(message.role),
+          parts: convert_message_content(message.content)
+        }
+      end)
+
+    schema_without_additional_props = Map.delete(mentor.json_schema, :additionalProperties)
+
+    body = %{
+      contents: contents,
+      generationConfig: %{
+        temperature: config[:temperature],
+        response_mime_type: "application/json",
+        response_schema: schema_without_additional_props
+      }
+    }
+
+    if system_instruction do
+      Map.put(body, :system_instruction, system_instruction)
+    else
+      body
+    end
+  end
+
+  defp extract_system_instruction(messages) do
+    system_message = Enum.find(messages, fn msg -> msg.role == "system" end)
+    regular_messages = Enum.reject(messages, fn msg -> msg.role == "system" end)
+
+    system_instruction =
+      case system_message do
+        %{content: content} when is_binary(content) ->
+          %{parts: [%{text: content}]}
+
+        %{content: content} when is_list(content) ->
+          %{parts: convert_message_content(content)}
+
+        nil ->
+          nil
+      end
+
+    {system_instruction, regular_messages}
+  end
+
+  defp convert_message_content(content) when is_binary(content) do
+    [%{text: content}]
+  end
+
+  defp convert_message_content(content) when is_list(content) do
+    Enum.map(content, fn
+      %{type: "text", text: text} ->
+        %{text: text}
+
+      text when is_binary(text) ->
+        %{text: text}
+
+      %{type: "image_url", image_url: %{url: url} = image_url} ->
+        handle_image_url(url, image_url)
+
+      %{type: "image", image: %{} = image} ->
+        image
+
+      %{type: "image_base64", data: data, mime_type: mime_type} ->
+        %{inline_data: %{data: data, mime_type: mime_type}}
+
+      other ->
+        %{text: "Unsupported content type: #{inspect(other)}"}
+    end)
+  end
+
+  defp handle_image_url("data:" <> rest = _url, _image_url) do
+    # Handle base64 data URLs
+    case String.split(rest, ";base64,", parts: 2) do
+      [mime_type, base64_data] ->
+        %{inline_data: %{data: base64_data, mime_type: mime_type}}
+
+      _ ->
+        %{text: "Invalid base64 image data"}
+    end
+  end
+
+  defp handle_image_url("http://" <> _rest = url, _image_url) do
+    %{file_data: %{file_uri: url, mime_type: guess_mime_type(url)}}
+  end
+
+  defp handle_image_url("https://" <> _rest = url, _image_url) do
+    %{file_data: %{file_uri: url, mime_type: guess_mime_type(url)}}
+  end
+
+  defp handle_image_url(_url, _image_url) do
+    %{text: "Unsupported image URL format"}
+  end
+
+  defp guess_mime_type(url) do
+    case Path.extname(url) |> String.downcase() do
+      ".jpg" -> "image/jpeg"
+      ".jpeg" -> "image/jpeg"
+      ".png" -> "image/png"
+      ".gif" -> "image/gif"
+      ".webp" -> "image/webp"
+      ".svg" -> "image/svg+xml"
+      # Default assumption
+      _ -> "image/jpeg"
+    end
+  end
path: lib/mentor/llm/adapters/gemini.ex
user: login: zoedsoupe
body: if I correctly understood, you're trying to abstract the API to send image/multi modal messages, but, we really need that?

mentor aims to be a very minimal abstraction around structured outputs, so users can append the message "schema" the way it fits better the application, I don't think its valuable to try to guess and build a custom body for gemini if the final users can directly build it on their application, makes sense? does I correctly understood?
created_at: 2025-04-15T20:11:32Z
author_association: OWNER
diff_hunk: @@ -0,0 +1,280 @@
+defmodule Mentor.LLM.Adapters.Gemini do
+  use Mentor.LLM.Adapter
+
+  @known_models ~w[gemini-2.0-pro gemini-2.0-pro-latest gemini-2.0-pro-vision gemini-2.0-flash gemini-2.0-flash-lite gemini-2.0-flash-latest gemini-2.0-flash-vision gemini-1.5-pro gemini-1.5-pro-latest gemini-1.5-flash]
+
+  # Example base64 image used in documentation
+  @doc_b64_image "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="
+
+  @options NimbleOptions.new!(
+             url: [
+               type: :string,
+               default: "https://generativelanguage.googleapis.com/v1beta/models",
+               doc: "Base API endpoint to use for sending requests"
+             ],
+             api_key: [
+               type: :string,
+               required: true,
+               doc: "Google Generative AI API key"
+             ],
+             model: [
+               type: {:or, [:string, {:in, @known_models}]},
+               required: true,
+               doc:
+                 "The Gemini model to query on, known models are: `#{inspect(@known_models, pretty: true)}`"
+             ],
+             temperature: [
+               type: :float,
+               default: 1.0,
+               doc:
+                 "What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic."
+             ],
+             http_options: [
+               type: :keyword_list,
+               default: [],
+               keys: [
+                 pool_timeout: [type: :integer],
+                 receive_timeout: [type: :integer],
+                 request_timeout: [type: :integer]
+               ]
+             ]
+           )
+
+  @moduledoc """
+  An adapter for integrating Google's Gemini language models with the Mentor framework.
+
+  This module implements the `Mentor.LLM.Adapter` behaviour, enabling communication between Mentor and Google Generative AI API. It facilitates sending prompts and receiving responses, ensuring compatibility with Mentor's expected data structures.
+
+  ## Options
+
+  #{NimbleOptions.docs(@options)}
+
+  ## Usage
+
+  To utilize this adapter, configure your `Mentor` instance with the appropriate options:
+
+      config = [
+        api_key: System.get_env("GEMINI_API_KEY"),
+        model: "gemini-2.0-pro"
+      ]
+
+      mentor = Mentor.start_chat_with!(Mentor.LLM.Adapters.Gemini, adapter_config: config)
+
+  ## Multimodal Support
+
+  The adapter supports multimodal inputs like images. You can use various image format input styles:
+
+  ### Using Base64 Data URLs (OpenAI style):
+
+      # Using the sample base64 image (1x1 transparent pixel)
+      b64_encoded_image = @doc_b64_image
+
+      Mentor.append_message(%{
+        role: "user",
+        content: [
+          %{
+            type: "image_url",
+            image_url: %{
+              url: "data:image/jpeg;base64,#{@doc_b64_image}"
+            }
+          }
+        ]
+      })
+
+  ### Using a Direct URL:
+
+      Mentor.append_message(%{
+        role: "user",
+        content: [
+          %{
+            type: "image_url",
+            image_url: %{
+              url: "https://example.com/image.jpg"
+            }
+          }
+        ]
+      })
+
+  ### Gemini Direct Format:
+
+      Mentor.append_message(%{
+        role: "user",
+        content: [
+          %{
+            type: "image_base64",
+            data: "base64_encoded_data",
+            mime_type: "image/jpeg"
+          }
+        ]
+      })
+
+  ## Considerations
+
+  - **API Key Security**: Ensure your Google API key is stored securely and not exposed in your codebase.
+  - **Model Availability**: Verify that the specified model is available and suitable for your use case. Refer to Google's official documentation for the most up-to-date list of models and their capabilities.
+  - **Vision Models**: For image processing, use vision-capable models.
+  - **Error Handling**: The `complete/1` function returns `{:ok, response}` on success or `{:error, reason}` on failure. Implement appropriate error handling in your application to manage these scenarios.
+  """
+
+  @keys Keyword.keys(@options.schema)
+
+  @impl true
+  def complete(%Mentor{config: config} = mentor) do
+    config = Keyword.take(config, @keys)
+
+    with {:ok, config} <- NimbleOptions.validate(config, @options),
+         {:ok, resp} <- make_gemini_request(mentor, config) do
+      if resp.status == 200 do
+        JSON.decode!(resp.body)
+        |> parse_response_body(mentor.__schema__)
+      else
+        {:error, resp}
+      end
+    end
+  end
+
+  defp make_gemini_request(%Mentor{} = mentor, config) do
+    body = make_gemini_body(mentor, config)
+    model = config[:model]
+    url = "#{config[:url]}/#{model}:generateContent?key=#{config[:api_key]}"
+
+    headers = [
+      {"content-type", "application/json"},
+      {"accept", "application/json"}
+    ]
+
+    mentor.http_client.request(url, body, headers, config[:http_options])
+  end
+
+  defp make_gemini_body(%Mentor{} = mentor, config) do
+    {system_instruction, regular_messages} = extract_system_instruction(mentor.messages)
+
+    contents =
+      Enum.map(regular_messages, fn message ->
+        %{
+          role: convert_role(message.role),
+          parts: convert_message_content(message.content)
+        }
+      end)
+
+    schema_without_additional_props = Map.delete(mentor.json_schema, :additionalProperties)
+
+    body = %{
+      contents: contents,
+      generationConfig: %{
+        temperature: config[:temperature],
+        response_mime_type: "application/json",
+        response_schema: schema_without_additional_props
+      }
+    }
+
+    if system_instruction do
+      Map.put(body, :system_instruction, system_instruction)
+    else
+      body
+    end
+  end
+
+  defp extract_system_instruction(messages) do
+    system_message = Enum.find(messages, fn msg -> msg.role == "system" end)
+    regular_messages = Enum.reject(messages, fn msg -> msg.role == "system" end)
+
+    system_instruction =
+      case system_message do
+        %{content: content} when is_binary(content) ->
+          %{parts: [%{text: content}]}
+
+        %{content: content} when is_list(content) ->
+          %{parts: convert_message_content(content)}
+
+        nil ->
+          nil
+      end
+
+    {system_instruction, regular_messages}
+  end
path: lib/mentor/llm/adapters/gemini.ex
user: login: zoedsoupe
body: gemini only support a single system message? 
created_at: 2025-04-15T20:26:57Z
author_association: OWNER
diff_hunk: @@ -0,0 +1,280 @@
+defmodule Mentor.LLM.Adapters.Gemini do
+  use Mentor.LLM.Adapter
+
+  @known_models ~w[gemini-2.0-pro gemini-2.0-pro-latest gemini-2.0-pro-vision gemini-2.0-flash gemini-2.0-flash-lite gemini-2.0-flash-latest gemini-2.0-flash-vision gemini-1.5-pro gemini-1.5-pro-latest gemini-1.5-flash]
+
+  # Example base64 image used in documentation
+  @doc_b64_image "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="
+
+  @options NimbleOptions.new!(
+             url: [
+               type: :string,
+               default: "https://generativelanguage.googleapis.com/v1beta/models",
+               doc: "Base API endpoint to use for sending requests"
+             ],
+             api_key: [
+               type: :string,
+               required: true,
+               doc: "Google Generative AI API key"
+             ],
+             model: [
+               type: {:or, [:string, {:in, @known_models}]},
+               required: true,
+               doc:
+                 "The Gemini model to query on, known models are: `#{inspect(@known_models, pretty: true)}`"
+             ],
+             temperature: [
+               type: :float,
+               default: 1.0,
+               doc:
+                 "What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic."
+             ],
+             http_options: [
+               type: :keyword_list,
+               default: [],
+               keys: [
+                 pool_timeout: [type: :integer],
+                 receive_timeout: [type: :integer],
+                 request_timeout: [type: :integer]
+               ]
+             ]
+           )
+
+  @moduledoc """
+  An adapter for integrating Google's Gemini language models with the Mentor framework.
+
+  This module implements the `Mentor.LLM.Adapter` behaviour, enabling communication between Mentor and Google Generative AI API. It facilitates sending prompts and receiving responses, ensuring compatibility with Mentor's expected data structures.
+
+  ## Options
+
+  #{NimbleOptions.docs(@options)}
+
+  ## Usage
+
+  To utilize this adapter, configure your `Mentor` instance with the appropriate options:
+
+      config = [
+        api_key: System.get_env("GEMINI_API_KEY"),
+        model: "gemini-2.0-pro"
+      ]
+
+      mentor = Mentor.start_chat_with!(Mentor.LLM.Adapters.Gemini, adapter_config: config)
+
+  ## Multimodal Support
+
+  The adapter supports multimodal inputs like images. You can use various image format input styles:
+
+  ### Using Base64 Data URLs (OpenAI style):
+
+      # Using the sample base64 image (1x1 transparent pixel)
+      b64_encoded_image = @doc_b64_image
+
+      Mentor.append_message(%{
+        role: "user",
+        content: [
+          %{
+            type: "image_url",
+            image_url: %{
+              url: "data:image/jpeg;base64,#{@doc_b64_image}"
+            }
+          }
+        ]
+      })
+
+  ### Using a Direct URL:
+
+      Mentor.append_message(%{
+        role: "user",
+        content: [
+          %{
+            type: "image_url",
+            image_url: %{
+              url: "https://example.com/image.jpg"
+            }
+          }
+        ]
+      })
+
+  ### Gemini Direct Format:
+
+      Mentor.append_message(%{
+        role: "user",
+        content: [
+          %{
+            type: "image_base64",
+            data: "base64_encoded_data",
+            mime_type: "image/jpeg"
+          }
+        ]
+      })
+
+  ## Considerations
+
+  - **API Key Security**: Ensure your Google API key is stored securely and not exposed in your codebase.
+  - **Model Availability**: Verify that the specified model is available and suitable for your use case. Refer to Google's official documentation for the most up-to-date list of models and their capabilities.
+  - **Vision Models**: For image processing, use vision-capable models.
+  - **Error Handling**: The `complete/1` function returns `{:ok, response}` on success or `{:error, reason}` on failure. Implement appropriate error handling in your application to manage these scenarios.
+  """
+
+  @keys Keyword.keys(@options.schema)
+
+  @impl true
+  def complete(%Mentor{config: config} = mentor) do
+    config = Keyword.take(config, @keys)
+
+    with {:ok, config} <- NimbleOptions.validate(config, @options),
+         {:ok, resp} <- make_gemini_request(mentor, config) do
+      if resp.status == 200 do
+        JSON.decode!(resp.body)
+        |> parse_response_body(mentor.__schema__)
+      else
+        {:error, resp}
+      end
+    end
+  end
+
+  defp make_gemini_request(%Mentor{} = mentor, config) do
+    body = make_gemini_body(mentor, config)
+    model = config[:model]
+    url = "#{config[:url]}/#{model}:generateContent?key=#{config[:api_key]}"
+
+    headers = [
+      {"content-type", "application/json"},
+      {"accept", "application/json"}
+    ]
+
+    mentor.http_client.request(url, body, headers, config[:http_options])
+  end
+
+  defp make_gemini_body(%Mentor{} = mentor, config) do
+    {system_instruction, regular_messages} = extract_system_instruction(mentor.messages)
+
+    contents =
+      Enum.map(regular_messages, fn message ->
+        %{
+          role: convert_role(message.role),
+          parts: convert_message_content(message.content)
+        }
+      end)
+
+    schema_without_additional_props = Map.delete(mentor.json_schema, :additionalProperties)
+
+    body = %{
+      contents: contents,
+      generationConfig: %{
+        temperature: config[:temperature],
+        response_mime_type: "application/json",
+        response_schema: schema_without_additional_props
+      }
+    }
+
+    if system_instruction do
+      Map.put(body, :system_instruction, system_instruction)
+    else
+      body
+    end
+  end
+
+  defp extract_system_instruction(messages) do
+    system_message = Enum.find(messages, fn msg -> msg.role == "system" end)
+    regular_messages = Enum.reject(messages, fn msg -> msg.role == "system" end)
+
+    system_instruction =
+      case system_message do
+        %{content: content} when is_binary(content) ->
+          %{parts: [%{text: content}]}
+
+        %{content: content} when is_list(content) ->
+          %{parts: convert_message_content(content)}
+
+        nil ->
+          nil
+      end
+
+    {system_instruction, regular_messages}
+  end
path: lib/mentor/llm/adapters/gemini.ex
user: login: yoschihirokaimoto
body: Yes, similar to Anthropic
created_at: 2025-04-15T20:30:25Z
author_association: CONTRIBUTOR
path: pages/quickstart.livemd
user: login: yoschihirokaimoto
body: Makes sense
created_at: 2025-04-15T20:50:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,280 @@
+defmodule Mentor.LLM.Adapters.Gemini do
+  use Mentor.LLM.Adapter
+
+  @known_models ~w[gemini-2.0-pro gemini-2.0-pro-latest gemini-2.0-pro-vision gemini-2.0-flash gemini-2.0-flash-lite gemini-2.0-flash-latest gemini-2.0-flash-vision gemini-1.5-pro gemini-1.5-pro-latest gemini-1.5-flash]
+
+  # Example base64 image used in documentation
+  @doc_b64_image "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="
+
+  @options NimbleOptions.new!(
+             url: [
+               type: :string,
+               default: "https://generativelanguage.googleapis.com/v1beta/models",
+               doc: "Base API endpoint to use for sending requests"
+             ],
+             api_key: [
+               type: :string,
+               required: true,
+               doc: "Google Generative AI API key"
+             ],
+             model: [
+               type: {:or, [:string, {:in, @known_models}]},
+               required: true,
+               doc:
+                 "The Gemini model to query on, known models are: `#{inspect(@known_models, pretty: true)}`"
+             ],
+             temperature: [
+               type: :float,
+               default: 1.0,
+               doc:
+                 "What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic."
+             ],
+             http_options: [
+               type: :keyword_list,
+               default: [],
+               keys: [
+                 pool_timeout: [type: :integer],
+                 receive_timeout: [type: :integer],
+                 request_timeout: [type: :integer]
+               ]
+             ]
+           )
+
+  @moduledoc """
+  An adapter for integrating Google's Gemini language models with the Mentor framework.
+
+  This module implements the `Mentor.LLM.Adapter` behaviour, enabling communication between Mentor and Google Generative AI API. It facilitates sending prompts and receiving responses, ensuring compatibility with Mentor's expected data structures.
+
+  ## Options
+
+  #{NimbleOptions.docs(@options)}
+
+  ## Usage
+
+  To utilize this adapter, configure your `Mentor` instance with the appropriate options:
+
+      config = [
+        api_key: System.get_env("GEMINI_API_KEY"),
+        model: "gemini-2.0-pro"
+      ]
+
+      mentor = Mentor.start_chat_with!(Mentor.LLM.Adapters.Gemini, adapter_config: config)
+
+  ## Multimodal Support
+
+  The adapter supports multimodal inputs like images. You can use various image format input styles:
+
+  ### Using Base64 Data URLs (OpenAI style):
+
+      # Using the sample base64 image (1x1 transparent pixel)
+      b64_encoded_image = @doc_b64_image
+
+      Mentor.append_message(%{
+        role: "user",
+        content: [
+          %{
+            type: "image_url",
+            image_url: %{
+              url: "data:image/jpeg;base64,#{@doc_b64_image}"
+            }
+          }
+        ]
+      })
+
+  ### Using a Direct URL:
+
+      Mentor.append_message(%{
+        role: "user",
+        content: [
+          %{
+            type: "image_url",
+            image_url: %{
+              url: "https://example.com/image.jpg"
+            }
+          }
+        ]
+      })
+
+  ### Gemini Direct Format:
+
+      Mentor.append_message(%{
+        role: "user",
+        content: [
+          %{
+            type: "image_base64",
+            data: "base64_encoded_data",
+            mime_type: "image/jpeg"
+          }
+        ]
+      })
+
+  ## Considerations
+
+  - **API Key Security**: Ensure your Google API key is stored securely and not exposed in your codebase.
+  - **Model Availability**: Verify that the specified model is available and suitable for your use case. Refer to Google's official documentation for the most up-to-date list of models and their capabilities.
+  - **Vision Models**: For image processing, use vision-capable models.
+  - **Error Handling**: The `complete/1` function returns `{:ok, response}` on success or `{:error, reason}` on failure. Implement appropriate error handling in your application to manage these scenarios.
+  """
+
+  @keys Keyword.keys(@options.schema)
+
+  @impl true
+  def complete(%Mentor{config: config} = mentor) do
+    config = Keyword.take(config, @keys)
+
+    with {:ok, config} <- NimbleOptions.validate(config, @options),
+         {:ok, resp} <- make_gemini_request(mentor, config) do
+      if resp.status == 200 do
+        JSON.decode!(resp.body)
+        |> parse_response_body(mentor.__schema__)
+      else
+        {:error, resp}
+      end
+    end
+  end
+
+  defp make_gemini_request(%Mentor{} = mentor, config) do
+    body = make_gemini_body(mentor, config)
+    model = config[:model]
+    url = "#{config[:url]}/#{model}:generateContent?key=#{config[:api_key]}"
+
+    headers = [
+      {"content-type", "application/json"},
+      {"accept", "application/json"}
+    ]
+
+    mentor.http_client.request(url, body, headers, config[:http_options])
+  end
+
+  defp make_gemini_body(%Mentor{} = mentor, config) do
+    {system_instruction, regular_messages} = extract_system_instruction(mentor.messages)
+
+    contents =
+      Enum.map(regular_messages, fn message ->
+        %{
+          role: convert_role(message.role),
+          parts: convert_message_content(message.content)
+        }
+      end)
+
+    schema_without_additional_props = Map.delete(mentor.json_schema, :additionalProperties)
+
+    body = %{
+      contents: contents,
+      generationConfig: %{
+        temperature: config[:temperature],
+        response_mime_type: "application/json",
+        response_schema: schema_without_additional_props
+      }
+    }
+
+    if system_instruction do
+      Map.put(body, :system_instruction, system_instruction)
+    else
+      body
+    end
+  end
+
+  defp extract_system_instruction(messages) do
+    system_message = Enum.find(messages, fn msg -> msg.role == "system" end)
+    regular_messages = Enum.reject(messages, fn msg -> msg.role == "system" end)
+
+    system_instruction =
+      case system_message do
+        %{content: content} when is_binary(content) ->
+          %{parts: [%{text: content}]}
+
+        %{content: content} when is_list(content) ->
+          %{parts: convert_message_content(content)}
+
+        nil ->
+          nil
+      end
+
+    {system_instruction, regular_messages}
+  end
+
+  defp convert_message_content(content) when is_binary(content) do
+    [%{text: content}]
+  end
+
+  defp convert_message_content(content) when is_list(content) do
+    Enum.map(content, fn
+      %{type: "text", text: text} ->
+        %{text: text}
+
+      text when is_binary(text) ->
+        %{text: text}
+
+      %{type: "image_url", image_url: %{url: url} = image_url} ->
+        handle_image_url(url, image_url)
+
+      %{type: "image", image: %{} = image} ->
+        image
+
+      %{type: "image_base64", data: data, mime_type: mime_type} ->
+        %{inline_data: %{data: data, mime_type: mime_type}}
+
+      other ->
+        %{text: "Unsupported content type: #{inspect(other)}"}
+    end)
+  end
+
+  defp handle_image_url("data:" <> rest = _url, _image_url) do
+    # Handle base64 data URLs
+    case String.split(rest, ";base64,", parts: 2) do
+      [mime_type, base64_data] ->
+        %{inline_data: %{data: base64_data, mime_type: mime_type}}
+
+      _ ->
+        %{text: "Invalid base64 image data"}
+    end
+  end
+
+  defp handle_image_url("http://" <> _rest = url, _image_url) do
+    %{file_data: %{file_uri: url, mime_type: guess_mime_type(url)}}
+  end
+
+  defp handle_image_url("https://" <> _rest = url, _image_url) do
+    %{file_data: %{file_uri: url, mime_type: guess_mime_type(url)}}
+  end
+
+  defp handle_image_url(_url, _image_url) do
+    %{text: "Unsupported image URL format"}
+  end
+
+  defp guess_mime_type(url) do
+    case Path.extname(url) |> String.downcase() do
+      ".jpg" -> "image/jpeg"
+      ".jpeg" -> "image/jpeg"
+      ".png" -> "image/png"
+      ".gif" -> "image/gif"
+      ".webp" -> "image/webp"
+      ".svg" -> "image/svg+xml"
+      # Default assumption
+      _ -> "image/jpeg"
+    end
+  end
+
+  defp convert_role("user"), do: "user"
+  defp convert_role("assistant"), do: "model"
+  defp convert_role("system"), do: "user"
path: lib/mentor/llm/adapters/gemini.ex
user: login: yoschihirokaimoto
body: As we discussed, since we will accept messages directly on the usage, I've removed this
created_at: 2025-04-15T23:34:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,280 @@
+defmodule Mentor.LLM.Adapters.Gemini do
+  use Mentor.LLM.Adapter
+
+  @known_models ~w[gemini-2.0-pro gemini-2.0-pro-latest gemini-2.0-pro-vision gemini-2.0-flash gemini-2.0-flash-lite gemini-2.0-flash-latest gemini-2.0-flash-vision gemini-1.5-pro gemini-1.5-pro-latest gemini-1.5-flash]
+
+  # Example base64 image used in documentation
+  @doc_b64_image "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="
+
+  @options NimbleOptions.new!(
+             url: [
+               type: :string,
+               default: "https://generativelanguage.googleapis.com/v1beta/models",
+               doc: "Base API endpoint to use for sending requests"
+             ],
+             api_key: [
+               type: :string,
+               required: true,
+               doc: "Google Generative AI API key"
+             ],
+             model: [
+               type: {:or, [:string, {:in, @known_models}]},
+               required: true,
+               doc:
+                 "The Gemini model to query on, known models are: `#{inspect(@known_models, pretty: true)}`"
+             ],
+             temperature: [
+               type: :float,
+               default: 1.0,
+               doc:
+                 "What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic."
+             ],
+             http_options: [
+               type: :keyword_list,
+               default: [],
+               keys: [
+                 pool_timeout: [type: :integer],
+                 receive_timeout: [type: :integer],
+                 request_timeout: [type: :integer]
+               ]
+             ]
+           )
+
+  @moduledoc """
+  An adapter for integrating Google's Gemini language models with the Mentor framework.
+
+  This module implements the `Mentor.LLM.Adapter` behaviour, enabling communication between Mentor and Google Generative AI API. It facilitates sending prompts and receiving responses, ensuring compatibility with Mentor's expected data structures.
+
+  ## Options
+
+  #{NimbleOptions.docs(@options)}
+
+  ## Usage
+
+  To utilize this adapter, configure your `Mentor` instance with the appropriate options:
+
+      config = [
+        api_key: System.get_env("GEMINI_API_KEY"),
+        model: "gemini-2.0-pro"
+      ]
+
+      mentor = Mentor.start_chat_with!(Mentor.LLM.Adapters.Gemini, adapter_config: config)
+
+  ## Multimodal Support
+
+  The adapter supports multimodal inputs like images. You can use various image format input styles:
+
+  ### Using Base64 Data URLs (OpenAI style):
+
+      # Using the sample base64 image (1x1 transparent pixel)
+      b64_encoded_image = @doc_b64_image
+
+      Mentor.append_message(%{
+        role: "user",
+        content: [
+          %{
+            type: "image_url",
+            image_url: %{
+              url: "data:image/jpeg;base64,#{@doc_b64_image}"
+            }
+          }
+        ]
+      })
+
+  ### Using a Direct URL:
+
+      Mentor.append_message(%{
+        role: "user",
+        content: [
+          %{
+            type: "image_url",
+            image_url: %{
+              url: "https://example.com/image.jpg"
+            }
+          }
+        ]
+      })
+
+  ### Gemini Direct Format:
+
+      Mentor.append_message(%{
+        role: "user",
+        content: [
+          %{
+            type: "image_base64",
+            data: "base64_encoded_data",
+            mime_type: "image/jpeg"
+          }
+        ]
+      })
+
+  ## Considerations
+
+  - **API Key Security**: Ensure your Google API key is stored securely and not exposed in your codebase.
+  - **Model Availability**: Verify that the specified model is available and suitable for your use case. Refer to Google's official documentation for the most up-to-date list of models and their capabilities.
+  - **Vision Models**: For image processing, use vision-capable models.
+  - **Error Handling**: The `complete/1` function returns `{:ok, response}` on success or `{:error, reason}` on failure. Implement appropriate error handling in your application to manage these scenarios.
+  """
+
+  @keys Keyword.keys(@options.schema)
+
+  @impl true
+  def complete(%Mentor{config: config} = mentor) do
+    config = Keyword.take(config, @keys)
+
+    with {:ok, config} <- NimbleOptions.validate(config, @options),
+         {:ok, resp} <- make_gemini_request(mentor, config) do
+      if resp.status == 200 do
+        JSON.decode!(resp.body)
+        |> parse_response_body(mentor.__schema__)
+      else
+        {:error, resp}
+      end
+    end
+  end
+
+  defp make_gemini_request(%Mentor{} = mentor, config) do
+    body = make_gemini_body(mentor, config)
+    model = config[:model]
+    url = "#{config[:url]}/#{model}:generateContent?key=#{config[:api_key]}"
+
+    headers = [
+      {"content-type", "application/json"},
+      {"accept", "application/json"}
+    ]
+
+    mentor.http_client.request(url, body, headers, config[:http_options])
+  end
+
+  defp make_gemini_body(%Mentor{} = mentor, config) do
+    {system_instruction, regular_messages} = extract_system_instruction(mentor.messages)
+
+    contents =
+      Enum.map(regular_messages, fn message ->
+        %{
+          role: convert_role(message.role),
+          parts: convert_message_content(message.content)
+        }
+      end)
+
+    schema_without_additional_props = Map.delete(mentor.json_schema, :additionalProperties)
+
+    body = %{
+      contents: contents,
+      generationConfig: %{
+        temperature: config[:temperature],
+        response_mime_type: "application/json",
+        response_schema: schema_without_additional_props
+      }
+    }
+
+    if system_instruction do
+      Map.put(body, :system_instruction, system_instruction)
+    else
+      body
+    end
+  end
+
+  defp extract_system_instruction(messages) do
+    system_message = Enum.find(messages, fn msg -> msg.role == "system" end)
+    regular_messages = Enum.reject(messages, fn msg -> msg.role == "system" end)
+
+    system_instruction =
+      case system_message do
+        %{content: content} when is_binary(content) ->
+          %{parts: [%{text: content}]}
+
+        %{content: content} when is_list(content) ->
+          %{parts: convert_message_content(content)}
+
+        nil ->
+          nil
+      end
+
+    {system_instruction, regular_messages}
+  end
+
+  defp convert_message_content(content) when is_binary(content) do
+    [%{text: content}]
+  end
+
+  defp convert_message_content(content) when is_list(content) do
+    Enum.map(content, fn
+      %{type: "text", text: text} ->
+        %{text: text}
+
+      text when is_binary(text) ->
+        %{text: text}
+
+      %{type: "image_url", image_url: %{url: url} = image_url} ->
+        handle_image_url(url, image_url)
+
+      %{type: "image", image: %{} = image} ->
+        image
+
+      %{type: "image_base64", data: data, mime_type: mime_type} ->
+        %{inline_data: %{data: data, mime_type: mime_type}}
+
+      other ->
+        %{text: "Unsupported content type: #{inspect(other)}"}
+    end)
+  end
+
+  defp handle_image_url("data:" <> rest = _url, _image_url) do
+    # Handle base64 data URLs
+    case String.split(rest, ";base64,", parts: 2) do
+      [mime_type, base64_data] ->
+        %{inline_data: %{data: base64_data, mime_type: mime_type}}
+
+      _ ->
+        %{text: "Invalid base64 image data"}
+    end
+  end
+
+  defp handle_image_url("http://" <> _rest = url, _image_url) do
+    %{file_data: %{file_uri: url, mime_type: guess_mime_type(url)}}
+  end
+
+  defp handle_image_url("https://" <> _rest = url, _image_url) do
+    %{file_data: %{file_uri: url, mime_type: guess_mime_type(url)}}
+  end
+
+  defp handle_image_url(_url, _image_url) do
+    %{text: "Unsupported image URL format"}
+  end
+
+  defp guess_mime_type(url) do
+    case Path.extname(url) |> String.downcase() do
+      ".jpg" -> "image/jpeg"
+      ".jpeg" -> "image/jpeg"
+      ".png" -> "image/png"
+      ".gif" -> "image/gif"
+      ".webp" -> "image/webp"
+      ".svg" -> "image/svg+xml"
+      # Default assumption
+      _ -> "image/jpeg"
+    end
+  end
path: lib/mentor/llm/adapters/gemini.ex
user: login: yoschihirokaimoto
body: Makes sense.
created_at: 2025-04-15T23:35:06Z
author_association: CONTRIBUTOR

PR comments:
user: login: zoedsoupe
created_at: 2025-04-15T20:38:05Z
body: I also would like to add some simple tests just to validate these internal logic
author_association: OWNER
user: login: zoedsoupe
created_at: 2025-04-16T13:20:34Z
body: will fix the credo issue on `main`
author_association: OWNER