Repository: microsoft/autogen
Pull request: added Gemini safety setting and Gemini generation config
<!-- Thank you for your contribution! Please review https://microsoft.github.io/autogen/docs/Contribute before opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR. If you don't have the access to it, we will shortly find a reviewer and assign them to your PR. -->

## Why are these changes needed?

* Added support for safety_setting and generation_config for Gemini client

Hi! 
I have also taken into account the comments from this PR #2410

### Important notes:
1 - If no parameters are passed, then default values for safety and generation config of Gemini will be used, whatever it is under the hood of gemini itself.
2 - Safety settings will work only when api_type is google, if api_type is not passed it will throw an error.
3 - Added mapping for generation params names, so that Autogen users may not bother with different namings for different model types. For examples, I have seen "max_tokens" in Autogen, and Gemini uses "max_output_tokens"
4 - Safety settings works fine when passed as strings (opposed to class definitions). Also it is how it is show in official docs
5 - I did not include parameter "n" (candidate count) because gemini breaks for n > 1 right now


Tests: Additionally to the tests that check that safety setting and generation params work, I tested few scripts that use openai-models, and I believe it will not brake anything.

Example usage:
```
safety_settings = [
    {"category": "HARM_CATEGORY_HARASSMENT", "threshold": "BLOCK_ONLY_HIGH"},
    {"category": "HARM_CATEGORY_HATE_SPEECH", "threshold": "BLOCK_ONLY_HIGH"},
    {"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT", "threshold": "BLOCK_ONLY_HIGH"},
    {"category": "HARM_CATEGORY_DANGEROUS_CONTENT", "threshold": "BLOCK_ONLY_HIGH"},
]

llm_config_gemini = {
  "config_list": [{"model": "gemini-pro", "api_key": os.environ["GEMINI_API_KEY"], "api_type": "google",
    "top_p":0.1, "temperature":1, "top_k":1, "max_tokens":2048,
    "safety_settings":safety_settings}],
  }
```
### TODO
* Should I update documentation? which one? 

## Related issue number

Closes #2428, #2409

## Checks

- [ ] I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes introduced in this PR.
- [ ] I've made sure all auto checks have passed.





Problem statement:
[Feature Request]: add gemini safety settings and generation config
### Is your feature request related to a problem? Please describe.

1 - Cannot control behavior of Gemini with generation params like top_p, top_k etc. 
2 - Default safety setting blocks too often, I want to rise threshold to BLOCK_ONLY_HIGH

### Describe the solution you'd like

I want to specify those params in llm_config when creating an agent

### Additional context

_No response_

Hints from issue discussion:

Commit message: added Gemini safety setting and Gemini generation config

Patch to review:
diff --git a/autogen/oai/gemini.py b/autogen/oai/gemini.py
index fcf7e09c025d..2fd934cc42dc 100644
--- a/autogen/oai/gemini.py
+++ b/autogen/oai/gemini.py
@@ -5,8 +5,18 @@
     llm_config={
         "config_list": [{
             "api_type": "google",
-            "model": "models/gemini-pro",
-            "api_key": os.environ.get("GOOGLE_API_KEY")
+            "model": "gemini-pro",
+            "api_key": os.environ.get("GOOGLE_API_KEY"),
+            "safety_settings": [
+                    {"category": "HARM_CATEGORY_HARASSMENT", "threshold": "BLOCK_ONLY_HIGH"},
+                    {"category": "HARM_CATEGORY_HATE_SPEECH", "threshold": "BLOCK_ONLY_HIGH"},
+                    {"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT", "threshold": "BLOCK_ONLY_HIGH"},
+                    {"category": "HARM_CATEGORY_DANGEROUS_CONTENT", "threshold": "BLOCK_ONLY_HIGH"}
+                    ],
+            "top_p":0.5,
+            "max_tokens": 2048, 
+            "temperature": 1.0,
+            "top_k": 5
             }
     ]}
 
@@ -93,12 +103,28 @@ def create(self, params: Dict) -> ChatCompletion:
         messages = params.get("messages", [])
         stream = params.get("stream", False)
         n_response = params.get("n", 1)
-        params.get("temperature", 0.5)
-        params.get("top_p", 1.0)
-        params.get("max_tokens", 4096)
+
+
+        # Mapping, where Key is a term used by Autogen, and Value is a term used by Gemini
+        generation_config_params_mapping = {
+            # "n": "candidate_count", # Gemini supports only `n=1`
+            "stop_sequences": "stop_sequences",
+            "max_tokens": "max_output_tokens", 
+            "temperature": "temperature",
+            "top_p": "top_p",
+            "top_k": "top_k",
+
+            "max_output_tokens": "max_output_tokens"
+        }
+        generation_config = {
+            gemini_term: params[autogen_term]
+            for autogen_term, gemini_term in generation_config_params_mapping.items()
+            if autogen_term in params
+        }
+        safety_settings = params.get("safety_settings", {})
+
 
         if stream:
-            # warn user that streaming is not supported
             warnings.warn(
                 "Streaming is not supported for Gemini yet, and it will have no effect. Please set stream=False.",
                 UserWarning,
@@ -112,7 +138,7 @@ def create(self, params: Dict) -> ChatCompletion:
             gemini_messages = oai_messages_to_gemini_messages(messages)
 
             # we use chat model by default
-            model = genai.GenerativeModel(model_name)
+            model = genai.GenerativeModel(model_name, generation_config=generation_config, safety_settings=safety_settings)
             genai.configure(api_key=self.api_key)
             chat = model.start_chat(history=gemini_messages[:-1])
             max_retries = 5
@@ -142,7 +168,7 @@ def create(self, params: Dict) -> ChatCompletion:
         elif model_name == "gemini-pro-vision":
             # B. handle the vision model
             # Gemini's vision model does not support chat history yet
-            model = genai.GenerativeModel(model_name)
+            model = genai.GenerativeModel(model_name, generation_config=generation_config, safety_settings=safety_settings)
             genai.configure(api_key=self.api_key)
             # chat = model.start_chat(history=gemini_messages[:-1])
             # response = chat.send_message(gemini_messages[-1])


Code review comments:
diff_hunk: @@ -93,12 +103,28 @@ def create(self, params: Dict) -> ChatCompletion:
         messages = params.get("messages", [])
         stream = params.get("stream", False)
         n_response = params.get("n", 1)
-        params.get("temperature", 0.5)
-        params.get("top_p", 1.0)
-        params.get("max_tokens", 4096)
+
+
+        # Mapping, where Key is a term used by Autogen, and Value is a term used by Gemini
+        generation_config_params_mapping = {
+            # "n": "candidate_count", # Gemini supports only `n=1`
+            "stop_sequences": "stop_sequences",
+            "max_tokens": "max_output_tokens", 
+            "temperature": "temperature",
+            "top_p": "top_p",
+            "top_k": "top_k",
+
+            "max_output_tokens": "max_output_tokens"
+        }
path: autogen/oai/gemini.py
text: @user1:
define as a constant out of the method.

@author:
added as a class variable, it can now be accessed as GeminiClient.PARAMS_MAPPING
```
{'max_tokens': 'max_output_tokens',
 'stop_sequences': 'stop_sequences',
 'temperature': 'temperature',
 'top_p': 'top_p',
 'top_k': 'top_k',
 'max_output_tokens': 'max_output_tokens'}
 ```