based on the documentation, add file uploading with messages to the html script

Generate text from text-and-image input
The Gemini API supports multimodal inputs that combine text with media files. The following example shows how to generate text from text-and-image input:


curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent?key=$GOOGLE_API_KEY" \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{
      "contents": [{
        "parts":[
            {"text": "Tell me about this instrument"},
            {
              "inline_data": {
                "mime_type":"image/jpeg",
                "data": "'$(base64 $B64FLAGS $IMG_PATH)'"
              }
            }
        ]
        }]
       }' 2> /dev/null

As with text-only prompting, multimodal prompting can involve various approaches and refinements. Depending on the output from this example, you might want to add steps to the prompt or be more specific in your instructions. To learn more, see File prompting strategies.

Generate a text stream
By default, the model returns a response after completing the entire text generation process. You can achieve faster interactions by not waiting for the entire result, and instead use streaming to handle partial results.

The following example shows how to implement streaming using the streamGenerateContent method to generate text from a text-only input prompt.


curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:streamGenerateContent?alt=sse&key=${GOOGLE_API_KEY}" \
        -H 'Content-Type: application/json' \
        --no-buffer \
        -d '{ "contents":[{"parts":[{"text": "Write a story about a magic backpack."}]}]}'

Build an interactive chat
You can use the Gemini API to build interactive chat experiences for your users. Using the chat feature of the API lets you collect multiple rounds of questions and responses, allowing users to step incrementally toward answers or get help with multipart problems. This feature is ideal for applications that require ongoing communication, such as chatbots, interactive tutors, or customer support assistants.

The following code example shows a basic chat implementation:


curl https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent?key=$GOOGLE_API_KEY \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{
      "contents": [
        {"role":"user",
         "parts":[{
           "text": "Hello"}]},
        {"role": "model",
         "parts":[{
           "text": "Great to meet you. What would you like to know?"}]},
        {"role":"user",
         "parts":[{
           "text": "I have two dogs in my house. How many paws are in my house?"}]},
      ]
    }' 2> /dev/null | grep "text"

Enable chat streaming
You can also use streaming with chat, as shown in the following example:


curl https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:streamGenerateContent?alt=sse&key=$GOOGLE_API_KEY \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{
      "contents": [
        {"role":"user",
         "parts":[{
           "text": "Hello"}]},
        {"role": "model",
         "parts":[{
           "text": "Great to meet you. What would you like to know?"}]},
        {"role":"user",
         "parts":[{
           "text": "I have two dogs in my house. How many paws are in my house?"}]},
      ]
    }' 2> /dev/null | grep "text"

Configure text generation
Every prompt you send to the model includes parameters that control how the model generates responses. You can use GenerationConfig to configure these parameters. If you don't configure the parameters, the model uses default options, which can vary by model.

The following example shows how to configure several of the available options.


curl https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent?key=$GOOGLE_API_KEY \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{
        "contents": [{
            "parts":[
                {"text": "Write a story about a magic backpack."}
            ]
        }],
        "safetySettings": [
            {
                "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
                "threshold": "BLOCK_ONLY_HIGH"
            }
        ],
        "generationConfig": {
            "stopSequences": [
                "Title"
            ],
            "temperature": 1.0,
            "maxOutputTokens": 800,
            "topP": 0.8,
            "topK": 10
        }
    }'  2> /dev/null | grep "text"

stopSequences specifies the set of character sequences (up to 5) that will stop output generation. If specified, the API will stop at the first appearance of a stop_sequence. The stop sequence won't be included as part of the response.

temperature controls the randomness of the output. Use higher values for more creative responses, and lower values for more deterministic responses. Values can range from [0.0, 2.0].

maxOutputTokens sets the maximum number of tokens to include in a candidate.

topP changes how the model selects tokens for output. Tokens are selected from the most to least probable until the sum of their probabilities equals the topP value. The default topP value is 0.95.

topK changes how the model selects tokens for output. A topK of 1 means the selected token is the most probable among all the tokens in the model's vocabulary, while a topK of 3 means that the next token is selected from among the 3 most probable using the temperature. Tokens are further filtered based on topP with the final token selected using temperature sampling.

Upload a document and generate content
You can use the File API to upload a document of any size. Always use the File API when the total request size (including the files, text prompt, system instructions, etc.) is larger than 20 MB.

Note: The File API lets you store up to 20 GB of files per project, with a per-file maximum size of 2 GB. Files are stored for 48 hours. They can be accessed in that period with your API key, but cannot be downloaded from the API. The File API is available at no cost in all regions where the Gemini API is available.
Call media.upload to upload a file using the File API. The following code uploads a document file and then uses the file in a call to models.generateContent.


NUM_BYTES=$(wc -c < "${PDF_PATH}")
DISPLAY_NAME=TEXT
tmp_header_file=upload-header.tmp

# Initial resumable request defining metadata.
# The upload url is in the response headers dump them to a file.
curl "${BASE_URL}/upload/v1beta/files?key=${GOOGLE_API_KEY}" \
  -D upload-header.tmp \
  -H "X-Goog-Upload-Protocol: resumable" \
  -H "X-Goog-Upload-Command: start" \
  -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \
  -H "X-Goog-Upload-Header-Content-Type: application/pdf" \
  -H "Content-Type: application/json" \
  -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null

upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r")
rm "${tmp_header_file}"

# Upload the actual bytes.
curl "${upload_url}" \
  -H "Content-Length: ${NUM_BYTES}" \
  -H "X-Goog-Upload-Offset: 0" \
  -H "X-Goog-Upload-Command: upload, finalize" \
  --data-binary "@${PDF_PATH}" 2> /dev/null > file_info.json

file_uri=$(jq ".file.uri" file_info.json)
echo file_uri=$file_uri

# Now generate content using that file
curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent?key=$GOOGLE_API_KEY" \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{
      "contents": [{
        "parts":[
          {"text": "Can you add a few more lines to this poem?"},
          {"file_data":{"mime_type": "application/pdf", "file_uri": '$file_uri'}}]
        }]
       }' 2> /dev/null > response.json

cat response.json
echo

jq ".candidates[].content.parts[].text" response.json

Get metadata for a file
You can verify the API successfully stored the uploaded file and get its metadata by calling files.get. Only the name (and by extension, the uri) are unique.


name=$(jq ".file.name" file_info.json)
# Get the file of interest to check state
curl https://generativelanguage.googleapis.com/v1beta/files/$name > file_info.json
# Print some information about the file you got
name=$(jq ".file.name" file_info.json)
echo name=$name
file_uri=$(jq ".file.uri" file_info.json)
echo file_uri=$file_uri


Technical details (images)
Gemini 1.5 Pro and 1.5 Flash support a maximum of 3,600 image files.

Images must be in one of the following image data MIME types:

PNG - image/png
JPEG - image/jpeg
WEBP - image/webp
HEIC - image/heic
HEIF - image/heif
Each image is equivalent to 258 tokens.

While there are no specific limits to the number of pixels in an image besides the model's context window, larger images are scaled down to a maximum resolution of 3072x3072 while preserving their original aspect ratio, while smaller images are scaled up to 768x768 pixels. There is no cost reduction for images at lower sizes, other than bandwidth, or performance improvement for images at higher resolution.

For best results:

Rotate images to the correct orientation before uploading.
Avoid blurry images.
If using a single image, place the text prompt after the image.
Upload an image and generate content
Use the media.upload method of the File API to upload an image of any size. (Always use the File API when the combination of files and system instructions that you intend to send is larger than 20 MB.)

Note: The File API lets you store up to 20 GB of files per project, with a per-file maximum size of 2 GB. Files are stored for 48 hours. They can be accessed in that period with your API key, but cannot be downloaded from the API. It is available at no cost in all regions where the Gemini API is available.
After uploading the file, you can make GenerateContent requests that reference the File API URI. Select the generative model and provide it with a text prompt and the uploaded image.


MIME_TYPE=$(file -b --mime-type "${IMG_PATH_2}")
NUM_BYTES=$(wc -c < "${IMG_PATH_2}")
DISPLAY_NAME=TEXT

tmp_header_file=upload-header.tmp

# Initial resumable request defining metadata.
# The upload url is in the response headers dump them to a file.
curl "${BASE_URL}/upload/v1beta/files?key=${GOOGLE_API_KEY}" \
  -D upload-header.tmp \
  -H "X-Goog-Upload-Protocol: resumable" \
  -H "X-Goog-Upload-Command: start" \
  -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \
  -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \
  -H "Content-Type: application/json" \
  -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null

upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r")
rm "${tmp_header_file}"

# Upload the actual bytes.
curl "${upload_url}" \
  -H "Content-Length: ${NUM_BYTES}" \
  -H "X-Goog-Upload-Offset: 0" \
  -H "X-Goog-Upload-Command: upload, finalize" \
  --data-binary "@${IMG_PATH_2}" 2> /dev/null > file_info.json

file_uri=$(jq ".file.uri" file_info.json)
echo file_uri=$file_uri

# Now generate content using that file
curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent?key=$GOOGLE_API_KEY" \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{
      "contents": [{
        "parts":[
          {"text": "Can you tell me about the instruments in this photo?"},
          {"file_data":
            {"mime_type": "image/jpeg", 
            "file_uri": '$file_uri'}
        }]
        }]
       }' 2> /dev/null > response.json

cat response.json
echo

jq ".candidates[].content.parts[].text" response.json


Upload an audio file and generate content
You can use the File API to upload an audio file of any size. Always use the File API when the total request size (including the files, text prompt, system instructions, etc.) is larger than 20 MB.

Note: The File API lets you store up to 20 GB of files per project, with a per-file maximum size of 2 GB. Files are stored for 48 hours. They can be accessed in that period with your API key, but cannot be downloaded from the API. The File API is available at no cost in all regions where the Gemini API is available.
Call media.upload to upload a file using the File API. The following code uploads an audio file and then uses the file in a call to models.generateContent.


MIME_TYPE=$(file -b --mime-type "${AUDIO_PATH}")
NUM_BYTES=$(wc -c < "${AUDIO_PATH}")
DISPLAY_NAME=AUDIO

tmp_header_file=upload-header.tmp

# Initial resumable request defining metadata.
# The upload url is in the response headers dump them to a file.
curl "${BASE_URL}/upload/v1beta/files?key=${GOOGLE_API_KEY}" \
  -D upload-header.tmp \
  -H "X-Goog-Upload-Protocol: resumable" \
  -H "X-Goog-Upload-Command: start" \
  -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \
  -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPE}" \
  -H "Content-Type: application/json" \
  -d "{'file': {'display_name': '${DISPLAY_NAME}'}}" 2> /dev/null

upload_url=$(grep -i "x-goog-upload-url: " "${tmp_header_file}" | cut -d" " -f2 | tr -d "\r")
rm "${tmp_header_file}"

# Upload the actual bytes.
curl "${upload_url}" \
  -H "Content-Length: ${NUM_BYTES}" \
  -H "X-Goog-Upload-Offset: 0" \
  -H "X-Goog-Upload-Command: upload, finalize" \
  --data-binary "@${AUDIO_PATH}" 2> /dev/null > file_info.json

file_uri=$(jq ".file.uri" file_info.json)
echo file_uri=$file_uri

# Now generate content using that file
curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent?key=$GOOGLE_API_KEY" \
    -H 'Content-Type: application/json' \
    -X POST \
    -d '{
      "contents": [{
        "parts":[
          {"text": "Describe this audio clip"},
          {"file_data":{"mime_type": "audio/mp3", "file_uri": '$file_uri'}}]
        }]
       }' 2> /dev/null > response.json

cat response.json
echo

jq ".candidates[].content.parts[].text" response.json


<!DOCTYPE html>
<html>
<head>
  <title>Google</title>
  <style>
    body {
      font-family: sans-serif;
      margin: 0;
      padding: 0;
      display: flex;
      justify-content: center;
      align-items: center;
      min-height: 100vh;
      background-color: #f0f0f0;
    }

    .container {
      display: flex;
      flex-direction: column;
      width: 90%;
      max-width: 600px;
      height: 80vh;
      background-color: #fff;
      border: 1px solid #ccc;
      border-radius: 8px;
      box-shadow: 0 2px 10px rgba(0,0,0,0.1);
      overflow: hidden;
    }

    /* Header for Toggle Buttons */
    .header {
      padding: 10px 20px;
      background-color: #f9f9f9;
      border-bottom: 1px solid #ddd;
      display: flex;
      flex-wrap: wrap;
      justify-content: space-between;
      align-items: center;
    }

    .toggle-container {
      display: flex;
      align-items: center;
      margin: 5px 0;
    }

    .toggle-container label {
      margin-right: 10px;
      font-size: 14px;
    }

    /* Toggle switch styles */
    .switch {
      position: relative;
      display: inline-block;
      width: 50px;
      height: 28px;
      margin-left: 10px;
    }

    .switch input {
      opacity: 0;
      width: 0;
      height: 0;
    }

    .slider {
      position: absolute;
      cursor: pointer;
      top: 0;
      left: 0;
      right: 0;
      bottom: 0;
      background-color: #ccc;
      transition: .4s;
      border-radius: 28px;
    }

    .slider:before {
      position: absolute;
      content: "";
      height: 22px;
      width: 22px;
      left: 3px;
      bottom: 3px;
      background-color: white;
      transition: .4s;
      border-radius: 50%;
    }

    input:checked + .slider {
      background-color: #2196F3;
    }

    input:checked + .slider:before {
      transform: translateX(22px);
    }

    /* Chat Container */
    #chat-container {
      flex: 1;
      padding: 15px;
      overflow-y: auto;
      background-color: #fafafa;
    }

    .message {
      padding: 10px 15px;
      margin-bottom: 10px;
      border-radius: 18px;
      max-width: 80%;
      word-wrap: break-word;
      position: relative;
      clear: both;
    }

    .user-message {
      background-color: #e0f7fa;
      align-self: flex-end;
      margin-left: auto;
      text-align: right;
    }

    .bot-message {
      background-color: #f1f8e9;
      align-self: flex-start;
      margin-right: auto;
    }

    /* Input Area */
    #input-area {
      display: flex;
      padding: 10px 20px;
      border-top: 1px solid #ddd;
      background-color: #f9f9f9;
    }

    #input-box {
      flex-grow: 1;
      padding: 10px;
      border: 1px solid #ccc;
      border-radius: 20px;
      resize: none;
      font-size: 14px;
      outline: none;
    }

    #send-button {
      padding: 10px 20px;
      margin-left: 10px;
      background-color: #007bff;
      color: #fff;
      border: none;
      border-radius: 20px;
      cursor: pointer;
      font-size: 14px;
      transition: background-color 0.3s;
    }

    #send-button:hover {
      background-color: #0056b3;
    }

    /* Typing Indicator */
    #typing-indicator {
      padding: 5px 20px;
      font-style: italic;
      color: #999;
      background-color: #f9f9f9;
      border-top: 1px solid #ddd;
      display: none;
    }

    /* Responsive Adjustments */
    @media (max-width: 768px) {
      .container {
        width: 95%;
        height: 90vh;
      }
    }

    @media (max-width: 480px) {
      .header {
        flex-direction: column;
        align-items: flex-start;
      }

      .toggle-container {
        width: 100%;
        justify-content: space-between;
      }

      #send-button {
        padding: 10px;
        margin-left: 5px;
      }
    }
  </style>
</head>

<body>
  <div class="container">
    <!-- Header with Toggle Buttons -->
    <div class="header">
      <!-- Toggle Button for Super Smart Mode -->
      <div class="toggle-container">
        <label for="superSmartToggle">Super Smart Mode</label>
        <label class="switch">
          <input type="checkbox" id="superSmartToggle">
          <span class="slider"></span>
        </label>
      </div>

      <!-- Toggle Button for Fast/Smart Model Switching -->
      <div class="toggle-container">
        <label for="modelToggle">Fast Mode</label>
        <label class="switch">
          <input type="checkbox" id="modelToggle">
          <span class="slider"></span>
        </label>
      </div>
    </div>

    <!-- Chat Messages -->
    <div id="chat-container"></div>

    <!-- Typing Indicator -->
    <div id="typing-indicator">Gemini is typing...</div> 

    <!-- Input Area -->
    <div id="input-area">
      <textarea id="input-box" placeholder="Enter your message..." rows="1"></textarea>
      <button id="send-button">Send</button>
    </div>
  </div>

  <script>
    // List of API keys. Replace these with your actual API keys.
    const API_KEYS = [
      'AIzaSyBdc_A8VxExwO6-0LY1--r2-Ym73lMg-h4',
      'AIzaSyBydeCpaNOIPSk794CpTGnfKrxbjCgJLu4',
      'AIzaSyCQ7tBHtctxlNs61Zm2Y__7KYeDLfz5dms',
      'AIzaSyAip_DFfKNC35MFIorEoNK4y6czP9P65-c'
    ];

    const chatContainer = document.getElementById('chat-container');
    const inputBox = document.getElementById('input-box');
    const sendButton = document.getElementById('send-button');
    const typingIndicator = document.getElementById('typing-indicator');
    const superSmartToggle = document.getElementById('superSmartToggle');
    const modelToggle = document.getElementById('modelToggle');

    // Initialize messageHistory based on toggles' initial state
    let messageHistory = "";

    // Current model variable
    let currentModel = 'gemini-1.5-pro-exp-0827'; // Default model

    // Event listener for the Super Smart Mode toggle switch
    superSmartToggle.addEventListener('change', () => {
      if (superSmartToggle.checked) {
        messageHistory = "tt" + messageHistory;
      } else {

        messageHistory = messageHistory.replace("tt", "");
      }
    });

    // Event listener for the Fast/Smart Model toggle switch
    modelToggle.addEventListener('change', () => {
      if (modelToggle.checked) {
        currentModel = 'gemini-1.5-flash';
      } else {
        currentModel = 'gemini-1.5-pro-exp-0827';
      }
      console.log(`Model switched to: ${currentModel}`);
    });

    sendButton.addEventListener('click', sendMessage);
    inputBox.addEventListener('keydown', (event) => {
      if (event.key === 'Enter' && !event.shiftKey) {
        event.preventDefault();
        sendMessage();
      }
    });

    /**
     * Attempts to fetch a response using the provided API key.
     * @param {string} apiKey - The API key to use for the request.
     * @param {string} prompt - The user message to send.
     * @param {string} model - The model to use for generating the response.
     * @returns {Promise<Object>} - The response data from the API.
     */
    async function fetchWithKey(apiKey, prompt, model) {
      const response = await fetch(`https://generativelanguage.googleapis.com/v1beta/models/${model}:generateContent?key=${apiKey}`, {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json'
        },
        body: JSON.stringify({
          contents: [
            {
              role: "user",
              parts: [
                { text: prompt } // Include history in the prompt
              ]
            }
          ],
          "generationConfig": {
            "temperature": 0.5,
            "topK": 64,
            "topP": 0.95,
            "maxOutputTokens": 8192,
            "responseMimeType": "text/plain"
          }
        })
      });

      if (!response.ok) {
        // Extract error message from response
        const errorData = await response.json();
        throw new Error(errorData.error?.message || 'Unknown API error');
      }

      const data = await response.json();
      return data;
    }

    /**
     * Sends the user's message and handles the bot's response using multiple API keys.
     */
    async function sendMessage() {
      const userMessageText = inputBox.value.trim();
      if (userMessageText === '') return;

      // Display user's message
      const userMessage = document.createElement('div');
      userMessage.classList.add('message', 'user-message');
      userMessage.textContent = userMessageText;
      chatContainer.appendChild(userMessage);
      chatContainer.scrollTop = chatContainer.scrollHeight;      
      
      inputBox.value = '';
      typingIndicator.style.display = 'block';

      let botMessageText = '';
      let success = false;

      // Prepare the prompt based on the current messageHistory and Super smart mode
      const currentMessageHistory = messageHistory + `User: ${userMessageText}\n`;

      for (const apiKey of API_KEYS) {  // Simplified loop
        try {
          const data = await fetchWithKey(apiKey, currentMessageHistory, currentModel); // Add user message here

          const fullResponse = data?.candidates[0]?.content?.parts[0]?.text; // Optional chaining

          if (!fullResponse) {
            throw new Error("Invalid response format: Missing content."); // Better error handling
          }

          if (superSmartToggle.checked) {
            // Parse response between <final_answer> and </final_answer>
            const finalAnswerStartIndex = fullResponse.indexOf("<final_answer>");
            const finalAnswerEndIndex = fullResponse.indexOf("</final_answer>");

            if (finalAnswerStartIndex !== -1 && finalAnswerEndIndex !== -1) {
              botMessageText = fullResponse.substring(finalAnswerStartIndex + "<final_answer>".length, finalAnswerEndIndex).trim();
            } else {
              botMessageText = "Error: Could not extract final answer. Invalid response format.";
              console.error("Final answer tags not found in response:", fullResponse);
            }
          } else {
            // When Super smart mode is off, use the entire response
            botMessageText = fullResponse.trim();
          }

          if (botMessageText) {
            // Update history *after* successful response
            messageHistory += `${botMessageText}\n`;
          }

          success = true;
          break; 
        } catch (error) {
          console.error(`Error with API key:`, error.message);
        }
      }

      const botMessage = document.createElement('div'); // Create element *outside* the loop
      botMessage.classList.add('message', 'bot-message');
      botMessage.textContent = success ? botMessageText : "I'm sorry, I'm currently unable to process your request. Please try again later."; // Display error or bot message
      chatContainer.appendChild(botMessage);
      chatContainer.scrollTop = chatContainer.scrollHeight;

      typingIndicator.style.display = 'none';
    }
  </script>
</body>
</html>