I am building a Flask web application that serves as a recipe generator. The application should utilize the Google Gemini API (specifically, the gemini-1.5-flash model) to generate recipes based on user-provided inputs. Users should have the option to either enter ingredients manually or upload an image of ingredients. The application should then generate a list of recipe names, and upon selecting a recipe, display detailed recipe information including:

Recipe name
List of ingredients
Step-by-step instructions
Who can eat this dish
Who should avoid this dish
Any other relevant information
The website should present this information in a neat and beautiful format, with an emphasis on a tremendously beautiful design.

Requirements:

User Interface:
The website should be visually appealing and modern.
Use embedded HTML, CSS, and JavaScript in a single file for simplicity.
The initial page should allow users to:
Enter ingredients (comma-separated).
OR upload an image of ingredients.
Submit the input to generate recipes.
Functionality:
Input Handling:
If ingredients are entered, use the Gemini API to generate a list of recipe names based on those ingredients.
If an image is uploaded, use the Gemini API's image capabilities to analyze the image and generate a list of recipe names based on the ingredients detected in the image.
Recipe Generation:
After the user submits their input, display a list of recipe names for the user to select from.
Recipe Details:
When a user selects a recipe, display detailed recipe information as specified.
Ensure the recipe details are presented in a clear, structured, and visually appealing manner.
Backend Logic:
Use the Google Gemini API (gemini-1.5-flash model) for both text and image processing.
When requesting recipe details from the API, ask the model to return the response in JSON format to facilitate parsing.
Handle any potential issues with JSON parsing, such as cases where the API might return responses that are not properly formatted as JSON.
Code Structure:
Provide the entire Flask application code, including all necessary functions, in a single file (app.py).
Include all HTML, CSS, and JavaScript within the code (embedded) in index.html, so that there are no separate template or static files.
Ensure that the code is clean, well-commented, and follows best practices.
Error Handling:
Implement error handling for cases such as:
The API not returning valid JSON.
Missing or invalid inputs from the user.
Issues with image upload and processing.
Design Considerations:
The website should be responsive and work well on different screen sizes.
Use attractive fonts, colors, and layout to enhance user experience.
Include any necessary features or options that would be suitable for such an application.
Previous Attempts and Issues:

Problem Encountered: After clicking on a recipe, the application does not display the recipe details. Only the headers ("Ingredients", "Instructions", "Who Can Eat This?", "Who Should Avoid This?") are shown without any content.
Attempted Solutions:
Modified the prompt sent to the Gemini API to instruct the model to output the recipe details in JSON format.
Added parsing logic to extract the JSON from the API's response, including using regular expressions.
Set the temperature parameter to 0 to make the model's output more deterministic.
Included code block delimiters (e.g., ```json) in the prompt to encapsulate the JSON data.
Persistent Issue: Despite the above efforts, the Gemini API sometimes returns responses that are not in the expected JSON format, leading to parsing failures and the inability to display the recipe details properly on the website.
Goals:

Provide a working Flask application that meets all the above requirements.
Ensure that the Gemini API reliably returns recipe details in JSON format, and that the application correctly parses and displays this information.
Include complete code with embedded HTML, CSS, and JavaScript in a single file.
Deliver error-free code where recipe details are properly displayed after selecting a recipe.
Provide clear and helpful error messages to the user if something goes wrong.
Enhance the website's design to make it look "tremendously beautiful" as per the requirements.
Request:

Please help me by providing the complete Flask application code that fulfills these requirements. Make sure that:

The application works correctly, and the recipe details are displayed properly after selecting a recipe.
The code includes all necessary imports, configurations, and functions.
All HTML, CSS, and JavaScript are embedded within the app.py file.
Proper error handling is in place to manage potential issues with the Gemini API responses.
The design of the website is visually appealing and aligns with the requirement of being "tremendously beautiful".
Additionally, if there are any suggestions on how to ensure the Gemini API returns responses in the expected JSON format (e.g., specific prompt engineering techniques, additional parameters), please include those in the code or comments.

""

here is the google gemini documentation
"
The Gemini API can generate text output when provided text, images, video, and audio as input.

This guide shows you how to generate text using the generateContent and streamGenerateContent methods. To learn about working with Gemini's vision and audio capabilities, refer to the Vision and Audio guides.

Before you begin: Set up your project and API key
Before calling the Gemini API, you need to set up your project and configure your API key.

 Expand to view how to set up your project and API key

Generate text from text-only input
The simplest way to generate text using the Gemini API is to provide the model with a single text-only input, as shown in this example:


import google.generativeai as genai

model = genai.GenerativeModel("gemini-1.5-flash")
response = model.generate_content("Write a story about a magic backpack.")
print(response.text)

In this case, the prompt ("Write a story about a magic backpack") doesn't include any output examples, system instructions, or formatting information. It's a zero-shot approach. For some use cases, a one-shot or few-shot prompt might produce output that's more aligned with user expectations. In some cases, you might also want to provide system instructions to help the model understand the task or follow specific guidelines.

Generate text from text-and-image input
The Gemini API supports multimodal inputs that combine text with media files. The following example shows how to generate text from text-and-image input:


import google.generativeai as genai

import PIL.Image

model = genai.GenerativeModel("gemini-1.5-flash")
organ = PIL.Image.open(media / "organ.jpg")
response = model.generate_content(["Tell me about this instrument", organ])
print(response.text)

As with text-only prompting, multimodal prompting can involve various approaches and refinements. Depending on the output from this example, you might want to add steps to the prompt or be more specific in your instructions. To learn more, see File prompting strategies.

Generate a text stream
By default, the model returns a response after completing the entire text generation process. You can achieve faster interactions by not waiting for the entire result, and instead use streaming to handle partial results.

The following example shows how to implement streaming using the streamGenerateContent method to generate text from a text-only input prompt.


import google.generativeai as genai

model = genai.GenerativeModel("gemini-1.5-flash")
response = model.generate_content("Write a story about a magic backpack.", stream=True)
for chunk in response:
    print(chunk.text)
    print("_" * 80)

Build an interactive chat
You can use the Gemini API to build interactive chat experiences for your users. Using the chat feature of the API lets you collect multiple rounds of questions and responses, allowing users to step incrementally toward answers or get help with multipart problems. This feature is ideal for applications that require ongoing communication, such as chatbots, interactive tutors, or customer support assistants.

The following code example shows a basic chat implementation:


import google.generativeai as genai

model = genai.GenerativeModel("gemini-1.5-flash")
chat = model.start_chat(
    history=[
        {"role": "user", "parts": "Hello"},
        {"role": "model", "parts": "Great to meet you. What would you like to know?"},
    ]
)
response = chat.send_message("I have 2 dogs in my house.")
print(response.text)
response = chat.send_message("How many paws are in my house?")
print(response.text)

Enable chat streaming
You can also use streaming with chat, as shown in the following example:


import google.generativeai as genai

model = genai.GenerativeModel("gemini-1.5-flash")
chat = model.start_chat(
    history=[
        {"role": "user", "parts": "Hello"},
        {"role": "model", "parts": "Great to meet you. What would you like to know?"},
    ]
)
response = chat.send_message("I have 2 dogs in my house.", stream=True)
for chunk in response:
    print(chunk.text)
    print("_" * 80)
response = chat.send_message("How many paws are in my house?", stream=True)
for chunk in response:
    print(chunk.text)
    print("_" * 80)

print(chat.history)

Configure text generation
Every prompt you send to the model includes parameters that control how the model generates responses. You can use GenerationConfig to configure these parameters. If you don't configure the parameters, the model uses default options, which can vary by model.

The following example shows how to configure several of the available options.


import google.generativeai as genai

model = genai.GenerativeModel("gemini-1.5-flash")
response = model.generate_content(
    "Tell me a story about a magic backpack.",
    generation_config=genai.types.GenerationConfig(
        # Only one candidate for now.
        candidate_count=1,
        stop_sequences=["x"],
        max_output_tokens=20,
        temperature=1.0,
    ),
)

print(response.text)

candidateCount specifies the number of generated responses to return. Currently, this value can only be set to 1. If unset, this will default to 1.

stopSequences specifies the set of character sequences (up to 5) that will stop output generation. If specified, the API will stop at the first appearance of a stop_sequence. The stop sequence won't be included as part of the response.

maxOutputTokens sets the maximum number of tokens to include in a candidate.

temperature controls the randomness of the output. Use higher values for more creative responses, and lower values for more deterministic responses. Values can range from [0.0, 2.0].

You can also configure individual calls to generateContent:


response = model.generate_content(
    'Write a story about a magic backpack.',
    generation_config = genai.GenerationConfig(
        max_output_tokens=1000,
        temperature=0.1,
    )
)
Any values set on the individual call override values on the model constructor.


Explore vision capabilities with the Gemini API

Python Node.js Go REST

Try a Colab notebook
View notebook on GitHub
The Gemini API is able to process images and videos, enabling a multitude of exciting developer use cases. Some of Gemini's vision capabilities include the ability to:

Caption and answer questions about images
Transcribe and reason over PDFs, including long documents up to 2 million token context window
Describe, segment, and extract information from videos, including both visual frames and audio, up to 90 minutes long
Detect objects in an image and return bounding box coordinates for them
This tutorial demonstrates some possible ways to prompt the Gemini API with images and video input, provides code examples, and outlines prompting best practices with multimodal vision capabilities. All output is text-only.

Before you begin: Set up your project and API key
Before calling the Gemini API, you need to set up your project and configure your API key.

 Expand to view how to set up your project and API key

Prompting with images
In this tutorial, you will upload images using the File API or as inline data and generate content based on those images.

Technical details (images)
Gemini 1.5 Pro and 1.5 Flash support a maximum of 3,600 image files.

Images must be in one of the following image data MIME types:

PNG - image/png
JPEG - image/jpeg
WEBP - image/webp
HEIC - image/heic
HEIF - image/heif
Each image is equivalent to 258 tokens.

While there are no specific limits to the number of pixels in an image besides the model's context window, larger images are scaled down to a maximum resolution of 3072x3072 while preserving their original aspect ratio, while smaller images are scaled up to 768x768 pixels. There is no cost reduction for images at lower sizes, other than bandwidth, or performance improvement for images at higher resolution.

For best results:

Rotate images to the correct orientation before uploading.
Avoid blurry images.
If using a single image, place the text prompt after the image.
Image input
For total image payload size less than 20MB, we recommend either uploading base64 encoded images or directly uploading locally stored image files.

Base64 encoded images
You can upload public image URLs by encoding them as Base64 payloads. We recommend using the httpx library to fetch the image URLs. The following code example shows how to do this:


import httpx
import os
import base64

model = genai.GenerativeModel(model_name = "gemini-1.5-pro")
image_path = "https://upload.wikimedia.org/wikipedia/commons/thumb/8/87/Palace_of_Westminster_from_the_dome_on_Methodist_Central_Hall.jpg/2560px-Palace_of_Westminster_from_the_dome_on_Methodist_Central_Hall.jpg"

image = httpx.get(image_path)

prompt = "Caption this image."
response = model.generate_content([{'mime_type':'image/jpeg', 'data': base64.b64encode(image.content).decode('utf-8')}, prompt])

print(response.text)
Multiple images
To prompt with multiple images in Base64 encoded format, you can do the following:


import httpx
import os
import base64

model = genai.GenerativeModel(model_name = "gemini-1.5-pro")
image_path_1 = "path/to/your/image1.jpeg"  # Replace with the actual path to your first image
image_path_2 = "path/to/your/image2.jpeg" # Replace with the actual path to your second image

image_1 = httpx.get(image_path_1)
image_2 = httpx.get(image_path_2)

prompt = "Generate a list of all the objects contained in both images."

response = model.generate_content([
{'mime_type':'image/jpeg', 'data': base64.b64encode(image_1.content).decode('utf-8')},
{'mime_type':'image/jpeg', 'data': base64.b64encode(image_2.content).decode('utf-8')}, prompt])

print(response.text)
Upload one or more locally stored image files
Alternatively, you can upload one or more locally stored image files.


import PIL.Image
import os
import google.generativeai as genai

image_path_1 = "path/to/your/image1.jpeg"  # Replace with the actual path to your first image
image_path_2 = "path/to/your/image2.jpeg" # Replace with the actual path to your second image

sample_file_1 = PIL.Image.open(image_path_1)
sample_file_2 = PIL.Image.open(image_path_2)

#Choose a Gemini model.
model = genai.GenerativeModel(model_name="gemini-1.5-pro")

prompt = "Write an advertising jingle based on the items in both images."

response = model.generate_content([prompt, sample_file_1, sample_file_2])

print(response.text)
Note that these inline data calls don't include many of the features available through the File API, such as getting file metadata, listing, or deleting files.

Large image payloads
When the combination of files and system instructions that you intend to send is larger than 20 MB in size, use the File API to upload those files.

Use the media.upload method of the File API to upload an image of any size.

Note: The File API lets you store up to 20 GB of files per project, with a per-file maximum size of 2 GB. Files are stored for 48 hours. They can be accessed in that period with your API key, but cannot be downloaded from the API. It is available at no cost in all regions where the Gemini API is available.
After uploading the file, you can make GenerateContent requests that reference the File API URI. Select the generative model and provide it with a text prompt and the uploaded image.


import google.generativeai as genai

myfile = genai.upload_file(media / "Cajun_instruments.jpg")
print(f"{myfile=}")

model = genai.GenerativeModel("gemini-1.5-flash")
result = model.generate_content(
    [myfile, "\n\n", "Can you tell me about the instruments in this photo?"]
)
print(f"{result.text=}")

OpenAI Compatibility
You can access Gemini's image understanding capabilities using the OpenAI libraries. This lets you integrate Gemini into existing OpenAI workflows by updating three lines of code and using your Gemini API key. See the Image understanding example for code demonstrating how to send images encoded as Base64 payloads.

Capabilities
This section outlines specific vision capabilities of the Gemini model, including object detection and bounding box coordinates.

Get a bounding box for an object
Gemini models are trained to return bounding box coordinates as relative widths or heights in the range of [0, 1]. These values are then scaled by 1000 and converted to integers. Effectively, the coordinates represent the bounding box on a 1000x1000 pixel version of the image. Therefore, you'll need to convert these coordinates back to the dimensions of your original image to accurately map the bounding boxes.


# Choose a Gemini model.
model = genai.GenerativeModel(model_name="gemini-1.5-pro")

prompt = "Return a bounding box for each of the objects in this image in [ymin, xmin, ymax, xmax] format."
response = model.generate_content([sample_file_1, prompt])

print(response.text)
The model returns bounding box coordinates in the format [ymin, xmin, ymax, xmax]. To convert these normalized coordinates to the pixel coordinates of your original image, follow these steps:

Divide each output coordinate by 1000.
Multiply the x-coordinates by the original image width.
Multiply the y-coordinates by the original image height.
To explore more detailed examples of generating bounding box coordinates and visualizing them on images, we encourage you to review our Object Detection cookbook example.

Prompting with video
In this tutorial, you will upload a video using the File API and generate content based on those images.

Note: The File API is required to upload video files, due to their size. However, the File API is only available for Python, Node.js, Go, and REST.
Technical details (video)
Gemini 1.5 Pro and Flash support up to approximately an hour of video data.

Video must be in one of the following video format MIME types:

video/mp4
video/mpeg
video/mov
video/avi
video/x-flv
video/mpg
video/webm
video/wmv
video/3gpp
The File API service extracts image frames from videos at 1 frame per second (FPS) and audio at 1Kbps, single channel, adding timestamps every second. These rates are subject to change in the future for improvements in inference.

Note: The details of fast action sequences may be lost at the 1 FPS frame sampling rate. Consider slowing down high-speed clips for improved inference quality.
Individual frames are 258 tokens, and audio is 32 tokens per second. With metadata, each second of video becomes ~300 tokens, which means a 1M context window can fit slightly less than an hour of video.

To ask questions about time-stamped locations, use the format MM:SS, where the first two digits represent minutes and the last two digits represent seconds.

For best results:

Use one video per prompt.
If using a single video, place the text prompt after the video.
Upload a video file using the File API
Note: The File API lets you store up to 20 GB of files per project, with a per-file maximum size of 2 GB. Files are stored for 48 hours. They can be accessed in that period with your API key, but they cannot be downloaded using any API. It is available at no cost in all regions where the Gemini API is available.
The File API accepts video file formats directly. This example uses the short NASA film "Jupiter's Great Red Spot Shrinks and Grows". Credit: Goddard Space Flight Center (GSFC)/David Ladd (2018).

"Jupiter's Great Red Spot Shrinks and Grows" is in the public domain and does not show identifiable people. (NASA image and media usage guidelines.)

Start by retrieving the short video:


wget https://storage.googleapis.com/generativeai-downloads/images/GreatRedSpot.mp4
Upload the video using the File API and print the URI.


# Upload the video and print a confirmation.
video_file_name = "GreatRedSpot.mp4"

print(f"Uploading file...")
video_file = genai.upload_file(path=video_file_name)
print(f"Completed upload: {video_file.uri}")
Verify file upload and check state
Verify the API has successfully received the files by calling the files.get method.

Note: Video files have a State field in the File API. When a video is uploaded, it will be in the PROCESSING state until it is ready for inference. Only ACTIVE files can be used for model inference.

import time

# Check whether the file is ready to be used.
while video_file.state.name == "PROCESSING":
    print('.', end='')
    time.sleep(10)
    video_file = genai.get_file(video_file.name)

if video_file.state.name == "FAILED":
  raise ValueError(video_file.state.name)

Prompt with a video and text
Once the uploaded video is in the ACTIVE state, you can make GenerateContent requests that specify the File API URI for that video. Select the generative model and provide it with the uploaded video and a text prompt.


# Create the prompt.
prompt = "Summarize this video. Then create a quiz with answer key based on the information in the video."

# Choose a Gemini model.
model = genai.GenerativeModel(model_name="gemini-1.5-pro")

# Make the LLM request.
print("Making LLM inference request...")
response = model.generate_content([video_file, prompt],
                                  request_options={"timeout": 600})

# Print the response, rendering any Markdown
Markdown(response.text)
Refer to timestamps in the content
You can use timestamps of the form HH:MM:SS to refer to specific moments in the video.


# Create the prompt.
prompt = "What are the examples given at 01:05 and 01:19 supposed to show us?"

# Choose a Gemini model.
model = genai.GenerativeModel(model_name="gemini-1.5-pro")

# Make the LLM request.
print("Making LLM inference request...")
response = model.generate_content([video_file, prompt],
                                  request_options={"timeout": 600})
print(response.text)
Transcribe video and provide visual descriptions
The Gemini models can transcribe and provide visual descriptions of video content by processing both the audio track and visual frames. For visual descriptions, the model samples the video at a rate of 1 frame per second. This sampling rate may affect the level of detail in the descriptions, particularly for videos with rapidly changing visuals.


# Create the prompt.
prompt = "Transcribe the audio from this video, giving timestamps for salient events in the video. Also provide visual descriptions."

# Choose a Gemini model.
model = genai.GenerativeModel(model_name="gemini-1.5-pro")

# Make the LLM request.
print("Making LLM inference request...")
response = model.generate_content([video_file, prompt],
                                  request_options={"timeout": 600})
print(response.text)
List files
You can list all files uploaded using the File API and their URIs using files.list.


import google.generativeai as genai

print("My files:")
for f in genai.list_files():
    print("  ", f.name)

Delete files
Files uploaded using the File API are automatically deleted after 2 days. You can also manually delete them using files.delete.


import google.generativeai as genai

myfile = genai.upload_file(media / "poem.txt")

myfile.delete()

try:
    # Error.
    model = genai.GenerativeModel("gemini-1.5-flash")
    result = model.generate_content([myfile, "Describe this file."])
except google.api_core.exceptions.PermissionDenied:
    pass
""