this is my app.py code in which it generates a ppt based on the user input..however the generated ppt is worst and the images are not generating...make sure to fetch the images from the internet and also make sure that the output text formatting should be fantastic...and also make sure to have various designs for each slide in the ppt and also make sure to have a proper title ,theme , headdigns and side headdings etc ...also it is not generating any flowchart...fix it too...the output text formatting is so worse that i am getting the astericks(**)...fix it also the images are not fetching from the internet...fix it too...u can use that google gemini model to fetch for the image links or images from the internet ...

"

import os
from flask import Flask, render_template, request, jsonify
import google.generativeai as genai
from pptx import Presentation
from pptx.util import Inches, Pt
from pptx.dml.color import RGBColor
from pptx.enum.shapes import MSO_SHAPE
from pptx.enum.text import PP_ALIGN
import io
import base64
from PIL import Image
import requests
import re
import random

app = Flask(__name__)

# Configure the Gemini API
genai.configure(api_key=os.environ.get("GEMINI_API_KEY"))
model = genai.GenerativeModel("gemini-1.5-pro")

@app.route('/')
def index():
    return render_template('index.html')

@app.route('/generate_ppt', methods=['POST'])
def generate_ppt():
    data = request.json
    topic = data['topic']
    include_flowcharts = data['includeFlowcharts']
    include_images = data['includeImages']
    num_slides = int(data['numSlides'])

    # Generate content using Gemini API
    prompt = f"""Create a detailed outline for a PowerPoint presentation on the topic: {topic}.
    Number of slides: {num_slides}
    Include flowcharts: {'Yes' if include_flowcharts else 'No'}
    Include images: {'Yes' if include_images else 'No'}
    
    For each slide, provide the following information in a structured format:
    [Slide X]
    Title: (Slide title)
    Content:
    - (Bullet point 1)
    - (Bullet point 2)
    - (Bullet point 3)
    Image: (If applicable, provide a short description for image search)
    Flowchart: (If applicable, provide a description of the flowchart)
    
    Make sure the content is engaging, informative, and well-structured.
    Ensure that the outline is clearly formatted and easy to parse."""

    response = model.generate_content(prompt)
    content = response.text

    # Create PowerPoint presentation
    prs = Presentation()
    prs.slide_width = Inches(16)
    prs.slide_height = Inches(9)

    # Define slide layouts
    title_slide_layout = prs.slide_layouts[0]
    content_slide_layout = prs.slide_layouts[1]

    # Define color schemes
    color_schemes = [
        {"background": RGBColor(240, 248, 255), "title": RGBColor(70, 130, 180), "text": RGBColor(0, 0, 0)},
        {"background": RGBColor(255, 250, 240), "title": RGBColor(210, 105, 30), "text": RGBColor(0, 0, 0)},
        {"background": RGBColor(240, 255, 240), "title": RGBColor(46, 139, 87), "text": RGBColor(0, 0, 0)},
        {"background": RGBColor(255, 240, 245), "title": RGBColor(199, 21, 133), "text": RGBColor(0, 0, 0)},
        {"background": RGBColor(245, 245, 245), "title": RGBColor(105, 105, 105), "text": RGBColor(0, 0, 0)}
    ]

    # Add title slide
    title_slide = prs.slides.add_slide(title_slide_layout)
    title = title_slide.shapes.title
    subtitle = title_slide.placeholders[1]
    title.text = topic
    subtitle.text = f"A {num_slides}-slide presentation"

    # Apply a random color scheme to the title slide
    color_scheme = random.choice(color_schemes)
    title_slide.background.fill.solid()
    title_slide.background.fill.fore_color.rgb = color_scheme["background"]
    title.text_frame.paragraphs[0].font.color.rgb = color_scheme["title"]
    subtitle.text_frame.paragraphs[0].font.color.rgb = color_scheme["text"]

    # Parse the generated content and create slides
    slides = re.split(r'\[Slide \d+\]', content)[1:]  # Split content into slides
    for slide_content in slides:
        slide = prs.slides.add_slide(content_slide_layout)
        
        # Apply a random color scheme to the slide
        color_scheme = random.choice(color_schemes)
        slide.background.fill.solid()
        slide.background.fill.fore_color.rgb = color_scheme["background"]
        
        # Extract slide information
        title_match = re.search(r'Title: (.+)', slide_content)
        content_match = re.search(r'Content:([\s\S]*?)(?:Image:|Flowchart:|$)', slide_content)
        image_match = re.search(r'Image: (.+)', slide_content)
        flowchart_match = re.search(r'Flowchart: (.+)', slide_content)

        # Set slide title
        if title_match:
            title = slide.shapes.title
            title.text = title_match.group(1).strip()
            title.text_frame.paragraphs[0].font.color.rgb = color_scheme["title"]
            title.text_frame.paragraphs[0].font.size = Pt(40)
            title.text_frame.paragraphs[0].alignment = PP_ALIGN.CENTER
        
        # Set slide content
        if content_match:
            content = content_match.group(1).strip()
            text_box = slide.shapes.add_textbox(Inches(0.5), Inches(1.5), Inches(7), Inches(5))
            text_frame = text_box.text_frame
            for point in content.split('\n'):
                p = text_frame.add_paragraph()
                p.text = point.strip('- ')
                p.level = 0
                p.font.color.rgb = color_scheme["text"]
                p.font.size = Pt(24)
        
        # Add image if required
        if include_images and image_match:
            image_desc = image_match.group(1).strip()
            image_url = fetch_image_url(image_desc)
            if image_url:
                response = requests.get(image_url)
                image = Image.open(io.BytesIO(response.content))
                image_path = f"temp_image_{slides.index(slide_content)}.png"
                image.save(image_path)
                slide.shapes.add_picture(image_path, Inches(8.5), Inches(2), width=Inches(6))
                os.remove(image_path)
        
        # Add flowchart if required
        if include_flowcharts and flowchart_match:
            flowchart_desc = flowchart_match.group(1).strip()
            # Use Gemini to generate a Mermaid flowchart
            flowchart_prompt = f"Create a simple Mermaid flowchart for: {flowchart_desc}"
            flowchart_response = model.generate_content(flowchart_prompt)
            flowchart_code = flowchart_response.text
            
            # Convert Mermaid to image using Mermaid CLI (assuming it's installed)
            with open("flowchart.mmd", "w") as f:
                f.write(flowchart_code)
            os.system(f"mmdc -i flowchart.mmd -o flowchart.png")
            slide.shapes.add_picture("flowchart.png", Inches(8.5), Inches(2), width=Inches(6))
            os.remove("flowchart.png")
            os.remove("flowchart.mmd")

        # Add slide number
        slide_number = slide.shapes.add_textbox(Inches(15), Inches(8.5), Inches(0.5), Inches(0.5))
        slide_number.text_frame.text = str(slides.index(slide_content) + 1)
        slide_number.text_frame.paragraphs[0].font.size = Pt(14)
        slide_number.text_frame.paragraphs[0].font.color.rgb = color_scheme["text"]

    # Save the presentation
    ppt_path = "generated_presentation.pptx"
    prs.save(ppt_path)

    # Convert to base64 for sending to the frontend
    with open(ppt_path, "rb") as file:
        encoded_ppt = base64.b64encode(file.read()).decode('utf-8')

    os.remove(ppt_path)

    return jsonify({"ppt_data": encoded_ppt})

def fetch_image_url(query):
    # Replace this with an actual image search API (e.g., Unsplash, Pixabay, or Google Custom Search)
    # For this example, we'll use Unsplash API (you need to sign up for a free API key)
    unsplash_api_key = os.environ.get("UNSPLASH_API_KEY")
    url = f"https://api.unsplash.com/search/photos?query={query}&client_id={unsplash_api_key}"
    response = requests.get(url)
    data = response.json()
    if data['results']:
        return data['results'][0]['urls']['regular']
    return None

if __name__ == '__main__':
    app.run(debug=True)
"
"
here i am attaching the google gemini api documentation....based on this give me the complete code
"
Generate text using the Gemini API

bookmark_border

Python Node.js Go REST

The Gemini API can generate text output from various types of input, including text, images, video, and audio. You can use text generation for various applications, including:

Creative writing
Describing or interpreting media assets
Text completion
Summarizing free-form text
Translating between languages
Chatbots
Your own novel use cases
This guide shows you how to generate text using the generateContent and streamGenerateContent APIs. The focus is on text output from text-only and text-and-image input. To learn more about multimodal prompting with video and audio files, see File prompting strategies.

Before you begin: Set up your project and API key
Before calling the Gemini API, you need to set up your project and configure your API key.

Expand to view how to set up your project and API key

Generate text from text-only input
The simplest way to generate text using the Gemini API is to provide the model with a single text-only input, as shown in this example:

model = genai.GenerativeModel("gemini-1.5-flash")
response = model.generate_content("Write a story about a magic backpack.")
print(response.text)

In this case, the prompt ("Write a story about a magic backpack") doesn't include any output examples, system instructions, or formatting information. It's a zero-shot approach. For some use cases, a one-shot or few-shot prompt might produce output that's more aligned with user expectations. In some cases, you might also want to provide system instructions to help the model understand the task or follow specific guidelines.

Generate text from text-and-image input
The Gemini API supports multimodal inputs that combine text with media files. The following example shows how to generate text from text-and-image input:

import PIL.Image

model = genai.GenerativeModel("gemini-1.5-flash")
organ = PIL.Image.open(media / "organ.jpg")
response = model.generate_content(["Tell me about this instrument", organ])
print(response.text)

As with text-only prompting, multimodal prompting can involve various approaches and refinements. Depending on the output from this example, you might want to add steps to the prompt or be more specific in your instructions. To learn more, see File prompting strategies.

Generate a text stream
By default, the model returns a response after completing the entire text generation process. You can achieve faster interactions by not waiting for the entire result, and instead use streaming to handle partial results.

The following example shows how to implement streaming using the streamGenerateContent method to generate text from a text-only input prompt.

model = genai.GenerativeModel("gemini-1.5-flash")
response = model.generate_content("Write a story about a magic backpack.", stream=True)
for chunk in response:
print(chunk.text)
print("_" * 80)

Build an interactive chat
You can use the Gemini API to build interactive chat experiences for your users. Using the chat feature of the API lets you collect multiple rounds of questions and responses, allowing users to step incrementally toward answers or get help with multipart problems. This feature is ideal for applications that require ongoing communication, such as chatbots, interactive tutors, or customer support assistants.

The following code example shows a basic chat implementation:

model = genai.GenerativeModel("gemini-1.5-flash")
chat = model.start_chat(
history=[
{"role": "user", "parts": "Hello"},
{"role": "model", "parts": "Great to meet you. What would you like to know?"},
]
)
response = chat.send_message("I have 2 dogs in my house.")
print(response.text)
response = chat.send_message("How many paws are in my house?")
print(response.text)

Configure text generation
Every prompt you send to the model includes parameters that control how the model generates responses. You can use GenerationConfig to configure these parameters. If you don't configure the parameters, the model uses default options, which can vary by model.

The following example shows how to configure several of the available options.

model = genai.GenerativeModel("gemini-1.5-flash")
response = model.generate_content(
"Tell me a story about a magic backpack.",
generation_config=genai.types.GenerationConfig(
# Only one candidate for now.
candidate_count=1,
stop_sequences=["x"],
max_output_tokens=20,
temperature=1.0,
),
)

print(response.text)

candidateCount specifies the number of generated responses to return. Currently, this value can only be set to 1. If unset, this will default to 1.

stopSequences specifies the set of character sequences (up to 5) that will stop output generation. If specified, the API will stop at the first appearance of a stop_sequence. The stop sequence won't be included as part of the response.

maxOutputTokens sets the maximum number of tokens to include in a candidate.

temperature controls the randomness of the output. Use higher values for more creative responses, and lower values for more deterministic responses. Values can range from [0.0, 2.0].

You can also configure individual calls to generateContent:

response = model.generate_content(
'Write a story about a magic backpack.',
generation_config = genai.GenerationConfig(
max_output_tokens=1000,
temperature=0.1,
)
)
Any values set on the individual call override values on the model constructor.

What's next
This guide shows how to use generateContent and streamGenerateContent to generate text outputs from text-only and text-and-image inputs. To learn more about generating text using the Gemini API, see the following resources:

File prompting strategies: The Gemini API supports prompting with text, image, audio, and video data, also known as multimodal prompting.
System instructions: System instructions let you steer the behavior of the model based on your specific needs and use cases.
Safety guidance: Sometimes generative AI models produce unexpected outputs, such as outputs that are inaccurate, biased, or offensive. Post-processing and human evaluation are essential to limit the risk of harm from such outputs.

Explore vision capabilities with the Gemini API

bookmark_border

Python Node.js Go REST

Try a Colab notebook
View notebook on GitHub
The Gemini API can run inference on images and videos passed to it. When passed an image, a series of images, or a video, Gemini can:

Describe or answer questions about the content
Summarize the content
Extrapolate from the content
This tutorial demonstrates some possible ways to prompt the Gemini API with images and video input. All output is text-only.

Before you begin: Set up your project and API key
Before calling the Gemini API, you need to set up your project and configure your API key.

Expand to view how to set up your project and API key

Prompting with images
In this tutorial, you will upload images using the File API or as inline data and generate content based on those images.

Technical details (images)
Gemini 1.5 Pro and 1.5 Flash support a maximum of 3,600 image files.

Images must be in one of the following image data MIME types:

PNG - image/png
JPEG - image/jpeg
WEBP - image/webp
HEIC - image/heic
HEIF - image/heif
Each image is equivalent to 258 tokens.

While there are no specific limits to the number of pixels in an image besides the model's context window, larger images are scaled down to a maximum resolution of 3072x3072 while preserving their original aspect ratio, while smaller images are scaled up to 768x768 pixels. There is no cost reduction for images at lower sizes, other than bandwidth, or performance improvement for images at higher resolution.

For best results:

Rotate images to the correct orientation before uploading.
Avoid blurry images.
If using a single image, place the text prompt after the image.
Upload an image and generate content
Use the media.upload method of the File API to upload an image of any size. (Always use the File API when the combination of files and system instructions that you intend to send is larger than 20 MB.)

Note: The File API lets you store up to 20 GB of files per project, with a per-file maximum size of 2 GB. Files are stored for 48 hours. They can be accessed in that period with your API key, but cannot be downloaded from the API. It is available at no cost in all regions where the Gemini API is available.
After uploading the file, you can make GenerateContent requests that reference the File API URI. Select the generative model and provide it with a text prompt and the uploaded image.

myfile = genai.upload_file(media / "Cajun_instruments.jpg")
print(f"{myfile=}")

model = genai.GenerativeModel("gemini-1.5-flash")
result = model.generate_content(
[myfile, "\n\n", "Can you tell me about the instruments in this photo?"]
)
print(f"{result.text=}")

Verify image file upload and get metadata
You can verify the API successfully stored the uploaded file and get its metadata by calling files.get. Only the name (and by extension, the uri) are unique.

myfile = genai.upload_file(media / "poem.txt")
file_name = myfile.name
print(file_name) # "files/*"

myfile = genai.get_file(file_name)
print(myfile)

Upload one or more locally stored image files
Alternatively, you can upload your own files.

When the combination of files and system instructions that you intend to send is larger than 20MB in size, use the File API to upload those files, as previously shown. Smaller files can instead be called locally from the Gemini API:

import PIL.Image

sample_file_2 = PIL.Image.open('piranha.jpg')
sample_file_3 = PIL.Image.open('firefighter.jpg')
Note that these inline data calls don't include many of the features available through the File API, such as getting file metadata, listing, or deleting files.

Prompt with multiple images
You can provide the Gemini API with any combination of images and text that fit within the model's context window. This example provides one short text prompt and the three images previously uploaded.

Choose a Gemini model.
model = genai.GenerativeModel(model_name="gemini-1.5-pro")

prompt = "Write an advertising jingle showing how the product in the first image could solve the problems shown in the second two images."

response = model.generate_content([prompt, sample_file, sample_file_2, sample_file_3])

Markdown(">" + response.text)
Get a bounding box for an object
You can ask the model for the coordinates of bounding boxes for objects in images. For object detection, the Gemini model has been trained to provide these coordinates as relative widths or heights in range [0,1], scaled by 1000 and converted to an integer. Effectively, the coordinates given are for a 1000x1000 version of the original image, and need to be converted back to the dimensions of the original image.

Choose a Gemini model.
model = genai.GenerativeModel(model_name="gemini-1.5-pro")

prompt = "Return a bounding box for the piranha. \n [ymin, xmin, ymax, xmax]"
response = model.generate_content([piranha, prompt])

print(response.text)
To convert these coordinates to the dimensions of the original image:

Divide each output coordinate by 1000.
Multiply the x-coordinates by the original image width.
Multiply the y-coordinates by the original image height.
Prompting with video
In this tutorial, you will upload a video using the File API and generate content based on those images.

Note: The File API is required to upload video files, due to their size. However, the File API is only available for Python, Node.js, Go, and REST.
Technical details (video)
Gemini 1.5 Pro and Flash support up to approximately an hour of video data.

Video must be in one of the following video format MIME types:

video/mp4
video/mpeg
video/mov
video/avi
video/x-flv
video/mpg
video/webm
video/wmv
video/3gpp
The File API service extracts image frames from videos at 1 frame per second (FPS) and audio at 1Kbps, single channel, adding timestamps every second. These rates are subject to change in the future for improvements in inference.

Note: The details of fast action sequences may be lost at the 1 FPS frame sampling rate. Consider slowing down high-speed clips for improved inference quality.
Individual frames are 258 tokens, and audio is 32 tokens per second. With metadata, each second of video becomes ~300 tokens, which means a 1M context window can fit slightly less than an hour of video.

To ask questions about time-stamped locations, use the format MM:SS, where the first two digits represent minutes and the last two digits represent seconds.

For best results:

Use one video per prompt.
If using a single video, place the text prompt after the video.
Upload a video file using the File API
Note: The File API lets you store up to 20 GB of files per project, with a per-file maximum size of 2 GB. Files are stored for 48 hours. They can be accessed in that period with your API key, but they cannot be downloaded using any API. It is available at no cost in all regions where the Gemini API is available.
The File API accepts video file formats directly. This example uses the short NASA film "Jupiter's Great Red Spot Shrinks and Grows". Credit: Goddard Space Flight Center (GSFC)/David Ladd (2018).

"Jupiter's Great Red Spot Shrinks and Grows" is in the public domain and does not show identifiable people. (NASA image and media usage guidelines.)

Start by retrieving the short video:

wget https://storage.googleapis.com/generativeai-downloads/images/GreatRedSpot.mp4
Upload the video using the File API and print the URI.

Upload the video and print a confirmation.
video_file_name = "GreatRedSpot.mp4"

print(f"Uploading file...")
video_file = genai.upload_file(path=video_file_name)
print(f"Completed upload: {video_file.uri}")
Verify file upload and check state
Verify the API has successfully received the files by calling the files.get method.

Note: Video files have a State field in the File API. When a video is uploaded, it will be in the PROCESSING state until it is ready for inference. Only ACTIVE files can be used for model inference.

import time

Check whether the file is ready to be used.
while video_file.state.name == "PROCESSING":
print('.', end='')
time.sleep(10)
video_file = genai.get_file(video_file.name)

if video_file.state.name == "FAILED":
raise ValueError(video_file.state.name)

Prompt with a video and text
Once the uploaded video is in the ACTIVE state, you can make GenerateContent requests that specify the File API URI for that video. Select the generative model and provide it with the uploaded video and a text prompt.

Create the prompt.
prompt = "Summarize this video. Then create a quiz with answer key based on the information in the video."

Choose a Gemini model.
model = genai.GenerativeModel(model_name="gemini-1.5-pro")

Make the LLM request.
print("Making LLM inference request...")
response = model.generate_content([video_file, prompt],
request_options={"timeout": 600})

Print the response, rendering any Markdown
Markdown(response.text)
Refer to timestamps in the content
You can use timestamps of the form MM:SS to refer to specific moments in the video.

Create the prompt.
prompt = "What are the examples given at 01:05 and 01:19 supposed to show us?"

Choose a Gemini model.
model = genai.GenerativeModel(model_name="gemini-1.5-pro")

Make the LLM request.
print("Making LLM inference request...")
response = model.generate_content([prompt, video_file],
request_options={"timeout": 600})
print(response.text)
Transcribe video and provide visual descriptions
If the video is not fast-paced (only 1 frame per second of video is sampled), it's possible to transcribe the video with visual descriptions for each shot.

Create the prompt.
prompt = "Transcribe the audio, giving timestamps. Also provide visual descriptions."

Choose a Gemini model.
model = genai.GenerativeModel(model_name="gemini-1.5-pro")

Make the LLM request.
print("Making LLM inference request...")
response = model.generate_content([prompt, video_file],
request_options={"timeout": 600})
print(response.text)
List files
You can list all files uploaded using the File API and their URIs using files.list.

print("My files:")
for f in genai.list_files():
print(" ", f.name)

Delete files
Files uploaded using the File API are automatically deleted after 2 days. You can also manually delete them using files.delete.

myfile = genai.upload_file(media / "poem.txt")

myfile.delete()

try:
# Error.
model = genai.GenerativeModel("gemini-1.5-flash")
result = model.generate_content([myfile, "Describe this file."])
except google.api_core.exceptions.PermissionDenied:
pass

What's next
This guide shows how to upload image and video files using the File API and then generate text outputs from image and video inputs. To learn more, see the following resources:

File prompting strategies: The Gemini API supports prompting with text, image, audio, and video data, also known as multimodal prompting.
System instructions: System instructions let you steer the behavior of the model based on your specific needs and use cases.
Safety guidance: Sometimes generative AI models produce unexpected outputs, such as outputs that are inaccurate, biased, or offensive. Post-processing and human evaluation are essential to limit the risk of harm from such outputs.

Long context

bookmark_border

Gemini 1.5 Flash comes standard with a 1-million-token context window, and Gemini 1.5 Pro comes with a 2-million-token context window. Historically, large language models (LLMs) were significantly limited by the amount of text (or tokens) that could be passed to the model at one time. The Gemini 1.5 long context window, with near-perfect retrieval (>99%), unlocks many new use cases and developer paradigms.

The code you already use for cases like text generation or multimodal inputs will work out of the box with long context.

Throughout this guide, you briefly explore the basics of the context window, how developers should think about long context, various real world use cases for long context, and ways to optimize the usage of long context.

What is a context window?
The basic way you use the Gemini 1.5 models is by passing information (context) to the model, which will subsequently generate a response. An analogy for the context window is short term memory. There is a limited amount of information that can be stored in someone's short term memory, and the same is true for generative models.

You can read more about how models work under the hood in our generative models guide.

Getting started with long context
Most generative models created in the last few years were only capable of processing 8,000 tokens at a time. Newer models pushed this further by accepting 32,000 tokens or 128,000 tokens. Gemini 1.5 is the first model capable of accepting 1 million tokens, and now 2 million tokens with Gemini 1.5 Pro.

In practice, 1 million tokens would look like:

50,000 lines of code (with the standard 80 characters per line)
All the text messages you have sent in the last 5 years
8 average length English novels
Transcripts of over 200 average length podcast episodes
Even though the models can take in more and more context, much of the conventional wisdom about using large language models assumes this inherent limitation on the model, which as of 2024, is no longer the case.

Some common strategies to handle the limitation of small context windows included:

Arbitrarily dropping old messages / text from the context window as new text comes in
Summarizing previous content and replacing it with the summary when the context window gets close to being full
Using RAG with semantic search to move data out of the context window and into a vector database
Using deterministic or generative filters to remove certain text / characters from prompts to save tokens
While many of these are still relevant in certain cases, the default place to start is now just putting all of the tokens into the context window. Because Gemini 1.5 models were purpose-built with a long context window, they are much more capable of in-context learning. For example, with only instructional materials (a 500-page reference grammar, a dictionary, and ≈ 400 extra parallel sentences) all provided in context, Gemini 1.5 Pro and Gemini 1.5 Flash are capable of learning to translate from English to Kalamang— a Papuan language with fewer than 200 speakers and therefore almost no online presence—with quality similar to a person who learned from the same materials.

This example underscores how you can start to think about what is possible with long context and the in-context learning capabilities of Gemini 1.5.

Long context use cases
While the standard use case for most generative models is still text input, the Gemini 1.5 model family enables a new paradigm of multimodal use cases. These models can natively understand text, video, audio, and images. They are accompanied by the Gemini API that takes in multimodal file types for convenience.

Long form text
Text has proved to be the layer of intelligence underpinning much of the momentum around LLMs. As mentioned earlier, much of the practical limitation of LLMs was because of not having a large enough context window to do certain tasks. This led to the rapid adoption of retrieval augmented generation (RAG) and other techniques which dynamically provide the model with relevant contextual information. Now, with larger and larger context windows (currently up to 2 million on Gemini 1.5 Pro), there are new techniques becoming available which unlock new use cases.

Some emerging and standard use cases for text based long context include:

Summarizing large corpuses of text
Previous summarization options with smaller context models would require a sliding window or another technique to keep state of previous sections as new tokens are passed to the model
Question and answering
Historically this was only possible with RAG given the limited amount of context and models' factual recall being low
Agentic workflows
Text is the underpinning of how agents keep state of what they have done and what they need to do; not having enough information about the world and the agent's goal is a limitation on the reliability of agents
Many-shot in-context learning is one of the most unique capabilities unlocked by long context models. Research has shown that taking the common "single shot" or "multi-shot" example paradigm, where the model is presented with one or a few examples of a task, and scaling that up to hundreds, thousands, or even hundreds of thousands of examples, can lead to novel model capabilities. This many-shot approach has also been shown to perform similarly to models which were fine-tuned for a specific task. For use cases where a Gemini model's performance is not yet sufficient for a production rollout, you can try the many-shot approach. As you might explore later in the long context optimization section, context caching makes this type of high input token workload much more economically feasible and even lower latency in some cases.

Long form video
Video content's utility has long been constrained by the lack of accessibility of the medium itself. It was hard to skim the content, transcripts often failed to capture the nuance of a video, and most tools don't process image, text, and audio together. With Gemini 1.5, the long-context text capabilities translate to the ability to reason and answer questions about multimodal inputs with sustained performance. Gemini 1.5 Flash, when tested on the needle in a video haystack problem with 1M tokens, obtained >99.8% recall of the video in the context window, and 1.5 Pro reached state of the art performance on the Video-MME benchmark.

Some emerging and standard use cases for video long context include:

Video question and answering
Video memory, as shown with Google's Project Astra
Video captioning
Video recommendation systems, by enriching existing metadata with new multimodal understanding
Video customization, by looking at a corpus of data and associated video metadata and then removing parts of videos that are not relevant to the viewer
Video content moderation
Real-time video processing
When working with videos, it is important to consider how the videos are processed into tokens, which affects billing and usage limits. You can learn more about prompting with video files in the Prompting guide.

Long form audio
The Gemini 1.5 models were the first natively multimodal large language models that could understand audio. Historically, the typical developer workflow would involve stringing together multiple domain specific models, like a speech-to-text model and a text-to-text model, in order to process audio. This led to additional latency required by performing multiple round-trip requests and decreased performance usually attributed to disconnected architectures of the multiple model setup.

On standard audio-haystack evaluations, Gemini 1.5 Pro is able to find the hidden audio in 100% of the tests and Gemini 1.5 Flash is able to find it in 98.7% of the tests. Gemini 1.5 Flash accepts up to 9.5 hours of audio in a single request and Gemini 1.5 Pro can accept up to 19 hours of audio using the 2-million-token context window. Further, on a test set of 15-minute audio clips, Gemini 1.5 Pro archives a word error rate (WER) of ~5.5%, much lower than even specialized speech-to-text models, without the added complexity of extra input segmentation and pre-processing.

Some emerging and standard use cases for audio context include:

Real-time transcription and translation
Podcast / video question and answering
Meeting transcription and summarization
Voice assistants
You can learn more about prompting with audio files in the Prompting guide.

Long context optimizations
The primary optimization when working with long context and the Gemini 1.5 models is to use context caching. Beyond the previous impossibility of processing lots of tokens in a single request, the other main constraint was the cost. If you have a "chat with your data" app where a user uploads 10 PDFs, a video, and some work documents, you would historically have to work with a more complex retrieval augmented generation (RAG) tool / framework in order to process these requests and pay a significant amount for tokens moved into the context window. Now, you can cache the files the user uploads and pay to store them on a per hour basis. The input / output cost per request with Gemini 1.5 Flash for example is ~4x less than the standard input / output cost, so if the user chats with their data enough, it becomes a huge cost saving for you as the developer.

Long context limitations
In various sections of this guide, we talked about how Gemini 1.5 models achieve high performance across various needle-in-a-haystack retrieval evals. These tests consider the most basic setup, where you have a single needle you are looking for. In cases where you might have multiple "needles" or specific pieces of information you are looking for, the model does not perform with the same accuracy. Performance can vary to a wide degree depending on the context. This is important to consider as there is an inherent tradeoff between getting the right information retrieved and cost. You can get ~99% on a single query, but you have to pay the input token cost every time you send that query. So for 100 pieces of information to be retrieved, if you needed 99% performance, you would likely need to send 100 requests. This is a good example of where context caching can significantly reduce the cost associated with using Gemini models while keeping the performance high.

FAQs
Do I lose model performance when I add more tokens to a query?
Generally, if you don't need tokens to be passed to the model, it is best to avoid passing them. However, if you have a large chunk of tokens with some information and want to ask questions about that information, the model is highly capable of extracting that information (up to 99% accuracy in many cases).

How does Gemini 1.5 Pro perform on the standard needle-in-a-haystack test?
Gemini 1.5 Pro achieves 100% recall up to 530k tokens and >99.7% recall up to 1M tokens.

How can I lower my cost with long-context queries?
If you have a similar set of tokens / context that you want to re-use many times, context caching can help reduce the costs associated with asking questions about that information.

How can I get access to the 2-million-token context window?
All developers now have access to the 2-million-token context window with Gemini 1.5 Pro.

Does the context length affect the model latency?
There is some fixed amount of latency in any given request, regardless of the size, but generally longer queries will have higher latency (time to first token).

Do the long context capabilities differ between Gemini 1.5 Flash and Gemini 1.5 Pro?
Yes, some of the numbers were mentioned in different sections of this guide, but generally Gemini 1.5 Pro is more performant on most long context use cases.
"