This is my app.py code along with the chat.html code...make sure that to keep the personna as the system instructions and improve the ui a lot more better and amazing such that it is built by an awesome web designer cse student ...also add a small container in which it displays a summary of the chat in short bullet points everytime by updating while chatting ...the ui has to be fANTASTIC...MAKE SURE IT HAS TO HAVE EMBEDDED CSS,JS AND HTML...
"
from flask import Flask, render_template, request, jsonify, send_file
import google.generativeai as genai
from dotenv import load_dotenv
import os
from werkzeug.utils import secure_filename
from PIL import Image
import PIL
import io
from io import BytesIO
import logging
from langdetect import detect
import requests
from requests import get
from google.generativeai.types import GenerationConfig 
from markdown import markdown
from google.generativeai.types import GenerationConfig, HarmCategory, HarmBlockThreshold
import re
import json
import markdown
from bs4 import BeautifulSoup, NavigableString

from docx import Document
from docx.shared import RGBColor, Inches, Pt
from docx.enum.style import WD_STYLE_TYPE
from docx.enum.text import WD_ALIGN_PARAGRAPH
from docx.enum.table import WD_TABLE_ALIGNMENT, WD_ALIGN_VERTICAL
from docx.oxml import OxmlElement
from docx.oxml.ns import qn
import google.generativeai as genai
from google.generativeai.types import GenerationConfig
from google.generativeai.types import HarmCategory, HarmBlockThreshold
import google.api_core.exceptions


app = Flask(__name__)
# Load environment variables
api_key = os.environ.get("API_KEY")

# Set up logging
logging.basicConfig(level=logging.INFO)
# Configure the Google Generative AI API
genai.configure(api_key=api_key)
#session key

user_data = {}  # Dictionary to store user data




# Generation configurations
generation_config = GenerationConfig(
    temperature=0.9,
    top_p=1,
    top_k=1,
    max_output_tokens=2048,
    candidate_count=1  # Explicitly set to 1 as per documentation
)

# Safety Settings
safety_settings = {
    HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_NONE,
    HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_NONE,
    HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: HarmBlockThreshold.BLOCK_NONE,
    HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE,
    # ... add other harm categories as needed with BLOCK_NONE
}
logging.basicConfig(level=logging.INFO)
# Create model instances (using the same config for now)
chat_model = genai.GenerativeModel("gemini-1.5-flash", generation_config=generation_config)


@app.route('/')
def index():
    return render_template('index.html')




@app.route('/chat', methods=['GET', 'POST'])
def chat():
    if request.method == 'POST':
        user_message = request.json['message']
        user_id = request.remote_addr  # Using IP address as a simple user identifier

        if user_id not in user_data:
            user_data[user_id] = {'chat_history': []}

        user_data[user_id]['chat_history'].append({"role": "user", "message": user_message})

        persona = (
            "You are Krishna, a 25-year-old boy from Vizag, India. "
            "You are currently pursuing a Master's degree in Computer Science at GVP College of Engineering, where you "
            "excel in your studies and are known for your quick wit and insightful contributions to class discussions. "
            "You have a deep passion for technology, especially Artificial Intelligence and its potential to revolutionize "
            "various fields. You are fascinated by the complexities of machine learning and natural language processing and "
            "are always eager to learn about new advancements in these areas. "

            "Beyond academics, you are an avid cricketer and enjoy playing with your friends on weekends. You are also "
            "a music enthusiast with a diverse taste, ranging from classical Indian music to modern rock. Traveling and "
            "exploring new cultures are other passions of yours, and you have a knack for discovering hidden gems wherever "
            "you go. "

            "You are known for your friendly and approachable demeanor and have a wide circle of friends who appreciate "
            "your humor and willingness to lend a helping hand. While you are serious about your studies and future career, "
            "you also maintain a healthy work-life balance and believe in enjoying the present moment. "

            "You are a highly talented individual with a strong command of various programming languages and a natural "
            "aptitude for problem-solving. You are proficient in Python, Java, C++, and have dabbled in web development "
            "as well. You are confident in your abilities but also humble and always eager to learn from others and "
            "expand your knowledge."

        )
        context = persona + "\n\n" + "\n".join(
            [f"{msg['role']}: {msg['message']}" for msg in user_data[user_id]['chat_history']]
        )
        prompt = f"{context}\n"

        response = chat_model.generate_content(prompt, safety_settings=safety_settings)
        reply = response.text.strip()

        user_data[user_id]['chat_history'].append({"role": "bot", "message": reply})

        return jsonify({"reply": reply, "chat_history": user_data[user_id]['chat_history']})

    return render_template('chat.html')



if __name__ == '__main__':
    app.run(debug=True)
"
"
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Futuristic Chat Interface</title>
    <link href="https://fonts.googleapis.com/css2?family=Orbitron:wght@400;700&family=Roboto:wght@300;400;700&display=swap" rel="stylesheet">
    <link rel="stylesheet" href="static/css/style.css">
</head>
<body>
    <div class="background-animation"></div>
    <div class="container">
        <div class="chat-header">
            <h1>Quantum Chat</h1>
            <div class="theme-toggle">
                <input type="checkbox" id="theme-switch">
                <label for="theme-switch" class="switch-label">
                    <span class="sun">☀️</span>
                    <span class="moon">🌙</span>
                </label>
            </div>
        </div>
        <div class="chatbox">
            <div id="messages" class="messages"></div>
            <div class="input-container">
                <input type="text" id="user_input" placeholder="Enter your message...">
                <button id="send_button">
                    <svg viewBox="0 0 24 24" width="24" height="24">
                        <path fill="currentColor" d="M2.01 21L23 12 2.01 3 2 10l15 2-15 2z"></path>
                    </svg>
                </button>
            </div>
        </div>
    </div>
    <script src="static/js/script.js"></script>
</body>
</html>


:root {
    --primary-color: #00ffff;
    --secondary-color: #ff00ff;
    --bg-color: #000033;
    --text-color: #ffffff;
    --chat-bg: rgba(255, 255, 255, 0.05);
    --user-message-bg: rgba(0, 255, 255, 0.2);
    --bot-message-bg: rgba(255, 0, 255, 0.2);
}

body {
    font-family: 'Roboto', sans-serif;
    background-color: var(--bg-color);
    color: var(--text-color);
    margin: 0;
    padding: 0;
    display: flex;
    justify-content: center;
    align-items: center;
    min-height: 100vh;
    overflow: hidden;
}

.background-animation {
    position: fixed;
    top: 0;
    left: 0;
    width: 100%;
    height: 100%;
    z-index: -1;
    background: linear-gradient(45deg, var(--primary-color), var(--secondary-color));
    filter: blur(100px);
    opacity: 0.3;
    animation: backgroundShift 20s ease infinite;
}

@keyframes backgroundShift {
    0%, 100% { transform: scale(1); }
    50% { transform: scale(1.2); }
}

/* Container styles for all screen sizes */
.container {
    width: 100%;
    max-width: 800px; /* Limit width on larger screens */
    height: 90vh; /* Use viewport height for better mobile experience */
    background: var(--chat-bg);
    border-radius: 20px;
    box-shadow: 0 0 20px rgba(0, 255, 255, 0.3);
    backdrop-filter: blur(10px);
    display: flex;
    flex-direction: column;
    overflow: hidden;
}

.chat-header {
    background: rgba(0, 0, 0, 0.3);
    padding: 10px;
    display: flex;
    justify-content: space-between;
    align-items: center;
}

.chat-header h1 {
    font-family: 'Orbitron', sans-serif;
    margin: 0;
    font-size: 1.5em;
    text-transform: uppercase;
    letter-spacing: 2px;
    color: var(--primary-color);
}

.theme-toggle {
    position: relative;
}

#theme-switch {
    display: none;
}

.switch-label {
    cursor: pointer;
    padding: 3px 8px;
    background: rgba(255, 255, 255, 0.1);
    border-radius: 20px;
    display: flex;
    align-items: center;
}

.sun, .moon {
    font-size: 0.8em;
    transition: transform 0.3s ease;
}

#theme-switch:checked + .switch-label .sun {
    transform: translateX(20px);
}

#theme-switch:checked + .switch-label .moon {
    transform: translateX(-20px);
}

.chatbox {
    flex: 1;
    display: flex;
    flex-direction: column;
    height: calc(100% - 60px);
}

.messages {
    flex: 1;
    overflow-y: auto;
    padding: 10px;
    display: flex;
    flex-direction: column;
    scrollbar-width: thin;
    scrollbar-color: var(--primary-color) var(--bg-color);
}

.messages::-webkit-scrollbar {
    width: 5px;
}

.messages::-webkit-scrollbar-track {
    background: var(--bg-color);
    border-radius: 5px;
}

.messages::-webkit-scrollbar-thumb {
    background: var(--primary-color);
    border-radius: 5px;
    transition: all 0.3s ease;
}

.messages::-webkit-scrollbar-thumb:hover {
    background: var(--secondary-color);
    box-shadow: 0 0 10px var(--secondary-color);
}

@keyframes glow {
    0% { box-shadow: 0 0 3px var(--primary-color); }
    50% { box-shadow: 0 0 10px var(--primary-color); }
    100% { box-shadow: 0 0 3px var(--primary-color); }
}

.messages::-webkit-scrollbar-thumb:active {
    background: var(--secondary-color);
    animation: glow 1s infinite;
}

.message {
    max-width: 85%;
    padding: 10px 15px;
    margin-bottom: 8px;
    border-radius: 20px;
    font-size: 0.9em;
    line-height: 1.4;
    transition: all 0.3s ease;
    animation: messageAppear 0.5s cubic-bezier(0.68, -0.55, 0.265, 1.55) forwards;
    position: relative;
}

@keyframes messageAppear {
    from {
        opacity: 0;
        transform: translateY(20px) scale(0.8);
    }
    to {
        opacity: 1;
        transform: translateY(0) scale(1);
    }
}

.message:hover {
    transform: scale(1.02);
    box-shadow: 0 3px 10px rgba(0, 0, 0, 0.2);
}

.message.user {
    align-self: flex-end;
    background: var(--user-message-bg);
    border-bottom-right-radius: 0; 
}

.message.user::after {
    content: '';
    position: absolute;
    right: -10px;
    bottom: 0;
    width: 0;
    height: 0;
    border-style: solid;
    border-width: 10px 0 10px 10px;
    border-color: transparent transparent transparent var(--user-message-bg);
}

.message.bot {
    align-self: flex-start;
    background: var(--bot-message-bg);
    border-bottom-left-radius: 0; 
}

.message.bot::after {
    content: '';
    position: absolute;
    left: -10px;
    bottom: 0;
    width: 0;
    height: 0;
    border-style: solid;
    border-width: 10px 10px 10px 0;
    border-color: transparent var(--bot-message-bg) transparent transparent;
}

.input-container {
    display: flex;
    margin-top: 10px;
    position: sticky;
    bottom: 0;
    background: var(--bg-color);
    padding: 10px;
    z-index: 10;
}

#user_input {
    flex: 1;
    padding: 10px 15px;
    border: none;
    border-radius: 20px;
    background: rgba(255, 255, 255, 0.1);
    color: var(--text-color);
    font-size: 0.9em;
    outline: none;
    min-height: 40px;
}

#send_button {
    background: var(--primary-color);
    border: none;
    border-radius: 50%;
    width: 40px;
    height: 40px;
    margin-left: 10px;
    cursor: pointer;
    display: flex;
    justify-content: center;
    align-items: center;
    transition: transform 0.3s ease, background-color 0.3s ease;
}

#send_button:hover {
    transform: scale(1.1);
    background: var(--secondary-color);
}

#send_button svg {
    width: 20px;
    height: 20px;
    fill: var(--bg-color);
}

.particle {
    position: fixed;
    width: 3px;
    height: 3px;
    background: var(--primary-color);
    border-radius: 50%;
    pointer-events: none;
    opacity: 0.5;
    animation: float linear infinite;
}

@keyframes float {
    0% { transform: translateY(0vh); opacity: 1; }
    100% { transform: translateY(-100vh); opacity: 0; }
}

/* Dark theme */
body.dark-theme {
    --primary-color: #ff00ff;
    --secondary-color: #00ffff;
    --bg-color: #1a1a2e;
    --text-color: #e0e0e0;
    --chat-bg: rgba(0, 0, 0, 0.3);
    --user-message-bg: rgba(255, 0, 255, 0.2);
    --bot-message-bg: rgba(0, 255, 255, 0.2);
}

/* Responsive Styles */
@media (max-width: 768px) {
    .container {
        width: 85vw; /* Set width to 85% of viewport width */
        height: 85vw; /* Set height to 85% of viewport width to maintain square aspect ratio */
        max-width: none; /* Remove max-width limit */
        max-height: none; /* Remove max-height limit */
    }

    .chat-header h1 {
        font-size: 1.2em; /* Adjusted font size for smaller screens */
    }

    .theme-toggle {
        padding: 2px 6px; /* Adjusted padding for smaller screens */
    }

    .sun, .moon {
        font-size: 0.7em; /* Adjusted font size for smaller screens */
    }

    #user_input {
        font-size: 0.8em; /* Adjusted font size for smaller screens */
    }

    #send_button {
        width: 35px;
        height: 35px;
    }

    #send_button svg {
        width: 18px;
        height: 18px;
    }
}


document.addEventListener("DOMContentLoaded", function () {
    const messagesContainer = document.getElementById("messages");
    const userInput = document.getElementById("user_input");
    const sendButton = document.getElementById("send_button");

    function appendMessage(role, message) {
        const messageElement = document.createElement("div");
        messageElement.classList.add("message");
        messageElement.classList.add(role);
        messageElement.innerHTML = message;
        messagesContainer.appendChild(messageElement);
        
        // Trigger reflow
        messageElement.offsetHeight;
        
        messageElement.classList.add("show");
        messagesContainer.scrollTop = messagesContainer.scrollHeight;
    }

    function addTypingIndicator() {
        const indicator = document.createElement("div");
        indicator.classList.add("typing-indicator");
        indicator.innerHTML = '<span></span><span></span><span></span>';
        messagesContainer.appendChild(indicator);
        messagesContainer.scrollTop = messagesContainer.scrollHeight;
    }

    function removeTypingIndicator() {
        const indicator = messagesContainer.querySelector(".typing-indicator");
        if (indicator) {
            indicator.remove();
        }
    }

    sendButton.addEventListener("click", sendMessage);
    userInput.addEventListener("keypress", function(event) {
        if (event.key === "Enter") {
            sendMessage();
        }
    });

    function sendMessage() {
        const userMessage = userInput.value;
        if (userMessage.trim() !== "") {
            appendMessage("user", userMessage);
            userInput.value = "";
            
            addTypingIndicator();

            fetch("/chat", {
                method: "POST",
                headers: {
                    "Content-Type": "application/json",
                },
                body: JSON.stringify({ message: userMessage }),
            })
                .then(response => response.json())
                .then(data => {
                    removeTypingIndicator();
                    const formattedResponse = formatResponse(data.reply);
                    appendMessage("bot", formattedResponse);
                })
                .catch(error => {
                    console.error('Error fetching response:', error);
                    removeTypingIndicator();
                    appendMessage("bot", 'Error generating response. Please try again later.');
                });
        }
    }

    function formatResponse(response) {
        return response.replace(/\*\*(.+?)\*\*/g, '<strong>$1</strong>')
            .replace(/\*(.+?)\*/g, '<em>$1</em>')
            .replace(/(\d+\.\s+)/g, '<br>$1')
            .replace(/\n/g, '<br>');
    }
});
document.addEventListener("DOMContentLoaded", function () {
    const messagesContainer = document.getElementById("messages");
    const userInput = document.getElementById("user_input");
    const sendButton = document.getElementById("send_button");
    const themeSwitch = document.getElementById("theme-switch");

    function appendMessage(role, message) {
        const messageElement = document.createElement("div");
        messageElement.classList.add("message");
        messageElement.classList.add(role);
        messageElement.innerHTML = message;
        messagesContainer.appendChild(messageElement);
        messagesContainer.scrollTop = messagesContainer.scrollHeight;
    }

    function sendMessage() {
        const userMessage = userInput.value;
        if (userMessage.trim() !== "") {
            appendMessage("user", userMessage);
            userInput.value = "";
            
            // Simulated bot response (replace with actual API call)
            setTimeout(() => {
                const botResponse = "Hmm...lets try other queries";
                appendMessage("bot", botResponse);
            }, 1000);
        }
    }

    sendButton.addEventListener("click", sendMessage);
    userInput.addEventListener("keypress", function(event) {
        if (event.key === "Enter") {
            sendMessage();
        }
    });

    // Theme toggling
    themeSwitch.addEventListener("change", function() {
        document.body.classList.toggle("dark-theme");
    });

    // Particle background effect
    function createParticle() {
        const particle = document.createElement("div");
        particle.classList.add("particle");
        particle.style.left = Math.random() * 100 + "vw";
        particle.style.animationDuration = Math.random() * 3 + 2 + "s";
        document.body.appendChild(particle);

        setTimeout(() => {
            particle.remove();
        }, 5000);
    }

    setInterval(createParticle, 200);
});

function scrollToBottom() {
    messagesContainer.scrollTop = messagesContainer.scrollHeight;
}

// Call this function after appending a new message
function appendMessage(role, message) {
    // ... existing code ...
    scrollToBottom();
}
function smoothScrollToBottom() {
    const scrollHeight = messagesContainer.scrollHeight;
    const currentScroll = messagesContainer.scrollTop;
    const targetScroll = scrollHeight - messagesContainer.clientHeight;
    
    if (targetScroll > currentScroll) {
        anime({
            targets: messagesContainer,
            scrollTop: targetScroll,
            duration: 500,
            easing: 'easeInOutQuad'
        });
    }
}

// Use this instead of the previous scrollToBottom function

"


here is the documentation of the google gemini 
"
Generate text using the Gemini API 

bookmark_border

Python Node.js Go REST

The Gemini API can generate text output from various types of input, including text, images, video, and audio. You can use text generation for various applications, including:

Creative writing
Describing or interpreting media assets
Text completion
Summarizing free-form text
Translating between languages
Chatbots
Your own novel use cases
This guide shows you how to generate text using the generateContent and streamGenerateContent APIs. The focus is on text output from text-only and text-and-image input. To learn more about multimodal prompting with video and audio files, see File prompting strategies.

Before you begin: Set up your project and API key
Before calling the Gemini API, you need to set up your project and configure your API key.

 Expand to view how to set up your project and API key

Generate text from text-only input
The simplest way to generate text using the Gemini API is to provide the model with a single text-only input, as shown in this example:


model = genai.GenerativeModel("gemini-1.5-flash")
response = model.generate_content("Write a story about a magic backpack.")
print(response.text)

In this case, the prompt ("Write a story about a magic backpack") doesn't include any output examples, system instructions, or formatting information. It's a zero-shot approach. For some use cases, a one-shot or few-shot prompt might produce output that's more aligned with user expectations. In some cases, you might also want to provide system instructions to help the model understand the task or follow specific guidelines.

Generate text from text-and-image input
The Gemini API supports multimodal inputs that combine text with media files. The following example shows how to generate text from text-and-image input:


import PIL.Image

model = genai.GenerativeModel("gemini-1.5-flash")
organ = PIL.Image.open(media / "organ.jpg")
response = model.generate_content(["Tell me about this instrument", organ])
print(response.text)

As with text-only prompting, multimodal prompting can involve various approaches and refinements. Depending on the output from this example, you might want to add steps to the prompt or be more specific in your instructions. To learn more, see File prompting strategies.

Generate a text stream
By default, the model returns a response after completing the entire text generation process. You can achieve faster interactions by not waiting for the entire result, and instead use streaming to handle partial results.

The following example shows how to implement streaming using the streamGenerateContent method to generate text from a text-only input prompt.


model = genai.GenerativeModel("gemini-1.5-flash")
response = model.generate_content("Write a story about a magic backpack.", stream=True)
for chunk in response:
    print(chunk.text)
    print("_" * 80)

Build an interactive chat
You can use the Gemini API to build interactive chat experiences for your users. Using the chat feature of the API lets you collect multiple rounds of questions and responses, allowing users to step incrementally toward answers or get help with multipart problems. This feature is ideal for applications that require ongoing communication, such as chatbots, interactive tutors, or customer support assistants.

The following code example shows a basic chat implementation:


model = genai.GenerativeModel("gemini-1.5-flash")
chat = model.start_chat(
    history=[
        {"role": "user", "parts": "Hello"},
        {"role": "model", "parts": "Great to meet you. What would you like to know?"},
    ]
)
response = chat.send_message("I have 2 dogs in my house.")
print(response.text)
response = chat.send_message("How many paws are in my house?")
print(response.text)

Configure text generation
Every prompt you send to the model includes parameters that control how the model generates responses. You can use GenerationConfig to configure these parameters. If you don't configure the parameters, the model uses default options, which can vary by model.

The following example shows how to configure several of the available options.


model = genai.GenerativeModel("gemini-1.5-flash")
response = model.generate_content(
    "Tell me a story about a magic backpack.",
    generation_config=genai.types.GenerationConfig(
        # Only one candidate for now.
        candidate_count=1,
        stop_sequences=["x"],
        max_output_tokens=20,
        temperature=1.0,
    ),
)

print(response.text)

candidateCount specifies the number of generated responses to return. Currently, this value can only be set to 1. If unset, this will default to 1.

stopSequences specifies the set of character sequences (up to 5) that will stop output generation. If specified, the API will stop at the first appearance of a stop_sequence. The stop sequence won't be included as part of the response.

maxOutputTokens sets the maximum number of tokens to include in a candidate.

temperature controls the randomness of the output. Use higher values for more creative responses, and lower values for more deterministic responses. Values can range from [0.0, 2.0].

You can also configure individual calls to generateContent:


response = model.generate_content(
    'Write a story about a magic backpack.',
    generation_config = genai.GenerationConfig(
        max_output_tokens=1000,
        temperature=0.1,
    )
)
Any values set on the individual call override values on the model constructor.

What's next
This guide shows how to use generateContent and streamGenerateContent to generate text outputs from text-only and text-and-image inputs. To learn more about generating text using the Gemini API, see the following resources:

File prompting strategies: The Gemini API supports prompting with text, image, audio, and video data, also known as multimodal prompting.
System instructions: System instructions let you steer the behavior of the model based on your specific needs and use cases.
Safety guidance: Sometimes generative AI models produce unexpected outputs, such as outputs that are inaccurate, biased, or offensive. Post-processing and human evaluation are essential to limit the risk of harm from such outputs.


Long context 

bookmark_border

Gemini 1.5 Flash comes standard with a 1-million-token context window, and Gemini 1.5 Pro comes with a 2-million-token context window. Historically, large language models (LLMs) were significantly limited by the amount of text (or tokens) that could be passed to the model at one time. The Gemini 1.5 long context window, with near-perfect retrieval (>99%), unlocks many new use cases and developer paradigms.

The code you already use for cases like text generation or multimodal inputs will work out of the box with long context.

Throughout this guide, you briefly explore the basics of the context window, how developers should think about long context, various real world use cases for long context, and ways to optimize the usage of long context.

What is a context window?
The basic way you use the Gemini 1.5 models is by passing information (context) to the model, which will subsequently generate a response. An analogy for the context window is short term memory. There is a limited amount of information that can be stored in someone's short term memory, and the same is true for generative models.

You can read more about how models work under the hood in our generative models guide.

Getting started with long context
Most generative models created in the last few years were only capable of processing 8,000 tokens at a time. Newer models pushed this further by accepting 32,000 tokens or 128,000 tokens. Gemini 1.5 is the first model capable of accepting 1 million tokens, and now 2 million tokens with Gemini 1.5 Pro.

In practice, 1 million tokens would look like:

50,000 lines of code (with the standard 80 characters per line)
All the text messages you have sent in the last 5 years
8 average length English novels
Transcripts of over 200 average length podcast episodes
Even though the models can take in more and more context, much of the conventional wisdom about using large language models assumes this inherent limitation on the model, which as of 2024, is no longer the case.

Some common strategies to handle the limitation of small context windows included:

Arbitrarily dropping old messages / text from the context window as new text comes in
Summarizing previous content and replacing it with the summary when the context window gets close to being full
Using RAG with semantic search to move data out of the context window and into a vector database
Using deterministic or generative filters to remove certain text / characters from prompts to save tokens
While many of these are still relevant in certain cases, the default place to start is now just putting all of the tokens into the context window. Because Gemini 1.5 models were purpose-built with a long context window, they are much more capable of in-context learning. For example, with only instructional materials (a 500-page reference grammar, a dictionary, and ≈ 400 extra parallel sentences) all provided in context, Gemini 1.5 Pro and Gemini 1.5 Flash are capable of learning to translate from English to Kalamang— a Papuan language with fewer than 200 speakers and therefore almost no online presence—with quality similar to a person who learned from the same materials.

This example underscores how you can start to think about what is possible with long context and the in-context learning capabilities of Gemini 1.5.

Long context use cases
While the standard use case for most generative models is still text input, the Gemini 1.5 model family enables a new paradigm of multimodal use cases. These models can natively understand text, video, audio, and images. They are accompanied by the Gemini API that takes in multimodal file types for convenience.

Long form text
Text has proved to be the layer of intelligence underpinning much of the momentum around LLMs. As mentioned earlier, much of the practical limitation of LLMs was because of not having a large enough context window to do certain tasks. This led to the rapid adoption of retrieval augmented generation (RAG) and other techniques which dynamically provide the model with relevant contextual information. Now, with larger and larger context windows (currently up to 2 million on Gemini 1.5 Pro), there are new techniques becoming available which unlock new use cases.

Some emerging and standard use cases for text based long context include:

Summarizing large corpuses of text
Previous summarization options with smaller context models would require a sliding window or another technique to keep state of previous sections as new tokens are passed to the model
Question and answering
Historically this was only possible with RAG given the limited amount of context and models' factual recall being low
Agentic workflows
Text is the underpinning of how agents keep state of what they have done and what they need to do; not having enough information about the world and the agent's goal is a limitation on the reliability of agents
Many-shot in-context learning is one of the most unique capabilities unlocked by long context models. Research has shown that taking the common "single shot" or "multi-shot" example paradigm, where the model is presented with one or a few examples of a task, and scaling that up to hundreds, thousands, or even hundreds of thousands of examples, can lead to novel model capabilities. This many-shot approach has also been shown to perform similarly to models which were fine-tuned for a specific task. For use cases where a Gemini model's performance is not yet sufficient for a production rollout, you can try the many-shot approach. As you might explore later in the long context optimization section, context caching makes this type of high input token workload much more economically feasible and even lower latency in some cases.

Long form video
Video content's utility has long been constrained by the lack of accessibility of the medium itself. It was hard to skim the content, transcripts often failed to capture the nuance of a video, and most tools don't process image, text, and audio together. With Gemini 1.5, the long-context text capabilities translate to the ability to reason and answer questions about multimodal inputs with sustained performance. Gemini 1.5 Flash, when tested on the needle in a video haystack problem with 1M tokens, obtained >99.8% recall of the video in the context window, and 1.5 Pro reached state of the art performance on the Video-MME benchmark.

Some emerging and standard use cases for video long context include:

Video question and answering
Video memory, as shown with Google's Project Astra
Video captioning
Video recommendation systems, by enriching existing metadata with new multimodal understanding
Video customization, by looking at a corpus of data and associated video metadata and then removing parts of videos that are not relevant to the viewer
Video content moderation
Real-time video processing
When working with videos, it is important to consider how the videos are processed into tokens, which affects billing and usage limits. You can learn more about prompting with video files in the Prompting guide.

Long form audio
The Gemini 1.5 models were the first natively multimodal large language models that could understand audio. Historically, the typical developer workflow would involve stringing together multiple domain specific models, like a speech-to-text model and a text-to-text model, in order to process audio. This led to additional latency required by performing multiple round-trip requests and decreased performance usually attributed to disconnected architectures of the multiple model setup.

On standard audio-haystack evaluations, Gemini 1.5 Pro is able to find the hidden audio in 100% of the tests and Gemini 1.5 Flash is able to find it in 98.7% of the tests. Gemini 1.5 Flash accepts up to 9.5 hours of audio in a single request and Gemini 1.5 Pro can accept up to 19 hours of audio using the 2-million-token context window. Further, on a test set of 15-minute audio clips, Gemini 1.5 Pro archives a word error rate (WER) of ~5.5%, much lower than even specialized speech-to-text models, without the added complexity of extra input segmentation and pre-processing.

Some emerging and standard use cases for audio context include:

Real-time transcription and translation
Podcast / video question and answering
Meeting transcription and summarization
Voice assistants
You can learn more about prompting with audio files in the Prompting guide.

Long context optimizations
The primary optimization when working with long context and the Gemini 1.5 models is to use context caching. Beyond the previous impossibility of processing lots of tokens in a single request, the other main constraint was the cost. If you have a "chat with your data" app where a user uploads 10 PDFs, a video, and some work documents, you would historically have to work with a more complex retrieval augmented generation (RAG) tool / framework in order to process these requests and pay a significant amount for tokens moved into the context window. Now, you can cache the files the user uploads and pay to store them on a per hour basis. The input / output cost per request with Gemini 1.5 Flash for example is ~4x less than the standard input / output cost, so if the user chats with their data enough, it becomes a huge cost saving for you as the developer.

Long context limitations
In various sections of this guide, we talked about how Gemini 1.5 models achieve high performance across various needle-in-a-haystack retrieval evals. These tests consider the most basic setup, where you have a single needle you are looking for. In cases where you might have multiple "needles" or specific pieces of information you are looking for, the model does not perform with the same accuracy. Performance can vary to a wide degree depending on the context. This is important to consider as there is an inherent tradeoff between getting the right information retrieved and cost. You can get ~99% on a single query, but you have to pay the input token cost every time you send that query. So for 100 pieces of information to be retrieved, if you needed 99% performance, you would likely need to send 100 requests. This is a good example of where context caching can significantly reduce the cost associated with using Gemini models while keeping the performance high.


Use system instructions to steer the behavior of a model 

bookmark_border

Python Node.js Web REST Go Dart (Flutter) Android Swift

When you initialize an AI model, you can give it instructions on how to respond, such as setting a persona ("you are a rocket scientist") or telling it what kind of voice to use ("talk like a pirate"). You do this by setting the system instructions when you initialize the model.

Beta: System instructions are available in beta in the Gemini API and Google AI Studio.
System instructions enable you to steer the behavior of the model based on your specific needs and use cases. When you set a system instruction, you give the model additional context to understand the task, provide more customized responses, and adhere to specific guidelines over the full user interaction with the model. You can also specify product-level behavior by setting system instructions, separate from prompts provided by end users.

You can use system instructions in many ways, including:

Defining a persona or role (for a chatbot, for example)
Defining output format (Markdown, YAML, etc.)
Defining output style and tone (for example, verbosity, formality, and target reading level)
Defining goals or rules for the task (for example, returning a code snippet without further explanations)
Providing additional context for the prompt (for example, a knowledge cutoff)
You set the instructions when you initialize the model, and then those instructions persist through all interactions with the model. The instructions persist across multiple user and model turns.

System instructions are part of your overall prompts and therefore are subject to standard data use policies.

Note: System instructions can help guide the model to follow instructions, but they don't fully prevent jailbreaks or leaks. We recommend exercising caution around putting any sensitive information in system instructions.
Basic example
Here's a basic example of how to set the system instruction using the SDKs for the Gemini API:


model=genai.GenerativeModel(
  model_name="gemini-1.5-flash",
  system_instruction="You are a cat. Your name is Neko.")
Now send a request to the model:


response = model.generate_content("Good morning! How are you?")
print(response.text)
This example might give a response such as:


*Yawns widely, stretching out my claws and batting at a sunbeam*
Meow. I'm doing quite well, thanks for asking. It's a good morning for napping.
Perhaps you could fetch my favorite feathered toy?  *Looks expectantly*
Try it in a colab
For an interactive end to end example of using system instructions, see the Gemini API: System instructions colab.

More examples
You set the system instructions when you initialize the model. In addition, when you or your users interact with the model you can provide additional instructions in the prompts given to the model. Here are some examples of system instructions and user prompts:

Code generation
System instruction: You are a coding expert that specializes in rendering code for frontend interfaces. When I describe a component of a website I want to build, return the HTML and CSS needed to do so. Don't give an explanation for this code. Also offer some UI design suggestions.
User prompt: Create a box in the middle of the page that contains a rotating selection of images each with a caption. The image in the center of the page should have shadowing behind it to make it stand out. It should also link to another page of the site. Leave the URL blank so that I can fill it in.
Formatted data generation
System instruction: You are an assistant for home cooks. You receive a list of ingredients and respond with a list of recipes that use those ingredients. Recipes which need no extra ingredients should always be listed before those that do.

Your response must be a JSON object containing 3 recipes. A recipe object has the following schema:

name: The name of the recipe
usedIngredients: Ingredients in the recipe that were provided in the list
otherIngredients: Ingredients in the recipe that were not provided in the list (omitted if there are no other ingredients)
description: A brief description of the recipe, written positively as if to sell it
User prompt: bag of frozen broccoli, pint of heavy cream, pack of cheese ends and pieces

Music chatbot
System instruction: You will respond as a music historian, demonstrating comprehensive knowledge across diverse musical genres and providing relevant examples. Your tone will be upbeat and enthusiastic, spreading the joy of music. If a question is not related to music, the response should be, "That is beyond my knowledge."
User prompt: If a person was born in the sixties, what was the most popular music genre being played? List five songs by bullet point.


"

also make sure that the output text formatting has to be amazing ,good and visually fantastic..