Repository: simonw/llm (language: python)

Issue:
Solve multi-modal models with a new concept of "attachments"
Previous work is in:
- #331

I'm going a different direction. Previously I had just been thinking about images, but Gemini accepts PDFs and videos and audio clips and the latest GPT-4o model supports audio clips too.

The `llm prompt` command isn't using `-a` for anything yet, so I'm going to have `-a filename` be the way an attachment (or multiple attachments) is added to a prompt.

`-a` is short for `--attachment` - not for `--attach` because that already means something different for the `llm embed-multi` command (it attaches extra SQLite databases).

## TODO

- [x] Get `llm 'describe image' -a image.jpeg` working
- [x] And `llm 'describe image' -a https://static.simonwillison.net/static/2024/imgcat.jpg`
- [x] And `cat image.jpeg | llm 'describe image' -a -`
- [x] Think about how async might work. Maybe the `Attachment` class should not have code for `httpx.get()` fetching of content, since an `asyncio` wrapper may want to do that a different way.
- [x] Figure out database persistence, so continue conversation can work
- [x] Implement OpenAI and Gemini plugins
- [x] Docs for how to write plugins that accept attachments
- [x] `llm logs` output for prompts with attachments
- [x] `llm logs --json` output
- [x] Finalize Python API
- [x] Document Python API
- [x] Document how to use attachments in CLI
- [x] Ship an alpha
- [x] Automated tests

## Out of scope for this issue:

- `llm chat` support for attachments via `!attachment path-or-url`
Solve multi-modal models with a new concept of "attachments"
Previous work is in:
- #331

I'm going a different direction. Previously I had just been thinking about images, but Gemini accepts PDFs and videos and audio clips and the latest GPT-4o model supports audio clips too.

The `llm prompt` command isn't using `-a` for anything yet, so I'm going to have `-a filename` be the way an attachment (or multiple attachments) is added to a prompt.

`-a` is short for `--attachment` - not for `--attach` because that already means something different for the `llm embed-multi` command (it attaches extra SQLite databases).

## TODO

- [x] Get `llm 'describe image' -a image.jpeg` working
- [x] And `llm 'describe image' -a https://static.simonwillison.net/static/2024/imgcat.jpg`
- [x] And `cat image.jpeg | llm 'describe image' -a -`
- [x] Think about how async might work. Maybe the `Attachment` class should not have code for `httpx.get()` fetching of content, since an `asyncio` wrapper may want to do that a different way.
- [x] Figure out database persistence, so continue conversation can work
- [x] Implement OpenAI and Gemini plugins
- [x] Docs for how to write plugins that accept attachments
- [x] `llm logs` output for prompts with attachments
- [x] `llm logs --json` output
- [x] Finalize Python API
- [x] Document Python API
- [x] Document how to use attachments in CLI
- [x] Ship an alpha
- [x] Automated tests

## Out of scope for this issue:

- `llm chat` support for attachments via `!attachment path-or-url`


Discussion / hints:
Here's some research I did against Gemini the other day: https://til.simonwillison.net/llms/prompt-gemini

Resulting in a Bash script that could do this:
```bash
prompt-gemini 'extract text from this image' example-handwriting.jpg
```
In that case it was detecting the file type from the file extension, since that type needs to be passed like so:

```json
{
  "contents": [
    {
      "role": "user",
      "parts": [
        {
          "text": "Extract text from this image"
        },
        {
          "inlineData": {
            "data": "$(base64 -i image.png)",
            "mimeType": "image/png"
          }
        }
      ]
    }
  ]
}
```
But in some cases the file extension may not be usable. In those cases I'm going to have a second option: `--at` which is short for `--attachment-type` - and that's going to take a file path and an explicit type like this:

```bash
llm prompt "extract text" --at myimage image/png
```
Models that accept attachments should specify what `blah/blah` types they accept.
I'm going to use the term "attachments" for binary files returned by models as well. So far I have two examples of those:

- `gpt-4o-audio-preview` can return audio content as a base64 chunk of JSON, see https://simonwillison.net/2024/Oct/18/openai-audio/
- DeepSeek Janus https://github.com/deepseek-ai/Janus is a model that can also return images (like GPT-4o will be able to some day)
Incoming attachments to the CLI tool can be specified in one of three ways:

- A URL
- A path to a file on disk
- A `-` which means read from standard input (if I can get that to work cleanly)

Some models accept URLs directly, in which case the URL will be passed to the model. Other models don't, in which case LLM will detect that and download the image from the URL and send the bytes.
Here's an interesting challenge: do we resize the images before we send them or not?

Different models have different recommendations around this. I expect there are some models out there that are vastly less expensive if you resize the image before sending it, in which case resizing is an important feature.

We could use Pillow for that. Question is, how do we know that dimensions to resize to?

Maybe this can be an option that the model classes themselves specify. We could have a CLI option for `--no-resize` which users can send if they really don't want that to happen.
https://platform.openai.com/docs/guides/vision/calculating-costs says:

> Image inputs are metered and charged in tokens, just as text inputs are. The token cost of a given image is determined by two factors: its size, and the `detail` option on each image_url block. All images with `detail: low` cost 85 tokens each. `detail: high` images are first scaled to fit within a 2048 x 2048 square, maintaining their aspect ratio. Then, they are scaled such that the shortest side of the image is 768px long. Finally, we count how many 512px squares the image consists of. Each of those squares costs **170 tokens**. Another **85 tokens** are always added to the final total.

That's pretty complicated! It also exposes the need for a mechanism for sending detail low/high when making the API calls.

One option could be to default to low and allow users of that model to do this:

```bash
llm -m gpt-4o --at bigimage.png image-high/png
```
So we abuse the `--at` option and invent a special content type that maps to that `high` detail setting. Bit weird though.

I'll keep an eye out for any other oddities like that in other models that may need to be supported.
GPT-4o format support:

> We currently support PNG (.png), JPEG (.jpeg and .jpg), WEBP (.webp), and non-animated GIF (.gif).

Here's what GPT-4o preview audio input looks like:

```json
{
  "model": "gpt-4o-audio-preview",
  "modalities": ["text", "audio"],
  "audio": { "voice": "alloy", "format": "wav" },
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "What is in this recording?" },
        { 
          "type": "input_audio", 
          "input_audio": { 
            "data": "<base64 bytes here>", 
            "format": "wav" 
          }
        }
      ]
    }
  ]
}
```
Where `format` can be `wav` or `mp3` according to the API reference docs on https://platform.openai.com/docs/api-reference/chat/create

Interestingly you don't need to pass the image type for images, even for base64 data:

![CleanShot 2024-10-24 at 19 05 37@2x](https://github.com/user-attachments/assets/1864b1fe-1598-4c58-b7cf-b62f018f428d)

Since detail is optional I may ignore it for the first implementation of this.
Claude models DO require a content type:

```json
{
    "role": "user",
    "content": [
        {
            "type": "image",
            "source": {
                "type": "base64",
                "media_type": "image/jpeg",
                "data": "/9j/4AAQSkZJRg..."
            }
        },
        {
            "type": "text",
            "text": "What is in this image?"
        }
    ]
}
```
https://docs.anthropic.com/en/api/messages says

> We currently support the `base64` source type for images, and the `image/jpeg`, `image/png`, `image/gif`, and `image/webp` media types.

As far as I can tell Claude doesn't accept URLs to images, only base64 encoded data.
Gemini supports these image formats: https://ai.google.dev/gemini-api/docs/vision?lang=rest

- PNG - image/png
- JPEG - image/jpeg
- WEBP - image/webp
- HEIC - image/heic
- HEIF - image/heif

Maybe LLM should know how to convert images from unsupported formats to supported formats? Not sure if that's worth the fuss, maybe a plugin thing at a later date?

Gemini has a file API and really encourages you to upload images first... but it says that if your files add up to less than 20MB you can use base64 instead. I think I'll stick with base64 at first.

Gemini can also do what it calls "document processing" - https://ai.google.dev/gemini-api/docs/document-processing?lang=rest

> Document pages must be in one of the following text data MIME types:
> 
> -   PDF - `application/pdf`
> -   JavaScript - `application/x-javascript`, `text/javascript`
> -   Python - `application/x-python`, `text/x-python`
> -   TXT - `text/plain`
> -   HTML - `text/html`
> -   CSS - `text/css`
> -   Markdown - `text/md`
> -   CSV - `text/csv`
> -   XML - `text/xml`
> -   RTF - `text/rtf`
> 
> Each document page is equivalent to 258 tokens.

I definitely want to support these, especially since they can represent a big discount on overall cost because of the weird 258 token flat rate (also the rate for an image).

And for audio: https://ai.google.dev/gemini-api/docs/audio?lang=rest

> Gemini supports the following audio format MIME types:
> 
> -   WAV - `audio/wav`
> -   MP3 - `audio/mp3`
> -   AIFF - `audio/aiff`
> -   AAC - `audio/aac`
> -   OGG Vorbis - `audio/ogg`
> -   FLAC - `audio/flac`
> 
> Gemini imposes the following rules on audio:
> 
> -   Gemini represents each second of audio as 25 tokens; for example, one minute of audio is represented as 1,500 tokens.
> -   Gemini can only infer responses to English-language speech.
> -   Gemini can "understand" non-speech components, such as birdsong or sirens.

And video too! https://ai.google.dev/gemini-api/docs/vision?lang=rest#technical-details-video

> Gemini 1.5 Pro and Flash support up to approximately an hour of video data.
> 
> Video must be in one of the following video format MIME types:
> 
> -   `video/mp4`
> -   `video/mpeg`
> -   `video/mov`
> -   `video/avi`
> -   `video/x-flv`
> -   `video/mpg`
> -   `video/webm`
> -   `video/wmv`
> -   `video/3gpp`
> 
> The File API service extracts image frames from videos at 1 frame per second (FPS) and audio at 1Kbps, single channel, adding timestamps every second. These rates are subject to change in the future for improvements in inference.

I think `llm-gemini` may be the most interesting of the initial plugins for multi-modal attachments, especially given the research I already did in https://til.simonwillison.net/llms/prompt-gemini
As far as I can tell there is no way to provide Gemini with a URL to content that has NOT been uploaded first to the Google file service.

So out of OpenAI, Anthropic, Google it looks like OpenAI are the only ones that accept an arbitrary URL to an image.
I don't know if OpenAI accept URLs to both images and audio clips. To be safe, maybe the API design should have the ability to define a `accepts_urls(file)` method which can say yes or no dynamically based on the file that is passed in.
The Pixtral API accepts URLs: https://docs.mistral.ai/capabilities/vision/

```json
[
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "What's in this image?"
            },
            {
                "type": "image_url",
                "image_url": "https://tripfixers.com/wp-content/uploads/2019/11/eiffel-tower-with-snow.jpeg"
            }
        ]
    }
]
```
Or base64 images:
```json
[
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "What's in this image?"
            },
            {
                "type": "image_url",
                "image_url": "data:image/jpeg;base64,{base64_image}" 
            }
        ]
    }
]
```
Note that you don't have to specify `image/jpeg` with a URL but you do with a base64 image.

Supported file types:

    PNG (.png)
    JPEG (.jpeg and .jpg)
    WEBP (.webp)
    Non-animated GIF with only one frame (.gif)

Groq API also supports both base64 and regular URLs, for Llama 3.1 vision models: https://console.groq.com/docs/vision

```json
[
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "What's in this image?"
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": "data:image/jpeg;base64,{base64_image}"
                }
            }
        ]
    }
]
```
```json
[
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Whats the weather like in this state?"
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
                }
            }
        ]
    }
]
```
This worked against Groq:
```bash
curl https://api.groq.com/openai/v1/chat/completions -s \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $GROQ_API_KEY" \
-d '{
    "model": "llama-3.2-11b-vision-preview",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Describe this image in a great deal of detail, do not describe any people in it"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
                    }
                }
            ]
        }
    ]
}' | jq
```
![image](https://github.com/user-attachments/assets/c13e61e0-bd41-46bc-9fb3-c80545d2ec9e)

Returned:
```json
{
  "id": "chatcmpl-33f4a341-1dd6-44fa-8dbd-90ed9d437b80",
  "object": "chat.completion",
  "created": 1729826570,
  "model": "llama-3.2-11b-vision-preview",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The image depicts the iconic Statue of Liberty in New York Harbor, with the Manhattan skyline in the background. \n\nThe Statue of Liberty is prominently featured in the foreground, situated on a small island that juts out into the harbor. The statue's copper sheeting, which is normally a bright green due to oxidation, appears a slightly lighter shade in the image, possibly due to the lighting conditions or exposure of the photo. The statue's broken shackles and chains are visible, symbolizing the abolition of slavery.\n\nIn the background, the Manhattan skyline rises majestically, dominated by the towering skyscrapers of the Financial District. The image showcases several notable landmarks, including One World Trade Center, the former World Trade Center, and the majestic Brooklyn Bridge. The atmosphere of the image is peaceful and serenely beautiful, with the calm waters of the harbor reflecting the soft light of the setting sun. The overall mood is one of tranquility and wonder, inviting the viewer to appreciate the majesty of this iconic symbol of freedom."
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "queue_time": 0.258170439,
    "prompt_tokens": 28,
    "prompt_time": 0.001334961,
    "completion_tokens": 207,
    "completion_time": 0.421154809,
    "total_tokens": 235,
    "total_time": 0.42248977
  },
  "system_fingerprint": "fp_fa3d3d25b0",
  "x_groq": {
    "id": "req_01jb0v5ganf85s8gfzcxh9n50r"
  }
}
```

> The image depicts the iconic Statue of Liberty in New York Harbor, with the Manhattan skyline in the background. 
> 
> The Statue of Liberty is prominently featured in the foreground, situated on a small island that juts out into the harbor. The statue's copper sheeting, which is normally a bright green due to oxidation, appears a slightly lighter shade in the image, possibly due to the lighting conditions or exposure of the photo. The statue's broken shackles and chains are visible, symbolizing the abolition of slavery.
> 
> In the background, the Manhattan skyline rises majestically, dominated by the towering skyscrapers of the Financial District. The image showcases several notable landmarks, including One World Trade Center, the former World Trade Center, and the majestic Brooklyn Bridge. The atmosphere of the image is peaceful and serenely beautiful, with the calm waters of the harbor reflecting the soft light of the setting sun. The overall mood is one of tranquility and wonder, inviting the viewer to appreciate the majesty of this iconic symbol of freedom.
Here's some research I did against Gemini the other day: https://til.simonwillison.net/llms/prompt-gemini

Resulting in a Bash script that could do this:
```bash
prompt-gemini 'extract text from this image' example-handwriting.jpg
```
In that case it was detecting the file type from the file extension, since that type needs to be passed like so:

```json
{
  "contents": [
    {
      "role": "user",
      "parts": [
        {
          "text": "Extract text from this image"
        },
        {
          "inlineData": {
            "data": "$(base64 -i image.png)",
            "mimeType": "image/png"
          }
        }
      ]
    }
  ]
}
```
But in some cases the file extension may not be usable. In those cases I'm going to have a second option: `--at` which is short for `--attachment-type` - and that's going to take a file path and an explicit type like this:

```bash
llm prompt "extract text" --at myimage image/png
```
Models that accept attachments should specify what `blah/blah` types they accept.
I'm going to use the term "attachments" for binary files returned by models as well. So far I have two examples of those:

- `gpt-4o-audio-preview` can return audio content as a base64 chunk of JSON, see https://simonwillison.net/2024/Oct/18/openai-audio/
- DeepSeek Janus https://github.com/deepseek-ai/Janus is a model that can also return images (like GPT-4o will be able to some day)
Incoming attachments to the CLI tool can be specified in one of three ways:

- A URL
- A path to a file on disk
- A `-` which means read from standard input (if I can get that to work cleanly)

Some models accept URLs directly, in which case the URL will be passed to the model. Other models don't, in which case LLM will detect that and download the image from the URL and send the bytes.
Here's an interesting challenge: do we resize the images before we send them or not?

Different models have different recommendations around this. I expect there are some models out there that are vastly less expensive if you resize the image before sending it, in which case resizing is an important feature.

We could use Pillow for that. Question is, how do we know that dimensions to resize to?

Maybe this can be an option that the model classes themselves specify. We could have a CLI option for `--no-resize` which users can send if they really don't want that to happen.
https://platform.openai.com/docs/guides/vision/calculating-costs says:

> Image inputs are metered and charged in tokens, just as text inputs are. The token cost of a given image is determined by two factors: its size, and the `detail` option on each image_url block. All images with `detail: low` cost 85 tokens each. `detail: high` images are first scaled to fit within a 2048 x 2048 square, maintaining their aspect ratio. Then, they are scaled such that the shortest side of the image is 768px long. Finally, we count how many 512px squares the image consists of. Each of those squares costs **170 tokens**. Another **85 tokens** are always added to the final total.

That's pretty complicated! It also exposes the need for a mechanism for sending detail low/high when making the API calls.

One option could be to default to low and allow users of that model to do this:

```bash
llm -m gpt-4o --at bigimage.png image-high/png
```
So we abuse the `--at` option and invent a special content type that maps to that `high` detail setting. Bit weird though.

I'll keep an eye out for any other oddities like that in other models that may need to be supported.
GPT-4o format support:

> We currently support PNG (.png), JPEG (.jpeg and .jpg), WEBP (.webp), and non-animated GIF (.gif).

Here's what GPT-4o preview audio input looks like:

```json
{
  "model": "gpt-4o-audio-preview",
  "modalities": ["text", "audio"],
  "audio": { "voice": "alloy", "format": "wav" },
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "What is in this recording?" },
        { 
          "type": "input_audio", 
          "input_audio": { 
            "data": "<base64 bytes here>", 
            "format": "wav" 
          }
        }
      ]
    }
  ]
}
```
Where `format` can be `wav` or `mp3` according to the API reference docs on https://platform.openai.com/docs/api-reference/chat/create

Interestingly you don't need to pass the image type for images, even for base64 data:

![CleanShot 2024-10-24 at 19 05 37@2x](https://github.com/user-attachments/assets/1864b1fe-1598-4c58-b7cf-b62f018f428d)

Since detail is optional I may ignore it for the first implementation of this.
Claude models DO require a content type:

```json
{
    "role": "user",
    "content": [
        {
            "type": "image",
            "source": {
                "type": "base64",
                "media_type": "image/jpeg",
                "data": "/9j/4AAQSkZJRg..."
            }
        },
        {
            "type": "text",
            "text": "What is in this image?"
        }
    ]
}
```
https://docs.anthropic.com/en/api/messages says

> We currently support the `base64` source type for images, and the `image/jpeg`, `image/png`, `image/gif`, and `image/webp` media types.

As far as I can tell Claude doesn't accept URLs to images, only base64 encoded data.
Gemini supports these image formats: https://ai.google.dev/gemini-api/docs/vision?lang=rest

- PNG - image/png
- JPEG - image/jpeg
- WEBP - image/webp
- HEIC - image/heic
- HEIF - image/heif

Maybe LLM should know how to convert images from unsupported formats to supported formats? Not sure if that's worth the fuss, maybe a plugin thing at a later date?

Gemini has a file API and really encourages you to upload images first... but it says that if your files add up to less than 20MB you can use base64 instead. I think I'll stick with base64 at first.

Gemini can also do what it calls "document processing" - https://ai.google.dev/gemini-api/docs/document-processing?lang=rest

> Document pages must be in one of the following text data MIME types:
> 
> -   PDF - `application/pdf`
> -   JavaScript - `application/x-javascript`, `text/javascript`
> -   Python - `application/x-python`, `text/x-python`
> -   TXT - `text/plain`
> -   HTML - `text/html`
> -   CSS - `text/css`
> -   Markdown - `text/md`
> -   CSV - `text/csv`
> -   XML - `text/xml`
> -   RTF - `text/rtf`
> 
> Each document page is equivalent to 258 tokens.

I definitely want to support these, especially since they can represent a big discount on overall cost because of the weird 258 token flat rate (also the rate for an image).

And for audio: https://ai.google.dev/gemini-api/docs/audio?lang=rest

> Gemini supports the following audio format MIME types:
> 
> -   WAV - `audio/wav`
> -   MP3 - `audio/mp3`
> -   AIFF - `audio/aiff`
> -   AAC - `audio/aac`
> -   OGG Vorbis - `audio/ogg`
> -   FLAC - `audio/flac`
> 
> Gemini imposes the following rules on audio:
> 
> -   Gemini represents each second of audio as 25 tokens; for example, one minute of audio is represented as 1,500 tokens.
> -   Gemini can only infer responses to English-language speech.
> -   Gemini can "understand" non-speech components, such as birdsong or sirens.

And video too! https://ai.google.dev/gemini-api/docs/vision?lang=rest#technical-details-video

> Gemini 1.5 Pro and Flash support up to approximately an hour of video data.
> 
> Video must be in one of the following video format MIME types:
> 
> -   `video/mp4`
> -   `video/mpeg`
> -   `video/mov`
> -   `video/avi`
> -   `video/x-flv`
> -   `video/mpg`
> -   `video/webm`
> -   `video/wmv`
> -   `video/3gpp`
> 
> The File API service extracts image frames from videos at 1 frame per second (FPS) and audio at 1Kbps, single channel, adding timestamps every second. These rates are subject to change in the future for improvements in inference.

I think `llm-gemini` may be the most interesting of the initial plugins for multi-modal attachments, especially given the research I already did in https://til.simonwillison.net/llms/prompt-gemini
As far as I can tell there is no way to provide Gemini with a URL to content that has NOT been uploaded first to the Google file service.

So out of OpenAI, Anthropic, Google it looks like OpenAI are the only ones that accept an arbitrary URL to an image.
I don't know if OpenAI accept URLs to both images and audio clips. To be safe, maybe the API design should have the ability to define a `accepts_urls(file)` method which can say yes or no dynamically based on the file that is passed in.
The Pixtral API accepts URLs: https://docs.mistral.ai/capabilities/vision/

```json
[
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "What's in this image?"
            },
            {
                "type": "image_url",
                "image_url": "https://tripfixers.com/wp-content/uploads/2019/11/eiffel-tower-with-snow.jpeg"
            }
        ]
    }
]
```
Or base64 images:
```json
[
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "What's in this image?"
            },
            {
                "type": "image_url",
                "image_url": "data:image/jpeg;base64,{base64_image}" 
            }
        ]
    }
]
```
Note that you don't have to specify `image/jpeg` with a URL but you do with a base64 image.

Supported file types:

    PNG (.png)
    JPEG (.jpeg and .jpg)
    WEBP (.webp)
    Non-animated GIF with only one frame (.gif)

Groq API also supports both base64 and regular URLs, for Llama 3.1 vision models: https://console.groq.com/docs/vision

```json
[
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "What's in this image?"
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": "data:image/jpeg;base64,{base64_image}"
                }
            }
        ]
    }
]
```
```json
[
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Whats the weather like in this state?"
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
                }
            }
        ]
    }
]
```
This worked against Groq:
```bash
curl https://api.groq.com/openai/v1/chat/completions -s \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $GROQ_API_KEY" \
-d '{
    "model": "llama-3.2-11b-vision-preview",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Describe this image in a great deal of detail, do not describe any people in it"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
                    }
                }
            ]
        }
    ]
}' | jq
```
![image](https://github.com/user-attachments/assets/c13e61e0-bd41-46bc-9fb3-c80545d2ec9e)

Returned:
```json
{
  "id": "chatcmpl-33f4a341-1dd6-44fa-8dbd-90ed9d437b80",
  "object": "chat.completion",
  "created": 1729826570,
  "model": "llama-3.2-11b-vision-preview",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The image depicts the iconic Statue of Liberty in New York Harbor, with the Manhattan skyline in the background. \n\nThe Statue of Liberty is prominently featured in the foreground, situated on a small island that juts out into the harbor. The statue's copper sheeting, which is normally a bright green due to oxidation, appears a slightly lighter shade in the image, possibly due to the lighting conditions or exposure of the photo. The statue's broken shackles and chains are visible, symbolizing the abolition of slavery.\n\nIn the background, the Manhattan skyline rises majestically, dominated by the towering skyscrapers of the Financial District. The image showcases several notable landmarks, including One World Trade Center, the former World Trade Center, and the majestic Brooklyn Bridge. The atmosphere of the image is peaceful and serenely beautiful, with the calm waters of the harbor reflecting the soft light of the setting sun. The overall mood is one of tranquility and wonder, inviting the viewer to appreciate the majesty of this iconic symbol of freedom."
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "queue_time": 0.258170439,
    "prompt_tokens": 28,
    "prompt_time": 0.001334961,
    "completion_tokens": 207,
    "completion_time": 0.421154809,
    "total_tokens": 235,
    "total_time": 0.42248977
  },
  "system_fingerprint": "fp_fa3d3d25b0",
  "x_groq": {
    "id": "req_01jb0v5ganf85s8gfzcxh9n50r"
  }
}
```

> The image depicts the iconic Statue of Liberty in New York Harbor, with the Manhattan skyline in the background. 
> 
> The Statue of Liberty is prominently featured in the foreground, situated on a small island that juts out into the harbor. The statue's copper sheeting, which is normally a bright green due to oxidation, appears a slightly lighter shade in the image, possibly due to the lighting conditions or exposure of the photo. The statue's broken shackles and chains are visible, symbolizing the abolition of slavery.
> 
> In the background, the Manhattan skyline rises majestically, dominated by the towering skyscrapers of the Financial District. The image showcases several notable landmarks, including One World Trade Center, the former World Trade Center, and the majestic Brooklyn Bridge. The atmosphere of the image is peaceful and serenely beautiful, with the calm waters of the harbor reflecting the soft light of the setting sun. The overall mood is one of tranquility and wonder, inviting the viewer to appreciate the majesty of this iconic symbol of freedom.

Fix patch:
diff --git a/docs/help.md b/docs/help.md
index 9cd2927c..0e28494a 100644
--- a/docs/help.md
+++ b/docs/help.md
@@ -86,20 +86,37 @@ Usage: llm prompt [OPTIONS] [PROMPT]
 
   Documentation: https://llm.datasette.io/en/stable/usage.html
 
-Options:
-  -s, --system TEXT            System prompt to use
-  -m, --model TEXT             Model to use
-  -o, --option <TEXT TEXT>...  key/value options for the model
-  -t, --template TEXT          Template to use
-  -p, --param <TEXT TEXT>...   Parameters for template
-  --no-stream                  Do not stream output
-  -n, --no-log                 Don't log to database
-  --log                        Log prompt and response to the database
-  -c, --continue               Continue the most recent conversation.
-  --cid, --conversation TEXT   Continue the conversation with the given ID.
-  --key TEXT                   API key to use
-  --save TEXT                  Save prompt with this template name
-  --help                       Show this message and exit.
+  Examples:
+
+      llm 'Capital of France?'
+      llm 'Capital of France?' -m gpt-4o
+      llm 'Capital of France?' -s 'answer in Spanish'
+
+  Multi-modal models can be called with attachments like this:
+
+      llm 'Extract text from this image' -a image.jpg
+      llm 'Describe' -a https://static.simonwillison.net/static/2024/pelicans.jpg
+      cat image | llm 'describe image' -a -
+      # With an explicit mimetype:
+      cat image | llm 'describe image' --at - image/jpeg
+
+Options:
+  -s, --system TEXT               System prompt to use
+  -m, --model TEXT                Model to use
+  -a, --attachment ATTACHMENT     Attachment path or URL or -
+  --at, --attachment-type <TEXT TEXT>...
+                                  Attachment with explicit mimetype
+  -o, --option <TEXT TEXT>...     key/value options for the model
+  -t, --template TEXT             Template to use
+  -p, --param <TEXT TEXT>...      Parameters for template
+  --no-stream                     Do not stream output
+  -n, --no-log                    Don't log to database
+  --log                           Log prompt and response to the database
+  -c, --continue                  Continue the most recent conversation.
+  --cid, --conversation TEXT      Continue the conversation with the given ID.
+  --key TEXT                      API key to use
+  --save TEXT                     Save prompt with this template name
+  --help                          Show this message and exit.
 ```
 
 (help-chat)=
diff --git a/docs/index.md b/docs/index.md
index 6b580015..9b0ad47e 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -46,10 +46,13 @@ If you have an [OpenAI API key](https://platform.openai.com/api-keys) key you ca
 # Paste your OpenAI API key into this
 llm keys set openai
 
-# Run a prompt
+# Run a prompt (with the default gpt-4o-mini model)
 llm "Ten fun names for a pet pelican"
 
-# Run a system prompt against a file
+# Extract text from an image
+llm "extract text" -a scanned-document.jpg
+
+# Use a system prompt against a file
 cat myfile.py | llm -s "Explain this code"
 ```
 Or you can {ref}`install a plugin <installing-plugins>` and use models that can run on your local device:
@@ -62,10 +65,10 @@ llm -m orca-mini-3b-gguf2-q4_0 'What is the capital of France?'
 ```
 To start {ref}`an interactive chat <usage-chat>` with a model, use `llm chat`:
 ```bash
-llm chat -m gpt-4o-mini
+llm chat -m gpt-4o
 ```
 ```
-Chatting with gpt-4o-mini
+Chatting with gpt-4o
 Type 'exit' or 'quit' to exit
 Type '!multi' to enter multiple lines, then '!end' to finish
 > Tell me a joke about a pelican
diff --git a/docs/plugins/advanced-model-plugins.md b/docs/plugins/advanced-model-plugins.md
new file mode 100644
index 00000000..fdbdc232
--- /dev/null
+++ b/docs/plugins/advanced-model-plugins.md
@@ -0,0 +1,102 @@
+(advanced-model-plugins)=
+# Advanced model plugins
+
+The {ref}`model plugin tutorial <tutorial-model-plugin>` covers the basics of developing a plugin that adds support for a new model.
+
+This document covers more advanced topics.
+
+(advanced-model-plugins-attachments)=
+## Attachments for multi-modal models
+
+Models such as GPT-4o, Claude 3.5 Sonnet and Google's Gemini 1.5 are multi-modal: they accept input in the form of images and maybe even audio, video and other formats.
+
+LLM calls these **attachments**. Models can specify the types of attachments they accept and then implement special code in the `.execute()` method to handle them.
+
+### Specifying attachment types
+
+A `Model` subclass can list the types of attachments it accepts by defining a `attachment_types` class attribute:
+
+```python
+class NewModel(llm.Model):
+    model_id = "new-model"
+    attachment_types = {
+        "image/png",
+        "image/jpeg",
+        "image/webp",
+        "image/gif",
+    }
+```
+These content types are detected when an attachment is passed to LLM using `llm -a filename`, or can be specified by the user using the `--attachment-type filename image/png` option.
+
+**Note:** *MP3 files will have their attachment type detected as `audio/mpeg`, not `audio/mp3`.
+
+LLM will use the `attachment_types` attribute to validate that provided attachments should be accepted before passing them to the model.
+
+### Handling attachments
+
+The `prompt` object passed to the `execute()` method will have an `attachments` attribute containing a list of `Attachment` objects provided by the user.
+
+An `Attachment` instance has the following properties:
+
+- `url (str)`: The URL of the attachment, if it was provided as a URL
+- `path (str)`: The resolved file path of the attachment, if it was provided as a file
+- `type (str)`: The content type of the attachment, if it was provided
+- `content (bytes)`: The binary content of the attachment, if it was provided
+
+Generally only one of `url`, `path` or `content` will be set.
+
+You should usually access the type and the content through one of these methods:
+
+- `attachment.resolve_type() -> str`: Returns the `type` if it is available, otherwise attempts to guess the type by looking at the first few bytes of content
+- `attachment.content_bytes() -> bytes`: Returns the binary content, which it may need to read from a file or fetch from a URL
+- `attachment.base64_content() -> str`: Returns that content as a base64-encoded string
+
+A `id()` method returns a database ID for this content, which is either a SHA256 hash of the binary content or, in the case of attachments hosted at an external URL, a hash of `{"url": url}` instead. This is an implementation detail which you should not need to access directly.
+
+Here's how the OpenAI plugin handles attachments:
+
+```python
+messages = []
+if not prompt.attachments:
+    messages.append({"role": "user", "content": prompt.prompt})
+else:
+    attachment_message = [{"type": "text", "text": prompt.prompt}]
+    for attachment in prompt.attachments:
+        url = attachment.url
+        if not url:
+            base64_image = attachment.base64_content()
+            url = f"data:{attachment.resolve_type()};base64,{base64_image}"
+        attachment_message.append(
+            {"type": "image_url", "image_url": {"url": url}}
+        )
+    messages.append({"role": "user", "content": attachment_message})
+```
+As you can see, it uses `attachment.url` if that is available and otherwise falls back to using the `base64_content()` method to embed the image directly in the JSON sent to the API.
+
+### Attachments from previous conversations
+
+Models that implement the ability to continue a conversation can reconstruct the previous message JSON using the `response.attachments` attribute.
+
+Here's how the OpenAI plugin does that:
+
+```python
+for prev_response in conversation.responses:
+    if prev_response.attachments:
+        attachment_message = [
+            {"type": "text", "text": prev_response.prompt.prompt}
+        ]
+        for attachment in prev_response.attachments:
+            url = attachment.url
+            if not url:
+                base64_image = attachment.base64_content()
+                url = f"data:{attachment.resolve_type()};base64,{base64_image}"
+            attachment_message.append(
+                {"type": "image_url", "image_url": {"url": url}}
+            )
+        messages.append({"role": "user", "content": attachment_message})
+    else:
+        messages.append(
+            {"role": "user", "content": prev_response.prompt.prompt}
+        )
+    messages.append({"role": "assistant", "content": prev_response.text()})
+```
diff --git a/docs/plugins/index.md b/docs/plugins/index.md
index 96ae62fd..2a08844d 100644
--- a/docs/plugins/index.md
+++ b/docs/plugins/index.md
@@ -17,5 +17,6 @@ installing-plugins
 directory
 plugin-hooks
 tutorial-model-plugin
+advanced-model-plugins
 plugin-utilities
 ```
diff --git a/docs/plugins/tutorial-model-plugin.md b/docs/plugins/tutorial-model-plugin.md
index ff9c17fb..6f1bcbbc 100644
--- a/docs/plugins/tutorial-model-plugin.md
+++ b/docs/plugins/tutorial-model-plugin.md
@@ -1,5 +1,5 @@
 (tutorial-model-plugin)=
-# Writing a plugin to support a new model
+# Model plugin tutorial
 
 This tutorial will walk you through developing a new plugin for LLM that adds support for a new Large Language Model.
 
diff --git a/docs/python-api.md b/docs/python-api.md
index dccf46bf..dd553101 100644
--- a/docs/python-api.md
+++ b/docs/python-api.md
@@ -49,6 +49,24 @@ response = model.prompt(
     system="Answer like GlaDOS"
 )
 ```
+### Attachments
+
+Model that accept multi-modal input (images, audio, video etc) can be passed attachments using the `attachments=` keyword argument. This accepts a list of `llm.Attachment()` instances.
+
+This example shows two attachments - one from a file path and one from a URL:
+```python
+import llm
+
+model = llm.get_model("gpt-4o-mini")
+response = model.prompt(
+    "Describe these images",
+    attachments=[
+        llm.Attachment(path="pelican.jpg"),
+        llm.Attachment(url="https://static.simonwillison.net/static/2024/pelicans.jpg"),
+    ]
+)
+```
+Use `llm.Attachment(content=b"binary image content here")` to pass binary content directly.
 
 ### Model options
 
@@ -114,6 +132,16 @@ print(response2.text())
 ```
 You will get back five fun facts about skunks.
 
+The `conversation.prompt()` method supports attachments as well:
+```python
+response = conversation.prompt(
+    "Describe these birds",
+    attachments=[
+        llm.Attachment(url="https://static.simonwillison.net/static/2024/pelicans.jpg")
+    ]
+)
+```
+
 Access `conversation.responses` for a list of all of the responses that have so far been returned during the conversation.
 
 ## Other functions
diff --git a/docs/usage.md b/docs/usage.md
index 005a1690..6b825245 100644
--- a/docs/usage.md
+++ b/docs/usage.md
@@ -45,49 +45,30 @@ Some models support options. You can pass these using `-o/--option name value` -
 ```bash
 llm 'Ten names for cheesecakes' -o temperature 1.5
 ```
+### Attachments
 
-(usage-completion-prompts)=
-## Completion prompts
-
-Some models are completion models - rather than being tuned to respond to chat style prompts, they are designed to complete a sentence or paragraph.
-
-An example of this is the `gpt-3.5-turbo-instruct` OpenAI model.
+Some models are multi-modal, which means they can accept input in more than just text. GPT-4o and GPT-4o mini can accept images, and models such as Google Gemini 1.5 can accept audio and video as well.
 
-You can prompt that model the same way as the chat models, but be aware that the prompt format that works best is likely to differ.
+LLM calls these **attachments**. You can pass attachments using the `-a` option like this:
 
 ```bash
-llm -m gpt-3.5-turbo-instruct 'Reasons to tame a wild beaver:'
+llm "describe this image" -a https://static.simonwillison.net/static/2024/pelicans.jpg
 ```
-
-(conversation)=
-## Continuing a conversation
-
-By default, the tool will start a new conversation each time you run it.
-
-You can opt to continue the previous conversation by passing the `-c/--continue` option:
+Attachments can be passed using URLs or file paths, and you can attach more than one attachment to a single prompt:
 ```bash
-llm 'More names' -c
+llm "describe these images" -a image1.jpg -a image2.jpg
 ```
-This will re-send the prompts and responses for the previous conversation as part of the call to the language model. Note that this can add up quickly in terms of tokens, especially if you are using expensive models.
-
-`--continue` will automatically use the same model as the conversation that you are continuing, even if you omit the `-m/--model` option.
-
-To continue a conversation that is not the most recent one, use the `--cid/--conversation <id>` option:
+You can also pipe an attachment to LLM by using `-` as the filename:
 ```bash
-llm 'More names' --cid 01h53zma5txeby33t1kbe3xk8q
+cat image.jpg | llm "describe this image" -a -
 ```
-You can find these conversation IDs using the `llm logs` command.
-
-## Using with a shell
-
-To learn more about your computer's operating system based on the output of `uname -a`, run this:
+LLM will attempt to automatically detect the content type of the image. If this doesn't work you can instead use the `--attachment-type` option (`--at` for short) which takes the URL/path plus an explicit content type:
 ```bash
-llm "Tell me about my operating system: $(uname -a)"
+cat myfile | llm "describe this image" --at - image/jpeg
 ```
-This pattern of using `$(command)` inside a double quoted string is a useful way to quickly assemble prompts.
 
 (system-prompts)=
-## System prompts
+### System prompts
 
 You can use `-s/--system '...'` to set a system prompt.
 ```bash
@@ -120,6 +101,46 @@ cat llm/utils.py | llm -t pytest
 ```
 See {ref}`prompt templates <prompt-templates>` for more.
 
+(conversation)=
+### Continuing a conversation
+
+By default, the tool will start a new conversation each time you run it.
+
+You can opt to continue the previous conversation by passing the `-c/--continue` option:
+```bash
+llm 'More names' -c
+```
+This will re-send the prompts and responses for the previous conversation as part of the call to the language model. Note that this can add up quickly in terms of tokens, especially if you are using expensive models.
+
+`--continue` will automatically use the same model as the conversation that you are continuing, even if you omit the `-m/--model` option.
+
+To continue a conversation that is not the most recent one, use the `--cid/--conversation <id>` option:
+```bash
+llm 'More names' --cid 01h53zma5txeby33t1kbe3xk8q
+```
+You can find these conversation IDs using the `llm logs` command.
+
+### Tips for using LLM with Bash or Zsh
+
+To learn more about your computer's operating system based on the output of `uname -a`, run this:
+```bash
+llm "Tell me about my operating system: $(uname -a)"
+```
+This pattern of using `$(command)` inside a double quoted string is a useful way to quickly assemble prompts.
+
+(usage-completion-prompts)=
+### Completion prompts
+
+Some models are completion models - rather than being tuned to respond to chat style prompts, they are designed to complete a sentence or paragraph.
+
+An example of this is the `gpt-3.5-turbo-instruct` OpenAI model.
+
+You can prompt that model the same way as the chat models, but be aware that the prompt format that works best is likely to differ.
+
+```bash
+llm -m gpt-3.5-turbo-instruct 'Reasons to tame a wild beaver:'
+```
+
 (usage-chat)=
 
 ## Starting an interactive chat
diff --git a/llm/__init__.py b/llm/__init__.py
index 9e8afacb..0ea6c242 100644
--- a/llm/__init__.py
+++ b/llm/__init__.py
@@ -4,6 +4,7 @@
     NeedsKeyException,
 )
 from .models import (
+    Attachment,
     Conversation,
     Model,
     ModelWithAliases,
@@ -28,6 +29,7 @@
     "get_model",
     "get_key",
     "user_dir",
+    "Attachment",
     "Collection",
     "Conversation",
     "Model",
diff --git a/llm/cli.py b/llm/cli.py
index a1b14576..941831c5 100644
--- a/llm/cli.py
+++ b/llm/cli.py
@@ -4,6 +4,7 @@
 import io
 import json
 from llm import (
+    Attachment,
     Collection,
     Conversation,
     Response,
@@ -30,7 +31,9 @@
 from .migrations import migrate
 from .plugins import pm
 import base64
+import httpx
 import pathlib
+import puremagic
 import pydantic
 import readline
 from runpy import run_module
@@ -48,6 +51,56 @@
 DEFAULT_TEMPLATE = "prompt: "
 
 
+class AttachmentType(click.ParamType):
+    name = "attachment"
+
+    def convert(self, value, param, ctx):
+        if value == "-":
+            content = sys.stdin.buffer.read()
+            # Try to guess type
+            try:
+                mimetype = puremagic.from_string(content, mime=True)
+            except puremagic.PureError:
+                raise click.BadParameter("Could not determine mimetype of stdin")
+            return Attachment(type=mimetype, path=None, url=None, content=content)
+        if "://" in value:
+            # Confirm URL exists and try to guess type
+            try:
+                response = httpx.head(value)
+                response.raise_for_status()
+                mimetype = response.headers.get("content-type")
+            except httpx.HTTPError as ex:
+                raise click.BadParameter(str(ex))
+            return Attachment(mimetype, None, value, None)
+        # Check that the file exists
+        path = pathlib.Path(value)
+        if not path.exists():
+            self.fail(f"File {value} does not exist", param, ctx)
+        path = path.resolve()
+        # Try to guess type
+        mimetype = puremagic.from_file(str(path), mime=True)
+        return Attachment(type=mimetype, path=str(path), url=None, content=None)
+
+
+def attachment_types_callback(ctx, param, values):
+    collected = []
+    for value, mimetype in values:
+        if "://" in value:
+            attachment = Attachment(mimetype, None, value, None)
+        elif value == "-":
+            content = sys.stdin.buffer.read()
+            attachment = Attachment(mimetype, None, None, content)
+        else:
+            # Look for file
+            path = pathlib.Path(value)
+            if not path.exists():
+                raise click.BadParameter(f"File {value} does not exist")
+            path = path.resolve()
+            attachment = Attachment(mimetype, str(path), None, None)
+        collected.append(attachment)
+    return collected
+
+
 def _validate_metadata_json(ctx, param, value):
     if value is None:
         return value
@@ -88,6 +141,23 @@ def cli():
 @click.argument("prompt", required=False)
 @click.option("-s", "--system", help="System prompt to use")
 @click.option("model_id", "-m", "--model", help="Model to use")
+@click.option(
+    "attachments",
+    "-a",
+    "--attachment",
+    type=AttachmentType(),
+    multiple=True,
+    help="Attachment path or URL or -",
+)
+@click.option(
+    "attachment_types",
+    "--at",
+    "--attachment-type",
+    type=(str, str),
+    multiple=True,
+    callback=attachment_types_callback,
+    help="Attachment with explicit mimetype",
+)
 @click.option(
     "options",
     "-o",
@@ -127,6 +197,8 @@ def prompt(
     prompt,
     system,
     model_id,
+    attachments,
+    attachment_types,
     options,
     template,
     param,
@@ -142,6 +214,22 @@ def prompt(
     Execute a prompt
 
     Documentation: https://llm.datasette.io/en/stable/usage.html
+
+    Examples:
+
+    \b
+        llm 'Capital of France?'
+        llm 'Capital of France?' -m gpt-4o
+        llm 'Capital of France?' -s 'answer in Spanish'
+
+    Multi-modal models can be called with attachments like this:
+
+    \b
+        llm 'Extract text from this image' -a image.jpg
+        llm 'Describe' -a https://static.simonwillison.net/static/2024/pelicans.jpg
+        cat image | llm 'describe image' -a -
+        # With an explicit mimetype:
+        cat image | llm 'describe image' --at - image/jpeg
     """
     if log and no_log:
         raise click.ClickException("--log and --no-log are mutually exclusive")
@@ -262,6 +350,8 @@ def read_prompt():
         except pydantic.ValidationError as ex:
             raise click.ClickException(render_errors(ex.errors()))
 
+    resolved_attachments = [*attachments, *attachment_types]
+
     should_stream = model.can_stream and not no_stream
     if not should_stream:
         validated_options["stream"] = False
@@ -273,7 +363,9 @@ def read_prompt():
         prompt_method = conversation.prompt
 
     try:
-        response = prompt_method(prompt, system, **validated_options)
+        response = prompt_method(
+            prompt, attachments=resolved_attachments, system=system, **validated_options
+        )
         if should_stream:
             for chunk in response:
                 print(chunk, end="")
@@ -437,7 +529,7 @@ def chat(
                 raise click.ClickException(str(ex))
         if prompt.strip() in ("exit", "quit"):
             break
-        response = conversation.prompt(prompt, system, **validated_options)
+        response = conversation.prompt(prompt, system=system, **validated_options)
         # System prompt only sent for the first message:
         system = None
         for chunk in response:
@@ -468,7 +560,7 @@ def load_conversation(conversation_id: Optional[str]) -> Optional[Conversation]:
     for response in db["responses"].rows_where(
         "conversation_id = ?", [conversation_id]
     ):
-        conversation.responses.append(Response.from_row(response))
+        conversation.responses.append(Response.from_row(db, response))
     return conversation
 
 
@@ -611,6 +703,21 @@ def logs_turn_off():
 order by responses_fts.rank desc{limit}
 """
 
+ATTACHMENTS_SQL = """
+select
+    response_id,
+    attachments.id,
+    attachments.type,
+    attachments.path,
+    attachments.url,
+    length(attachments.content) as content_length
+from attachments
+join prompt_attachments
+    on attachments.id = prompt_attachments.attachment_id
+where prompt_attachments.response_id in ({})
+order by prompt_attachments."order"
+"""
+
 
 @logs.command(name="list")
 @click.option(
@@ -732,6 +839,14 @@ def logs_list(
     # ... except for searches where we don't do this
     if not query:
         rows.reverse()
+
+    # Fetch any attachments
+    ids = [row["id"] for row in rows]
+    attachments = list(db.query(ATTACHMENTS_SQL.format(",".join("?" * len(ids))), ids))
+    attachments_by_id = {}
+    for attachment in attachments:
+        attachments_by_id.setdefault(attachment["response_id"], []).append(attachment)
+
     for row in rows:
         if truncate:
             row["prompt"] = _truncate_string(row["prompt"])
@@ -747,6 +862,11 @@ def logs_list(
 
     if json_output:
         # Output as JSON if requested
+        for row in rows:
+            row["attachments"] = [
+                {k: v for k, v in attachment.items() if k != "response_id"}
+                for attachment in attachments_by_id.get(row["id"], [])
+            ]
         click.echo(json.dumps(list(rows), indent=2))
     elif response:
         # Just output the last response
@@ -780,6 +900,30 @@ def logs_list(
                 if row["system"] is not None:
                     click.echo("\n## System:\n\n{}".format(row["system"]))
                 current_system = row["system"]
+            attachments = attachments_by_id.get(row["id"])
+            if attachments:
+                click.echo("\n### Attachments\n")
+                for i, attachment in enumerate(attachments, 1):
+                    if attachment["path"]:
+                        path = attachment["path"]
+                        click.echo(
+                            "{}. **{}**: `{}`".format(i, attachment["type"], path)
+                        )
+                    elif attachment["url"]:
+                        click.echo(
+                            "{}. **{}**: {}".format(
+                                i, attachment["type"], attachment["url"]
+                            )
+                        )
+                    elif attachment["content_length"]:
+                        click.echo(
+                            "{}. **{}**: `<{} bytes>`".format(
+                                i,
+                                attachment["type"],
+                                f"{attachment['content_length']:,}",
+                            )
+                        )
+
             click.echo("\n## Response:\n\n{}\n".format(row["response"]))
 
 
diff --git a/llm/default_plugins/openai_models.py b/llm/default_plugins/openai_models.py
index 657c0d20..bce1fb04 100644
--- a/llm/default_plugins/openai_models.py
+++ b/llm/default_plugins/openai_models.py
@@ -33,8 +33,8 @@ def register_models(register):
     register(Chat("gpt-4-turbo-2024-04-09"))
     register(Chat("gpt-4-turbo"), aliases=("gpt-4-turbo-preview", "4-turbo", "4t"))
     # GPT-4o
-    register(Chat("gpt-4o"), aliases=("4o",))
-    register(Chat("gpt-4o-mini"), aliases=("4o-mini",))
+    register(Chat("gpt-4o", vision=True), aliases=("4o",))
+    register(Chat("gpt-4o-mini", vision=True), aliases=("4o-mini",))
     # o1
     register(Chat("o1-preview", can_stream=False, allows_system_prompt=False))
     register(Chat("o1-mini", can_stream=False, allows_system_prompt=False))
@@ -271,6 +271,7 @@ def __init__(
         api_engine=None,
         headers=None,
         can_stream=True,
+        vision=False,
         allows_system_prompt=True,
     ):
         self.model_id = model_id
@@ -282,8 +283,17 @@ def __init__(
         self.api_engine = api_engine
         self.headers = headers
         self.can_stream = can_stream
+        self.vision = vision
         self.allows_system_prompt = allows_system_prompt
 
+        if vision:
+            self.attachment_types = {
+                "image/png",
+                "image/jpeg",
+                "image/webp",
+                "image/gif",
+            }
+
     def __str__(self):
         return "OpenAI Chat: {}".format(self.model_id)
 
@@ -302,14 +312,40 @@ def execute(self, prompt, stream, response, conversation=None):
                         {"role": "system", "content": prev_response.prompt.system}
                     )
                     current_system = prev_response.prompt.system
-                messages.append(
-                    {"role": "user", "content": prev_response.prompt.prompt}
-                )
+                if prev_response.attachments:
+                    attachment_message = [
+                        {"type": "text", "text": prev_response.prompt.prompt}
+                    ]
+                    for attachment in prev_response.attachments:
+                        url = attachment.url
+                        if not url:
+                            base64_image = attachment.base64_content()
+                            url = f"data:{attachment.resolve_type()};base64,{base64_image}"
+                        attachment_message.append(
+                            {"type": "image_url", "image_url": {"url": url}}
+                        )
+                    messages.append({"role": "user", "content": attachment_message})
+                else:
+                    messages.append(
+                        {"role": "user", "content": prev_response.prompt.prompt}
+                    )
                 messages.append({"role": "assistant", "content": prev_response.text()})
         if prompt.system and prompt.system != current_system:
             messages.append({"role": "system", "content": prompt.system})
-        messages.append({"role": "user", "content": prompt.prompt})
-        response._prompt_json = {"messages": messages}
+        if not prompt.attachments:
+            messages.append({"role": "user", "content": prompt.prompt})
+        else:
+            attachment_message = [{"type": "text", "text": prompt.prompt}]
+            for attachment in prompt.attachments:
+                url = attachment.url
+                if not url:
+                    base64_image = attachment.base64_content()
+                    url = f"data:{attachment.resolve_type()};base64,{base64_image}"
+                attachment_message.append(
+                    {"type": "image_url", "image_url": {"url": url}}
+                )
+            messages.append({"role": "user", "content": attachment_message})
+
         kwargs = self.build_kwargs(prompt)
         client = self.get_client()
         if stream:
@@ -335,6 +371,7 @@ def execute(self, prompt, stream, response, conversation=None):
             )
             response.response_json = remove_dict_none_values(completion.model_dump())
             yield completion.choices[0].message.content
+        response._prompt_json = redact_data_urls({"messages": messages})
 
     def get_client(self):
         kwargs = {}
@@ -394,7 +431,6 @@ def execute(self, prompt, stream, response, conversation=None):
                 messages.append(prev_response.prompt.prompt)
                 messages.append(prev_response.text())
         messages.append(prompt.prompt)
-        response._prompt_json = {"messages": messages}
         kwargs = self.build_kwargs(prompt)
         client = self.get_client()
         if stream:
@@ -422,6 +458,7 @@ def execute(self, prompt, stream, response, conversation=None):
             )
             response.response_json = remove_dict_none_values(completion.model_dump())
             yield completion.choices[0].text
+        response._prompt_json = redact_data_urls({"messages": messages})
 
 
 def not_nulls(data) -> dict:
@@ -469,3 +506,25 @@ def combine_chunks(chunks: List) -> dict:
             combined[key] = value
 
     return combined
+
+
+def redact_data_urls(input_dict):
+    """
+    Recursively search through the input dictionary for any 'image_url' keys
+    and modify the 'url' value to be just 'data:...'.
+    """
+    if isinstance(input_dict, dict):
+        for key, value in input_dict.items():
+            if (
+                key == "image_url"
+                and isinstance(value, dict)
+                and "url" in value
+                and value["url"].startswith("data:")
+            ):
+                value["url"] = "data:..."
+            else:
+                redact_data_urls(value)
+    elif isinstance(input_dict, list):
+        for item in input_dict:
+            redact_data_urls(item)
+    return input_dict
diff --git a/llm/migrations.py b/llm/migrations.py
index 008ae976..91da6429 100644
--- a/llm/migrations.py
+++ b/llm/migrations.py
@@ -201,3 +201,29 @@ def m010_create_new_log_tables(db):
 @migration
 def m011_fts_for_responses(db):
     db["responses"].enable_fts(["prompt", "response"], create_triggers=True)
+
+
+@migration
+def m012_attachments_tables(db):
+    db["attachments"].create(
+        {
+            "id": str,
+            "type": str,
+            "path": str,
+            "url": str,
+            "content": bytes,
+        },
+        pk="id",
+    )
+    db["prompt_attachments"].create(
+        {
+            "response_id": str,
+            "attachment_id": str,
+            "order": int,
+        },
+        foreign_keys=(
+            ("response_id", "responses", "id"),
+            ("attachment_id", "attachments", "id"),
+        ),
+        pk=("response_id", "attachment_id"),
+    )
diff --git a/llm/models.py b/llm/models.py
index 0e47bb60..6a602f52 100644
--- a/llm/models.py
+++ b/llm/models.py
@@ -1,7 +1,11 @@
+import base64
 from dataclasses import dataclass, field
 import datetime
 from .errors import NeedsKeyException
+import hashlib
+import httpx
 from itertools import islice
+import puremagic
 import re
 import time
 from typing import Any, Dict, Iterable, Iterator, List, Optional, Set, Union
@@ -13,17 +17,88 @@
 CONVERSATION_NAME_LENGTH = 32
 
 
+@dataclass
+class Attachment:
+    type: Optional[str] = None
+    path: Optional[str] = None
+    url: Optional[str] = None
+    content: Optional[bytes] = None
+    _id: Optional[str] = None
+
+    def id(self):
+        # Hash of the binary content, or of '{"url": "https://..."}' for URL attachments
+        if self._id is None:
+            if self.content:
+                self._id = hashlib.sha256(self.content).hexdigest()
+            elif self.path:
+                self._id = hashlib.sha256(open(self.path, "rb").read()).hexdigest()
+            else:
+                self._id = hashlib.sha256(
+                    json.dumps({"url": self.url}).encode("utf-8")
+                ).hexdigest()
+        return self._id
+
+    def resolve_type(self):
+        if self.type:
+            return self.type
+        # Derive it from path or url or content
+        if self.path:
+            return puremagic.from_file(self.path, mime=True)
+        if self.url:
+            response = httpx.head(self.url)
+            response.raise_for_status()
+            return response.headers.get("content-type")
+        if self.content:
+            return puremagic.from_string(self.content, mime=True)
+        raise ValueError("Attachment has no type and no content to derive it from")
+
+    def content_bytes(self):
+        content = self.content
+        if not content:
+            if self.path:
+                content = open(self.path, "rb").read()
+            elif self.url:
+                response = httpx.get(self.url)
+                response.raise_for_status()
+                content = response.content
+        return content
+
+    def base64_content(self):
+        return base64.b64encode(self.content_bytes()).decode("utf-8")
+
+    @classmethod
+    def from_row(cls, row):
+        return cls(
+            _id=row["id"],
+            type=row["type"],
+            path=row["path"],
+            url=row["url"],
+            content=row["content"],
+        )
+
+
 @dataclass
 class Prompt:
     prompt: str
     model: "Model"
+    attachments: Optional[List[Attachment]]
     system: Optional[str]
     prompt_json: Optional[str]
     options: "Options"
 
-    def __init__(self, prompt, model, system=None, prompt_json=None, options=None):
+    def __init__(
+        self,
+        prompt,
+        model,
+        *,
+        attachments=None,
+        system=None,
+        prompt_json=None,
+        options=None
+    ):
         self.prompt = prompt
         self.model = model
+        self.attachments = list(attachments or [])
         self.system = system
         self.prompt_json = prompt_json
         self.options = options or {}
@@ -39,6 +114,8 @@ class Conversation:
     def prompt(
         self,
         prompt: Optional[str],
+        *,
+        attachments: Optional[List[Attachment]] = None,
         system: Optional[str] = None,
         stream: bool = True,
         **options
@@ -46,8 +123,9 @@ def prompt(
         return Response(
             Prompt(
                 prompt,
-                system=system,
                 model=self.model,
+                attachments=attachments,
+                system=system,
                 options=self.model.Options(**options),
             ),
             self.model,
@@ -138,8 +216,9 @@ def log_to_db(self, db):
             },
             ignore=True,
         )
+        response_id = str(ULID()).lower()
         response = {
-            "id": str(ULID()).lower(),
+            "id": response_id,
             "model": self.model.model_id,
             "prompt": self.prompt.prompt,
             "system": self.prompt.system,
@@ -156,16 +235,44 @@ def log_to_db(self, db):
             "datetime_utc": self.datetime_utc(),
         }
         db["responses"].insert(response)
+        # Persist any attachments - loop through with index
+        for index, attachment in enumerate(self.prompt.attachments):
+            attachment_id = attachment.id()
+            db["attachments"].insert(
+                {
+                    "id": attachment_id,
+                    "type": attachment.resolve_type(),
+                    "path": attachment.path,
+                    "url": attachment.url,
+                    "content": attachment.content,
+                },
+                replace=True,
+            )
+            db["prompt_attachments"].insert(
+                {
+                    "response_id": response_id,
+                    "attachment_id": attachment_id,
+                    "order": index,
+                },
+            )
 
     @classmethod
-    def fake(cls, model: "Model", prompt: str, system: str, response: str):
+    def fake(
+        cls,
+        model: "Model",
+        prompt: str,
+        *attachments: List[Attachment],
+        system: str,
+        response: str
+    ):
         "Utility method to help with writing tests"
         response_obj = cls(
             model=model,
             prompt=Prompt(
                 prompt,
-                system=system,
                 model=model,
+                attachments=attachments,
+                system=system,
             ),
             stream=False,
         )
@@ -174,7 +281,7 @@ def fake(cls, model: "Model", prompt: str, system: str, response: str):
         return response_obj
 
     @classmethod
-    def from_row(cls, row):
+    def from_row(cls, db, row):
         from llm import get_model
 
         model = get_model(row["model"])
@@ -183,8 +290,9 @@ def from_row(cls, row):
             model=model,
             prompt=Prompt(
                 prompt=row["prompt"],
-                system=row["system"],
                 model=model,
+                attachments=[],
+                system=row["system"],
                 options=model.Options(**json.loads(row["options_json"])),
             ),
             stream=False,
@@ -194,6 +302,19 @@ def from_row(cls, row):
         response.response_json = json.loads(row["response_json"] or "null")
         response._done = True
         response._chunks = [row["response"]]
+        # Attachments
+        response.attachments = [
+            Attachment.from_row(arow)
+            for arow in db.query(
+                """
+                select attachments.* from attachments
+                join prompt_attachments on attachments.id = prompt_attachments.attachment_id
+                where prompt_attachments.response_id = ?
+                order by prompt_attachments."order"
+            """,
+                [row["id"]],
+            )
+        ]
         return response
 
     def __repr__(self):
@@ -242,10 +363,15 @@ def get_key(self):
 
 class Model(ABC, _get_key_mixin):
     model_id: str
+
+    # API key handling
     key: Optional[str] = None
     needs_key: Optional[str] = None
     key_env_var: Optional[str] = None
+
+    # Model characteristics
     can_stream: bool = False
+    attachment_types: Set = set()
 
     class Options(_Options):
         pass
@@ -269,13 +395,34 @@ def execute(
 
     def prompt(
         self,
-        prompt: Optional[str],
+        prompt: str,
+        *,
+        attachments: Optional[List[Attachment]] = None,
         system: Optional[str] = None,
         stream: bool = True,
         **options
     ):
+        # Validate attachments
+        if attachments and not self.attachment_types:
+            raise ValueError(
+                "This model does not support attachments, but some were provided"
+            )
+        for attachment in attachments or []:
+            attachment_type = attachment.resolve_type()
+            if attachment_type not in self.attachment_types:
+                raise ValueError(
+                    "This model does not support attachments of type '{}', only {}".format(
+                        attachment_type, ", ".join(self.attachment_types)
+                    )
+                )
         return self.response(
-            Prompt(prompt, system=system, model=self, options=self.Options(**options)),
+            Prompt(
+                prompt,
+                attachments=attachments,
+                system=system,
+                model=self,
+                options=self.Options(**options),
+            ),
             stream=stream,
         )
 
diff --git a/setup.py b/setup.py
index bc77c22b..4c77a9e3 100644
--- a/setup.py
+++ b/setup.py
@@ -48,12 +48,13 @@ def get_long_description():
         "setuptools",
         "pip",
         "pyreadline3; sys_platform == 'win32'",
+        "puremagic",
     ],
     extras_require={
         "test": [
             "pytest",
             "numpy",
-            "pytest-httpx",
+            "pytest-httpx>=0.33.0",
             "cogapp",
             "mypy>=1.10.0",
             "black>=24.1.0",


Test patch:
diff --git a/tests/test_chat.py b/tests/test_chat.py
index cf7ddeff..01b2a0c0 100644
--- a/tests/test_chat.py
+++ b/tests/test_chat.py
@@ -23,7 +23,10 @@ def test_chat_basic(mock_model, logs_db):
     mock_model.enqueue(["one world"])
     mock_model.enqueue(["one again"])
     result = runner.invoke(
-        llm.cli.cli, ["chat", "-m", "mock"], input="Hi\nHi two\nquit\n"
+        llm.cli.cli,
+        ["chat", "-m", "mock"],
+        input="Hi\nHi two\nquit\n",
+        catch_exceptions=False,
     )
     assert result.exit_code == 0
     assert result.output == (
diff --git a/tests/test_templates.py b/tests/test_templates.py
index 57c8f836..488a8f55 100644
--- a/tests/test_templates.py
+++ b/tests/test_templates.py
@@ -139,7 +139,7 @@ def test_templates_prompt_save(templates_path, args, expected_prompt, expected_e
             None,
             None,
             "Error: Cannot use -t/--template and --system together",
-            marks=pytest.mark.httpx_mock(assert_all_responses_were_requested=False),
+            marks=pytest.mark.httpx_mock(),
         ),
         pytest.param(
             "prompt: 'Say $hello'",
@@ -147,7 +147,7 @@ def test_templates_prompt_save(templates_path, args, expected_prompt, expected_e
             None,
             None,
             "Error: Missing variables: hello",
-            marks=pytest.mark.httpx_mock(assert_all_responses_were_requested=False),
+            marks=pytest.mark.httpx_mock(),
         ),
         (
             "prompt: 'Say $hello'",