Repository: simonw/llm (language: python)

Issue:
Add ruff for linting
Following:
- #77
Support for `-o key value` options such as `temperature`
It would be nice to add temperature as another command param --temp, just like --system
Plugin hook: `register_models`
Blocks:
- #32 
- #33 
- #20 


Switch plugin tutorial to use pyproject.toml instead of setup.py
Based on what I figured out in this TIL: https://til.simonwillison.net/python/pyproject


Discussion / hints:

Since I'm adding support for different models in:
- #53 

I think for the CLI interface here I'm going to use something more like this:

```bash
llm "Ten names for a pelican" -m gpt-4 -o temperature 0.1
```
So `-o/--option name value` is the way of setting custom options (since `-p/--param` is already taken for template parameters).

That way if there's a plugin that adds a new model with other kinds of options the same `--option name value` syntax still works.

I'll make it so options can validate their parameters though, to ensure people get a useful error message if they try `-o temprature 0.1` instead with a typo.
I'm not crazy about `-o` since it could be confused to mean "output". `-p` is taken. Options:

- `-s/--setting` - no, because I want to use `-s` as a new shortcut for `--system`
- `-o/--option` - best current idea?
- `-c/--config` - I use `-c` for continue already
- `-v/--var` - I don't like this, I feel like variable is a different concept from a key/value configuration setting
- `-k/--key` - I'm already using `--key` for API keys
I think it's `-o/--option` then - where different models accept different options.
> Having key/value options like this should be flexible to handle pretty much anything - especially since option values could be strings, which means that super-complex concepts like `logit_bias` could even be fed a JSON string:
>
> ```bash
> llm -m gpt-4 "Once upon a" -o logit_bias '{"2435":-100, "640":-100}'
> ```
> Example from https://help.openai.com/en/articles/5247780-using-logit-bias-to-define-token-probability

_Originally posted by @simonw in https://github.com/simonw/llm/issues/53#issuecomment-1604968877_
            
I don't think this needs to take any arguments - the signature is:

```python
@hookspec
def register_models():
    """Return a list of Models"""
```
I think the list of models is a list of classes that are subclasses of a `Model` base class - or should it be a list of instances? I think classes.
Or... maybe it takes an `llm` argument which is similar to `datasette` in that it's an object offering a documented API for various useful things, like looking up configuraiton settings and loading templates and suchlike.

```python
@hookspec
def register_models(llm):
    """Return a list of Models"""
```
I think it returns objects, not classes - because then plugins like the OpenAI one can define a single class and return multiple instances of it, each for a different model that uses the same underlying API.
I want to call the common base model `llm.BaseModel` but I'm a little concerned that it clashes with Pydantic's `BaseModel`. I could call it `llm.Model` instead.
I played around with https://github.com/nomic-ai/gpt4all - it runs a `localhost` web server which emulates the OpenAI API, such that all you need to do is set a `OPENAI_API_BASE` environment variable and then use the existing `openai` library - though it has to be used in `Completion` and not `ChatCompletion` mode.

Experimental code for that here: https://github.com/simonw/llm/commit/a196db1ed08b4582ef85fbd3eb6c71759309adf0

Had to turn on this option:

![CleanShot 2023-06-23 at 07 56 27@2x](https://github.com/simonw/llm/assets/9599/15ed39ee-6376-49c3-a384-e98067b62c76)

Then set this environment variable:

```bash
export OPENAI_API_BASE=http://localhost:4891/v1
```
Then I created this template called `mpt`:
```yaml
prompt: "### Human:\n$input\n### Assistant:"
model: mpt-7b-chat
```
And now I can do this:
```bash
llm -t mpt --no-stream 'What is the capital city of Australia?'
```
```
Canberra.
```
In  the logs:
```bash
llm logs -n 1
```
```json
[
  {
    "id": 538,
    "model": "mpt-7b-chat",
    "timestamp": "2023-06-23 14:58:30.208931",
    "prompt": "### Human:\nWhat is the capital city of Australia?\n### Assistant:",
    "system": null,
    "response": "Canberra.",
    "chat_id": null,
    "debug": "{\"model\": \"mpt-7b-chat\", \"usage\": {\"completion_tokens\": 4, \"prompt_tokens\": 26, \"total_tokens\": 30}}",
    "duration_ms": 1756,
    "prompt_json": null
  }
]
```
That `gpt4all` experiment demonstrated that I do need a way to define models that are "classes" of models that the user can then customize.

In this case the `gpt4all` plugin model has a URL which should be configurable, including multiple times - what if you have a `localhost:4891` instance but your team also has a `gpt4all.mycompany.corp:4891` instance that you want to be able to query?

Likewise, I found https://github.com/ortegaalfredo which offers models via Tor where a plugin (encapsulating this library https://github.com/ortegaalfredo/neuroengine/blob/main/neuroengine.py - see https://news.ycombinator.com/item?id=36423590) could be configured to point at different hosted instances.
Design challenge: how to best handle ongoing conversations.

Models based on `openai.ChatCompletion` need a `messages=` list of specifically formatted JSON message objects.

Other models might instead implement chat by building their own strings that look something like this:

```
### Human:
What is the capital city of Australia?
### Assistant:
Canberra
### Human:
What is its population?
### Assistant:
```
Which bit of the code should be responsible for these things? Both retrieving previous conversation messages and figuring out how to format them to be sent to the model.

The conversation prompt there has some overlap with the existing prompt template mechanism, although that doesn't currently support loops.
My intuition is that the Model class should do as much of this as possible, since that way future unanticipated model features can still be built in plugins without needing changes to the design of core.
At which point, what is the responsibility of `LLM` itself? I think it's these things:

- Provide a consistent CLI interface for running prompts, including easily running the same prompt against different models
- Provide a web UI for the same purpose
- Maintain a store of prompts and responses in a SQLite database (plus maybe a plugin hook for sending those store prompts and responses to other systems)
Inspired by:
- #63
Models have options - things like `temperature` and `top_k` and so-on.

Each model instance should be able to validate these, so that when the CLI is called like this:
```bash
llm -m my-model -o temprature 0.3
```
A validation error can be shown if the option names are invalid (or the data types passed are wrong).
Having key/value options like this should be flexible to handle pretty much anything - especially since option values could be strings, which means that super-complex concepts like `logit_bias` could even be fed a JSON string:

```bash
llm -m gpt-4 "Once upon a" -o logit_bias '{2435:-100, 640:-100}'
```
Example from https://help.openai.com/en/articles/5247780-using-logit-bias-to-define-token-probability
Got that prototype working here:
- https://github.com/simonw/llm/issues/36#issuecomment-1604996255

Mainly learned that models are going to need to do type conversions on their options - the OpenAI client library needs floating point for temperature, and `logit_bias` needs to be a `{int: int}` dictionary which is invalid JSON because in JSON keys must be strings.
I should support the same option being passed multiple times, so I can have a `gpt-4` option for suppressing tokens without looking up their IDs that looks like this:
```bash
llm -m gpt-4 "Once upon a" -o suppress "time" -o suppress " time" -o supress "Time"
```
Or maybe:
```bash
llm -m gpt-4 "Once upon a" -o suppress '["time", " time", "Time", "_time"]`
```
If I'm going to use Pydantic for option validation then supporting multiple `-o` with the same name is a bit tricky, so I think I'll go with the second option.
This also means models need a mechanism by which they can return help.

Maybe `llm models show gpt-4` should be able to output help for the various options.
I model instance can execute a prompt, and can validate its options.
```python
model.prompt(prompt, system=system, options=options)
```
```
errors = model.validate_options(options)
```
Where `options` is a list of `str`, `Any` tuples.
First prototype of validating options:
```python
class Model:
    options = {}

    def validate_options(
        self, options: Iterable[Tuple[str, Any]]
    ) -> Iterable[Tuple[str, Any]]:
        invalid_options = {}
        valid_options = []
        for key, value in options:
            if key not in self.options:
                invalid_options[(key, value)] = "Invalid option: {}".format(key)
            else:
                expected_type = self.options[key]
                try:
                    cleaned_value = expected_type(value)
                except ValueError:
                    invalid_options[
                        (key, value)
                    ] = "Option {}: value {} should be {}".format(
                        key, value, expected_type
                    )
                valid_options.append((key, cleaned_value))
        if invalid_options:
            raise OptionsError(invalid_options)
        return valid_options


class Foo(Model):
    options = {
        "id": str,
        "score": float,
    }


class OptionsError(Exception):
    def __init__(self, invalid_options: Dict[Tuple[str, Any], str]):
        super().__init__(f"Invalid options found: {invalid_options}")
        self.invalid_options: Dict[Tuple[str, Any], str] = invalid_options
```
Then I decided to try Pydantic instead, since I already use that for validating YAML for the templates.
This seems to work:
```python
from pydantic import BaseModel, ValidationError, validator

class Model:
    class Options(BaseModel):
        class Config:
            extra = "forbid"

    def validate_options(
        self, options: Iterable[Tuple[str, Any]]
    ) -> Iterable[Tuple[str, Any]]:
        try:
            return self.Options(**options)
        except ValidationError:
            raise


class Foo(Model):
    class Options(Model.Options):
        id: str
        score: float
        keywords: List[str]
        @validator('keywords', pre=True)
        def keywords_must_be_list(cls, v):
            if isinstance(v, str):
                try:
                    return json.loads(v)
                except json.JSONDecodeError:
                    raise ValueError('keywords must be a list or a JSON-encoded list')
            return v
```
Then:
```pycon
>>> from llm import Foo
>>> foo = Foo()
>>> foo.validate_options({'id': 1, 'score': '3.4', 'keywords': '["one", "two"]'})
Options(id='1', score=3.4, keywords=['one', 'two'])
>>> foo.validate_options({'id': 1, 'score': '3.4', 'keywords': ["one", "two"]})
Options(id='1', score=3.4, keywords=['one', 'two'])
>>> foo.validate_options({'id': 1, 'score': '3.4', 'keywords': ["one", "two", 3]})
Options(id='1', score=3.4, keywords=['one', 'two', '3'])
>>> foo.validate_options({'id': 1, 'score': '3.4', 'keywords': ["one", "two", {"one"}]})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/simon/Dropbox/Development/llm/llm/__init__.py", line 18, in validate_options
    return self.Options(**options)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for Options
keywords -> 2
  str type expected (type=type_error.str)
>>> 
```
That `validate_options` method may not be worth it - I can do this instead:
```pycon
>>> Foo.Options(**{'id': 1, 'score': '3.4', 'keywords': ["one", "two", 3]})
Options(id='1', score=3.4, keywords=['one', 'two', '3'])
```
Should models perhaps have a `prompt()` method an a separate `conversation()` method, where the second method handles ongoing conversations that might need to include previous messages? (`-c/--continue` mode)
The model `.prompt()` method should return a `Response` which can be iterated to get a stream of tokens, but also offers a `.text` accessor for the competed string and a `.debug` property with debug information.

Accessing the properties causes the iterator to be exhausted first. 

Maybe also a `.json` property with raw response JSON. And perhaps a `.prompt` that refers back to a representation of the original prompt.

That way the returned object from `.prompt()` has everything needed to both display the result and to store it in the database.

Maybe the `.log()` method deals with instances of `Response`.
Question: if a prompt is part of a conversation is it necessary to serialize the full sent prompt in the DB, even though doing so will duplicate messages in storage many times over?

Reason not to is efficiency of storage, but maybe that's a premature optimization.
New concept for the plugin hook:

```python
@hookspec
def register_models(register):
    """Register additional models by returning one or more Model subclasses"""
```
The hook implementations then call that `register(...)` function for each model they want to register.

I can then add further arguments to the hook in the future, while shipping a minimally viable version quickly. I don't need to design the full `llm` interface.
... except, maybe the version that returns a list of models is better? I'm going to try that one first - so `register_models()` taking no arguments at all.
This seems a little confusing though:
```python
@hookspec
def register_commands(cli):
    """Register additional CLI commands, e.g. 'llm mycommand ...'"""


@hookspec
def register_models():
    "Return a list of model instances representing LLM models that can be called"
```
`register_commands(cli)` doesn't return anything, it requires adding things to `cli`.

`register_models()` returns a list.

So perhaps `register_models()` should have a different name, if it works differently. Perhaps `load_models()`.
Aha! The reason I wanted to do `register_models(register)` was to handle aliases:

```python
@hookimpl
def register_models(register):
    register(Chat("gpt-3.5-turbo"), aliases=("3.5", "chatgpt"))
    register(Chat("gpt-3.5-turbo-16k"), aliases=("chatgpt-16k", "3.5-16k"))
    register(Chat("gpt-4"), aliases=("4", "gpt4"))
    register(Chat("gpt-4-32k"), aliases=("4-32k",))
```


Fix patch:
diff --git a/Justfile b/Justfile
index b961b74c..e6b3f8ad 100644
--- a/Justfile
+++ b/Justfile
@@ -1,6 +1,10 @@
 # Run tests and linters
 @default: test lint
 
+# Install dependencies and test dependencies
+@init:
+  pipenv run pip install -e '.[test]'
+
 # Run pytest with supplied options
 @test *options:
   pipenv run pytest {{options}}
@@ -9,6 +13,8 @@
 @lint:
   pipenv run black . --check
   pipenv run cog --check README.md docs/*.md
+  pipenv run mypy llm
+  pipenv run ruff .
 
 # Rebuild docs with cog
 @cog:
@@ -21,3 +27,8 @@
 # Apply Black
 @black:
   pipenv run black .
+
+# Run automatic fixes
+@fix: cog
+  pipenv run ruff . --fix
+  pipenv run black .
diff --git a/MANIFEST.in b/MANIFEST.in
new file mode 100644
index 00000000..f4089709
--- /dev/null
+++ b/MANIFEST.in
@@ -0,0 +1,1 @@
+global-exclude tests/*
diff --git a/docs/help.md b/docs/help.md
index feacf7f2..9528b3f3 100644
--- a/docs/help.md
+++ b/docs/help.md
@@ -60,6 +60,8 @@ Commands:
   install    Install packages from PyPI into the same environment as LLM
   keys       Manage stored API keys for different models
   logs       Tools for exploring logged prompts and responses
+  models     Manage available models
+  openai     Commands for working directly with the OpenAI API
   plugins    List installed plugins
   templates  Manage stored prompt templates
   uninstall  Uninstall Python packages from the LLM environment
@@ -73,17 +75,18 @@ Usage: llm prompt [OPTIONS] [PROMPT]
   Documentation: https://llm.datasette.io/en/stable/usage.html
 
 Options:
-  --system TEXT               System prompt to use
-  -m, --model TEXT            Model to use
-  -t, --template TEXT         Template to use
-  -p, --param <TEXT TEXT>...  Parameters for template
-  --no-stream                 Do not stream output
-  -n, --no-log                Don't log to database
-  -c, --continue              Continue the most recent conversation.
-  --chat INTEGER              Continue the conversation with the given chat ID.
-  --key TEXT                  API key to use
-  --save TEXT                 Save prompt with this template name
-  --help                      Show this message and exit.
+  -s, --system TEXT            System prompt to use
+  -m, --model TEXT             Model to use
+  -o, --option <TEXT TEXT>...  key/value options for the model
+  -t, --template TEXT          Template to use
+  -p, --param <TEXT TEXT>...   Parameters for template
+  --no-stream                  Do not stream output
+  -n, --no-log                 Don't log to database
+  -c, --continue               Continue the most recent conversation.
+  --chat INTEGER               Continue the conversation with the given chat ID.
+  --key TEXT                   API key to use
+  --save TEXT                  Save prompt with this template name
+  --help                       Show this message and exit.
 ```
 ### llm init-db --help
 ```
@@ -167,6 +170,37 @@ Options:
   -t, --truncate       Truncate long strings in output
   --help               Show this message and exit.
 ```
+### llm models --help
+```
+Usage: llm models [OPTIONS] COMMAND [ARGS]...
+
+  Manage available models
+
+Options:
+  --help  Show this message and exit.
+
+Commands:
+  default  Show or set the default model
+  list     List available models
+```
+#### llm models list --help
+```
+Usage: llm models list [OPTIONS]
+
+  List available models
+
+Options:
+  --help  Show this message and exit.
+```
+#### llm models default --help
+```
+Usage: llm models default [OPTIONS] [MODEL]
+
+  Show or set the default model
+
+Options:
+  --help  Show this message and exit.
+```
 ### llm templates --help
 ```
 Usage: llm templates [OPTIONS] COMMAND [ARGS]...
@@ -229,13 +263,14 @@ Options:
 ```
 ### llm install --help
 ```
-Usage: llm install [OPTIONS] PACKAGES...
+Usage: llm install [OPTIONS] [PACKAGES]...
 
   Install packages from PyPI into the same environment as LLM
 
 Options:
-  -U, --upgrade  Upgrade packages to latest version
-  --help         Show this message and exit.
+  -U, --upgrade             Upgrade packages to latest version
+  -e, --editable DIRECTORY  Install a project in editable mode from this path
+  --help                    Show this message and exit.
 ```
 ### llm uninstall --help
 ```
@@ -247,4 +282,27 @@ Options:
   -y, --yes  Don't ask for confirmation
   --help     Show this message and exit.
 ```
+### llm openai --help
+```
+Usage: llm openai [OPTIONS] COMMAND [ARGS]...
+
+  Commands for working directly with the OpenAI API
+
+Options:
+  --help  Show this message and exit.
+
+Commands:
+  models  List models available to you from the OpenAI API
+```
+#### llm openai models --help
+```
+Usage: llm openai models [OPTIONS]
+
+  List models available to you from the OpenAI API
+
+Options:
+  --json      Output as JSON
+  --key TEXT  OpenAI API key
+  --help      Show this message and exit.
+```
 <!-- [[[end]]] -->
\ No newline at end of file
diff --git a/docs/index.md b/docs/index.md
index b3a285fc..0a796e66 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -29,9 +29,11 @@ maxdepth: 3
 ---
 setup
 usage
+python-api
 templates
 logging
 plugins
+tutorial-model-plugin
 help
 contributing
 changelog
diff --git a/docs/logging.md b/docs/logging.md
index 45bddf31..b6382e1f 100644
--- a/docs/logging.md
+++ b/docs/logging.md
@@ -18,8 +18,6 @@ On my Mac that outputs:
 ```
 This will differ for other operating systems.
 
-(You can customize the location of this file by setting a path in the `LLM_LOG_PATH` environment variable.)
-
 Once that SQLite database has been created any prompts you run will be logged to that database.
 
 To avoid logging a prompt, pass `--no-log` or `-n` to the command:
@@ -64,7 +62,7 @@ import sqlite_utils
 import re
 db = sqlite_utils.Database(memory=True)
 migrate(db)
-schema = db["log"].schema
+schema = db["logs"].schema
 
 def cleanup_sql(sql):
     first_line = sql.split('(')[0]
@@ -77,16 +75,19 @@ cog.out(
 )
 ]]] -->
 ```sql
-CREATE TABLE "log" (
+CREATE TABLE "logs" (
   [id] INTEGER PRIMARY KEY,
   [model] TEXT,
-  [timestamp] TEXT,
   [prompt] TEXT,
   [system] TEXT,
+  [prompt_json] TEXT,
+  [options_json] TEXT,
   [response] TEXT,
-  [chat_id] INTEGER REFERENCES [log]([id]),
-  [debug] TEXT,
-  [duration_ms] INTEGER
+  [response_json] TEXT,
+  [reply_to_id] INTEGER REFERENCES [logs]([id]),
+  [chat_id] INTEGER REFERENCES [logs]([id]),
+  [duration_ms] INTEGER,
+  [datetime_utc] TEXT
 );
 ```
 <!-- [[[end]]] -->
diff --git a/docs/python-api.md b/docs/python-api.md
new file mode 100644
index 00000000..d0386ee0
--- /dev/null
+++ b/docs/python-api.md
@@ -0,0 +1,37 @@
+# Python API
+
+LLM provides a Python API for executing prompts, in addition to the command-line interface.
+
+Understanding this API is also important for writing plugins.
+
+The API consists of the following key classes:
+
+- `Model` - represents a language model against which prompts can be executed
+- `Prompt` - a prompt that can be prepared and then executed against a model
+- `Response` - the response executing a prompt against a model
+- `Template` - a reusable template for generating prompts
+
+## Prompt
+
+A prompt object represents all of the information needed to be passed to the LLM. This could be a single prompt string, but it might also include a separate system prompt, various settings (for temperature etc) or even a JSON array of previous messages.
+
+## Model
+
+The `Model` class is an abstract base class that needs to be subclassed to provide a concrete implementation. Different LLMs will use different implementations of this class.
+
+Model instances provide the following methods:
+
+- `prompt(prompt: str, stream: bool, ...options) -> Response` - a convenience wrapper which creates a `Prompt` instance and then executes it. This is the most common way to use LLM models.
+- `response(prompt: Prompt, stream: bool) -> Response` - execute a prepared Prompt instance against the model and return a `Response`.
+
+Models usually return subclasses of `Response` that are specific to that model.
+
+## Response
+
+The response from an LLM. This could encapusulate a string of text, but for streaming APIs this class will be iterable, with each iteration yielding a short string of text as it is generated.
+
+Calling `.text()` will return the full text of the response, waiting for the stream to stop executing if necessary.
+
+## Template
+
+Templates are reusable objects that can be used to generate prompts. They are used  by the {ref}`prompt-templates` feature.
diff --git a/docs/setup.md b/docs/setup.md
index 62dd1c2c..f5812f1d 100644
--- a/docs/setup.md
+++ b/docs/setup.md
@@ -89,3 +89,17 @@ The environment variable will be used only if no `--key` option is passed to the
 If no environment variable is found, the tool will fall back to checking `keys.json`.
 
 You can force the tool to use the key from `keys.json` even if an environment variable has also been set using `llm "prompt" --key openai`.
+
+## Custom directory location
+
+This tool stores various files - prompt templates, stored keys, preferences, a database of logs - in a directory on your computer.
+
+On macOS this is `~/Library/Application Support/io.datasette.llm/`.
+
+On Linux it may be something like `~/.config/io.datasette.llm/`.
+
+You can set a custom location for this directory by setting the `LLM_USER_PATH` environment variable:
+
+```bash
+export LLM_USER_PATH=/path/to/my/custom/directory
+```
diff --git a/docs/tutorial-model-plugin.md b/docs/tutorial-model-plugin.md
new file mode 100644
index 00000000..e43a4567
--- /dev/null
+++ b/docs/tutorial-model-plugin.md
@@ -0,0 +1,601 @@
+# Writing a plugin to support a new model
+
+This tutorial will walk you through developing a new plugin for LLM that adds support for a new Large Language Model.
+
+We will be developing a plugin that implements a simple [Markov chain](https://en.wikipedia.org/wiki/Markov_chain) to generate words based on an input string. Markov chains are not technically large language models, but they provide a useful exercise for demonstrating how the LLM tool can be extended through plugins.
+
+## The initial structure of the plugin
+
+<!-- [ You can use the llm-simple-plugin cookiecutter template to skip this step - follow instructions in the README.
+
+If you're not using cookiecutter, ]  -->
+
+First create a new directory with the name of your plugin - it should be called something like `llm-markov`.
+```bash
+mkdir llm-markov
+cd llm-markov
+```
+In that directory create a file called `llm_markov.py` containing this:
+
+```python
+import llm
+
+@llm.hookimpl
+def register_models(register):
+    register(Markov())
+
+class Markov(llm.Model):
+    model_id = "markov"
+
+    def execute(self, prompt, stream, response):
+        return ["hello world"]
+```
+
+The `def register_models()` function here is called by the plugin system (thanks to the `@hookimpl` decorator). It uses the `register()` function passed to it to register an instance of the new model.
+
+The `Markov` class implements the model. It sets a `model_id` - an identifier that can be passed to `ll -m` in order to identify the model to be executed.
+
+The logic for executing the model goes in the `execute()` method. We'll extend this to do something more useful in a later step.
+
+Next, create a `pyproject.toml` file. This is necessary to tell LLM how to load your plugin:
+
+```toml
+[project]
+name = "llm-markov"
+version = "0.1"
+
+[project.entry-points.llm]
+markov = "llm_markov"
+```
+
+This is the simplest possible configuration. It defines a plugin name and provides an [entry point](https://setuptools.pypa.io/en/latest/userguide/entry_point.html) for `llm` telling it how to load the plugin.
+
+If you are comfortable with Python virtual environments you can create one now for your project, activate it and run `pip install llm` before the next step.
+
+If you aren't familiar with virtual environments, don't worry: you can develop plugins without them. You'll need to have LLM installed using Homebrew or `pipx` or one of the [other installation options](https://llm.datasette.io/en/latest/setup.html#installation).
+
+## Installing your plugin to try it out
+
+Having created a directory with a `pyproject.toml` file and an `llm_markov.py` file, you can install your plugin into LLM by running this from inside your `llm-markov` directory:
+
+```bash
+llm install -e .
+```
+
+The `-e` stands for "editable" - it means you'll be able to make further changes to the `llm_markov.py` file that will be reflected without you having to reinstall the plugin.
+
+The `.` means the current directory. You can also install editable plugins by passing a path to their directory this:
+```bash
+llm install -e path/to/llm-markov
+```
+To confirm that your plugin has installed correctly, run this command:
+```bash
+llm plugins
+```
+The output should look like this:
+```json
+[
+  {
+    "name": "llm-markov",
+    "hooks": [
+      "register_models"
+    ],
+    "version": "0.1"
+  },
+  {
+    "name": "llm.default_plugins.openai_models",
+    "hooks": [
+      "register_commands",
+      "register_models"
+    ]
+  }
+]
+```
+This command lists default plugins that are included with LLM as well as new plugins that have been installed.
+
+Now let's try the plugin by running a prompt through it:
+```bash
+llm -m markov "the cat sat on the mat"
+```
+It outputs:
+```
+hello world
+```
+Next, we'll make it execute and return the results of a Markov chain.
+
+## Building the Markov chain
+
+Markov chains can be thought of as the simplest possible example of a generative language model. They work by building an index of words that have been seen following other words.
+
+Here's what that index looks like for the phrase "the cat sat on the mat"
+```json
+{
+  "the": ["cat", "mat"],
+  "cat": ["sat"],
+  "sat": ["on"],
+  "on": ["the"]
+}
+```
+Here's a Python function that builds that data structure from a text input:
+```python
+def build_markov_table(text):
+    words = text.split()
+    transitions = {}
+    # Loop through all but the last word
+    for i in range(len(words) - 1):
+        word = words[i]
+        next_word = words[i + 1]
+        transitions.setdefault(word, []).append(next_word)
+    return transitions
+```
+We can try that out by pasting it into the interactive Python interpreter and running this:
+```pycon
+>>> transitions = build_markov_table("the cat sat on the mat")
+>>> transitions
+{'the': ['cat', 'mat'], 'cat': ['sat'], 'sat': ['on'], 'on': ['the']}
+```
+## Executing the Markov chain
+
+To execute the model, we start with a word. We look at the options for words that might come next and pick one of those at random. Then we repeat that process until we have produced the desired number of output words.
+
+Some words might not have any following words from our training sentence. For our implementation we wil fall back on picking a random word from our collection.
+
+We will implement this as a [Python generator], using the yield keyword to produce each token:
+```python
+def generate(transitions, length, start_word=None):
+    all_words = list(transitions.keys())
+    next_word = start_word or random.choice(all_words)
+    for i in range(length):
+        yield next_word
+        options = transitions.get(next_word) or all_words
+        next_word = random.choice(options)
+```
+If you aren't familiar with generators, the above code could also be implemented like this - creating a Python list and returning it at the end of the function:
+```python
+def generate_list(transitions, length, start_word=None):
+    all_words = list(transitions.keys())
+    next_word = start_word or random.choice(all_words)
+    output = []
+    for i in range(length):
+        output.append(next_word)
+        options = transitions.get(next_word) or all_words
+        next_word = random.choice(options)
+    return output
+```
+You can try out the `generate()` function like this:
+```python
+lookup = build_markov_table("the cat sat on the mat")
+for word in generate(transitions, 20):
+    print(word)
+```
+Or you can generate a full string sentence with it like this:
+```python
+sentence = " ".join(generate(transitions, 20))
+```
+## Adding that to the plugin
+
+Our `execute()` method from earlier currently returns the list `["hello world"]`.
+
+Update that to use our new Markov chain generator instead. Here's the full text of the new `llm_markov.py` file:
+
+```python
+import llm
+import random
+
+@llm.hookimpl
+def register_models(register):
+    register(Markov())
+
+def build_markov_table(text):
+    words = text.split()
+    transitions = {}
+    # Loop through all but the last word
+    for i in range(len(words) - 1):
+        word = words[i]
+        next_word = words[i + 1]
+        transitions.setdefault(word, []).append(next_word)
+    return transitions
+
+def generate(transitions, length, start_word=None):
+    all_words = list(transitions.keys())
+    next_word = start_word or random.choice(all_words)
+    for i in range(length):
+        yield next_word
+        options = transitions.get(next_word) or all_words
+        next_word = random.choice(options)
+
+class Markov(llm.Model):
+    model_id = "markov"
+
+    def execute(self, prompt, stream, response):
+        text = prompt.prompt
+        transitions = build_markov_table(text)
+        for word in generate(transitions, 20):
+            yield word + ' '
+```
+The `execute()` method can access the text prompt that the user provided using` prompt.prompt` - `prompt` is a `Prompt` object that might include other more advanced input details as well.
+
+Now when you run this you should see the output of the Markov chain!
+```bash
+llm -m markov "the cat sat on the mat"
+```
+```
+the mat the cat sat on the cat sat on the mat cat sat on the mat cat sat on
+```
+
+## Prompts and responses are logged to the database
+
+The prompt and the response will be logged to a SQLite database automatically by LLM. You can see the single most recent addition to the logs using:
+```
+llm logs -n 1
+```
+The output should look something like this:
+```json
+[
+  {
+    "id": 621,
+    "model": "markov",
+    "prompt": "the cat sat on the mat",
+    "system": null,
+    "prompt_json": null,
+    "options_json": {},
+    "response": "on the cat sat on the cat sat on the mat on the mat sat on the mat on the ",
+    "response_json": null,
+    "reply_to_id": null,
+    "chat_id": null,
+    "duration_ms": 0,
+    "datetime_utc": "2023-07-06T01:31:48.074373"
+  }
+]
+```
+
+Plugins can log additional information to the database by assigning a dictionary to the `response._response_json` property during the `execute()` method.
+
+Here's how to include that full `transitions` table in the `response_json` in the log:
+```python
+    def execute(self, prompt, stream, response):
+        text = self.prompt.prompt
+        transitions = build_markov_table(text)
+        for word in generate(transitions, 20):
+            yield word + ' '
+        response._response_json = {"transitions": transitions}
+```
+
+Now when you run the logs command you'll see that too:
+```bash
+llm logs -n 1
+```
+```json
+[
+  {
+    "id": 623,
+    "model": "markov",
+    "prompt": "the cat sat on the mat",
+    "system": null,
+    "prompt_json": null,
+    "options_json": {},
+    "response": "on the mat the cat sat on the cat sat on the mat sat on the cat sat on the ",
+    "response_json": {
+      "transitions": {
+        "the": [
+          "cat",
+          "mat"
+        ],
+        "cat": [
+          "sat"
+        ],
+        "sat": [
+          "on"
+        ],
+        "on": [
+          "the"
+        ]
+      }
+    },
+    "reply_to_id": null,
+    "chat_id": null,
+    "duration_ms": 0,
+    "datetime_utc": "2023-07-06T01:34:45.376637"
+  }
+]
+```
+In this particular case this isn't a great idea here though: the `transitions` table is duplicate information, since it can be reproduced from the input data - and it can get really large for longer prompts.
+
+## Adding options
+
+LLM models can take options. For large language models these can be things like `temperature` or `top_k`.
+
+Options are passed using the `-o/--option` command line parameters, for example:
+```bash
+llm -m gpt4 "ten pet pelican names" -o temperature 1.5
+```
+We're going to add two options to our Markov chain model:
+
+- `length`: the number of words to generate
+- `delay`: a floating point number of seconds to delay in between each output token
+
+The `delay` token will let us simulate a streaming language model, where tokens take time to generate and are returned by the `execute()` function as they become ready.
+
+Options are defined using an inner class on the model, called `Options`. It should extend the `llm.Options` class.
+
+First, add this import to the top of your 'llm_markov.py' file:
+```python
+from typing import Optional
+```
+Then add this `Options` class to your model:
+```python
+class Markov(Model):
+    model_id = "markov"
+
+    class Options(llm.Options):
+        length: Optional[int] = None
+        delay: Optional[float] = None
+
+```
+
+Let's add extra validation rules to our options. Length must be at least 2. Duration must be between 0 and 10.
+
+The `Options` class uses [Pydantic 2](https://pydantic.org/), which can support all sorts of advanced validation rules.
+
+Add this import to the top of your 'llm_markov.py' file:
+```python
+from pydantic import field_validator
+```
+
+We can add a Pydantic field validator for our two new rules:
+
+```python
+    class Options(llm.Options):
+        length: Optional[int] = None
+        delay: Optional[float] = None
+
+        @field_validator("length")
+        def validate_length(cls, length):
+            if length is None:
+                return None
+            if length < 2:
+                raise ValueError("length must be >= 2")
+            return length
+
+        @field_validator("delay")
+        def validate_delay(cls, delay):
+            if delay is None:
+                return None
+            if not 0 <= delay <= 10:
+                raise ValueError("delay must be between 0 and 10")
+            return delay
+```
+Lets test our options validation:
+```bash
+llm -m markov "the cat sat on the mat" -o length -1
+```
+```
+Error: length
+  Value error, length must be >= 2
+```
+
+Next, we will modify our `execute()` method to handle those options. Add this to the beginning of `llm_markov.py`:
+```python
+import time
+```
+Then replace the `execute()` method with this one:
+```python
+    def execute(self, prompt, stream, response):
+        text = prompt.prompt
+        transitions = build_markov_table(text)
+        length = prompt.options.length or 10
+        for word in generate(transitions, length):
+            yield word + ' '
+            if prompt.options.delay:
+                time.sleep(prompt.options.delay)
+```
+Add `can_stream = True` to the top of the `Markov` model class, on the line below `model_id = "markov". This tells LLM that the model is able to stream content to the console.
+
+The full `llm_markov.py` file should now look like this:
+```python
+import llm
+import random
+import time
+from typing import Optional
+from pydantic import field_validator
+
+@llm.hookimpl
+def register_models(register):
+    register(Markov())
+
+def build_markov_table(text):
+    words = text.split()
+    transitions = {}
+    # Loop through all but the last word
+    for i in range(len(words) - 1):
+        word = words[i]
+        next_word = words[i + 1]
+        transitions.setdefault(word, []).append(next_word)
+    return transitions
+
+def generate(transitions, length, start_word=None):
+    all_words = list(transitions.keys())
+    next_word = start_word or random.choice(all_words)
+    for i in range(length):
+        yield next_word
+        options = transitions.get(next_word) or all_words
+        next_word = random.choice(options)
+
+class Markov(llm.Model):
+    model_id = "markov"
+    can_stream = True
+
+    class Options(llm.Options):
+        length: Optional[int] = None
+        delay: Optional[float] = None
+
+        @field_validator("length")
+        def validate_length(cls, length):
+            if length is None:
+                return None
+            if length < 2:
+                raise ValueError("length must be >= 2")
+            return length
+
+        @field_validator("delay")
+        def validate_delay(cls, delay):
+            if delay is None:
+                return None
+            if not 0 <= delay <= 10:
+                raise ValueError("delay must be between 0 and 10")
+            return delay
+
+    def execute(self, prompt, stream, response):
+        text = prompt.prompt
+        transitions = build_markov_table(text)
+        length = prompt.options.length or 10
+        for word in generate(transitions, length):
+            yield word + ' '
+            if prompt.options.delay:
+                time.sleep(prompt.options.delay)
+```
+Now we can request a 20 word completion with a 0.1s delay between tokens like this:
+```bash
+llm -m markov "the cat sat on the mat" \
+  -o length 20 -o delay 0.1
+```
+LLM provides a `--no-stream` option users can use to turn off streaming. Using that option causes LLM to gather the response from the stream and then return it to the console in one block. You can try that like this:
+```bash
+llm -m markov "the cat sat on the mat" \
+  -o length 20 -o delay 0.1 --no-stream
+```
+In this case it will still delay for 2s total while it gathers the tokens, then output them all at once.
+
+That `--no-stream` option causes the `stream` argument passed to `execute()` to be false. Your `execute()` method can then behave differently depending on whether it is streaming or not.
+
+Options are also logged to the database. You can see those here:
+```bash
+llm logs -n 1
+```
+```json
+[
+  {
+    "id": 636,
+    "model": "markov",
+    "prompt": "the cat sat on the mat",
+    "system": null,
+    "prompt_json": null,
+    "options_json": {
+      "length": 20,
+      "delay": 0.1
+    },
+    "response": "the mat on the mat on the cat sat on the mat sat on the mat cat sat on the ",
+    "response_json": null,
+    "reply_to_id": null,
+    "chat_id": null,
+    "duration_ms": 2063,
+    "datetime_utc": "2023-07-07T03:02:28.232970"
+  }
+]
+```
+
+## Distributing your plugin
+
+There are many different options for distributing your new plugin so other people can try it out.
+
+You can create a downloadable wheel or `.zip` or `.tar.gz` files, or share the plugin through GitHub Gists or repositories.
+
+You can also publish your plugin to PyPI, the Python Package Index.
+
+### Wheels and sdist packages
+
+The easiest option is to produce a distributable package is to use the `build` command. First, install the `build` package by running this:
+```bash
+python -m pip install build
+```
+Then run `build` in your plugin directory to create the packages:
+```bash
+python -m build
+```
+This will create two files: `dist/llm-markov-0.1.tar.gz` and `dist/llm-markov-0.1-py3-none-any.whl`.
+
+Either of these files can be used to install the plugin:
+
+```bash
+llm install dist/llm_markov-0.1-py3-none-any.whl
+```
+If you host this file somewhere online other people will be able to install it using `pip install` against the URL to your package:
+```bash
+llm install 'https://.../llm_markov-0.1-py3-none-any.whl'
+```
+You can run the following command at any time to uninstall your plugin, which is useful for testing out different installation methods:
+```bash
+llm uninstall llm-markov -y
+```
+
+### GitHub Gists
+
+A neat quick option for distributing a simple plugin is to host it in a GitHub Gist. These are available for free with a GitHub account, and can be public or private. Gists can contain multiple files but don't support directory structures - which is OK, because our plugin is just two files, `pyproject.toml` and `llm_markov.py`.
+
+Here's an example Gist I created for this tutorial:
+
+[https://gist.github.com/simonw/6e56d48dc2599bffba963cef0db27b6d](https://gist.github.com/simonw/6e56d48dc2599bffba963cef0db27b6d)
+
+You can turn a Gist into an installable `.zip` URL by right-clicking on the "Download ZIP" button and selecting "Copy Link". Here's that link for my example Gist:
+
+`https://gist.github.com/simonw/6e56d48dc2599bffba963cef0db27b6d/archive/945a46fd05522a718eab79fa736a9105cb4c27c7.zip`
+
+The plugin can be installed using the `llm install` command like this:
+```bash
+llm install 'https://gist.github.com/simonw/6e56d48dc2599bffba963cef0db27b6d/archive/945a46fd05522a718eab79fa736a9105cb4c27c7.zip'
+```
+
+## GitHub repositories
+
+The same trick works for regular GitHub repositories as well: the "Download ZIP" button can be found by clicking the green "Code" button at the top of the repository. The URL which that provide scan then be used to install the plugin that lives in that repository.
+
+## Publishing plugins to PyPI
+
+The [Python Package Index (PyPI)](https://pypi.org/) is the official repository for Python packages. You can upload your plugin to PyPI and reserve a name for it - once you have done that, anyone will be able to install your plugin using `llm install <name>`.
+
+Follow [these instructions](https://packaging.python.org/en/latest/tutorials/packaging-projects/#uploading-the-distribution-archives) to publish a package to PyPI. The short version:
+```bash
+python -m pip install twine
+python -m twine upload dist/*
+```
+You will need an account on PyPI, then you can enter your username and password - or create a token in the PyPI settings and use `__token__` as the username and the token as the password.
+
+## Adding metadata
+
+Before uploading a package to PyPI it's a good idea to add documentation and expand `pyproject.toml` with additional metadata.
+
+Create a `README.md` file in the root of your plugin directory with instructions about how to install, configure and use your plugin.
+
+You can then replace `pyproject.toml` with something like this:
+
+```toml
+[project]
+name = "llm-markov"
+version = "0.1"
+description = "Plugin for LLM adding a Markov chain generating model"
+readme = "README.md"
+authors = [{name = "Simon Willison"}]
+license = {text = "Apache-2.0"}
+classifiers = [
+    "License :: OSI Approved :: Apache Software License"
+]
+dependencies = [
+    "llm"
+]
+requires-python = ">3.7"
+
+[project.urls]
+homepage = "https://github.com/simonw/llm-markov"
+Changelog = "https://github.com/simonw/llm-markov/releases"
+Issues = "https://github.com/simonw/llm-markov/issues"
+
+[project.entry-points.llm]
+markov = "llm_markov"
+```
+This will pull in your README to be displayed as part of your project's listing page on PyPI.
+
+It adds `llm` as a dependency, ensuring it will be installed if someone tries to install your plugin package without it.
+
+It adds some links to useful pages (you can drop the `project.urls` section if those links are not useful for your project).
+
+You should drop a `LICENSE` file into the GitHub repository for your package as well. I like to use the Apache 2 license [like this](https://github.com/simonw/llm/blob/main/LICENSE).
diff --git a/docs/usage.md b/docs/usage.md
index e4f99b82..54a4d7a5 100644
--- a/docs/usage.md
+++ b/docs/usage.md
@@ -52,7 +52,7 @@ This pattern of using `$(command)` inside a double quoted string is a useful way
 
 ## System prompts
 
-You can use `--system '...'` to set a system prompt.
+You can use `-s/--system '...'` to set a system prompt.
 
     llm 'SQL to calculate total sales by month' \
       --system 'You are an exaggerated sentient cheesecake that knows SQL and talks about cheesecake a lot'
@@ -60,4 +60,41 @@ You can use `--system '...'` to set a system prompt.
 This is useful for piping content to standard input, for example:
 
     curl -s 'https://simonwillison.net/2023/May/15/per-interpreter-gils/' | \
-      llm --system 'Suggest topics for this post as a JSON array'
+      llm -s 'Suggest topics for this post as a JSON array'
+
+## Listing available models
+
+The `llm models list` command lists every model that can be used with LLM, along with any aliases:
+
+```
+llm models list
+```
+Example output:
+```
+OpenAI Chat: gpt-3.5-turbo (aliases: 3.5, chatgpt)
+OpenAI Chat: gpt-3.5-turbo-16k (aliases: chatgpt-16k, 3.5-16k)
+OpenAI Chat: gpt-4 (aliases: 4, gpt4)
+OpenAI Chat: gpt-4-32k (aliases: 4-32k)
+PaLM 2: chat-bison-001 (aliases: palm, palm2)
+```
+You can use pass the full model name or any of the aliases to the `-m/--model` option:
+
+```
+llm -m chatgpt-16k 'As many names for cheesecakes as you can think of, with detailed descriptions'
+```
+Models that have been installed using plugins will be shown here as well.
+
+## Setting a custom model
+
+The model used when calling `llm` without the `-m/--model` option defaults to `gpt-3.5-turbo` - the fastest and least expensive OpenAI model, and the same model family that powers ChatGPT.
+
+You can use the `llm models default` command to set a different default model. For GPT-4 (slower and more expensive, but more capable) run this:
+
+```bash
+llm models default gpt-4
+```
+You can view the current model by running this:
+```
+llm models default
+```
+Any of the supported aliases for a model can be passed to this command.
\ No newline at end of file
diff --git a/llm/__init__.py b/llm/__init__.py
index 1d8bd894..db5a7adb 100644
--- a/llm/__init__.py
+++ b/llm/__init__.py
@@ -1,55 +1,103 @@
-from pydantic import BaseModel
-import string
-from typing import Optional
-from .hookspecs import hookimpl  # noqa
-from .hookspecs import hookspec  # noqa
-
-
-class Template(BaseModel):
-    name: str
-    prompt: Optional[str]
-    system: Optional[str]
-    model: Optional[str]
-    defaults: Optional[dict]
-
-    class Config:
-        extra = "forbid"
-
-    class MissingVariables(Exception):
-        pass
-
-    def execute(self, input, params=None):
-        params = params or {}
-        params["input"] = input
-        if self.defaults:
-            for k, v in self.defaults.items():
-                if k not in params:
-                    params[k] = v
-        if not self.prompt:
-            system = self.interpolate(self.system, params)
-            prompt = input
-        else:
-            prompt = self.interpolate(self.prompt, params)
-            system = self.interpolate(self.system, params)
-        return prompt, system
-
-    @classmethod
-    def interpolate(cls, text, params):
-        if not text:
-            return text
-        # Confirm all variables in text are provided
-        string_template = string.Template(text)
-        vars = cls.extract_vars(string_template)
-        missing = [p for p in vars if p not in params]
-        if missing:
-            raise cls.MissingVariables(
-                "Missing variables: {}".format(", ".join(missing))
-            )
-        return string_template.substitute(**params)
-
-    @staticmethod
-    def extract_vars(string_template):
-        return [
-            match.group("named")
-            for match in string_template.pattern.finditer(string_template.template)
-        ]
+from .hookspecs import hookimpl
+from .models import (
+    LogMessage,
+    Model,
+    ModelWithAliases,
+    Options,
+    Prompt,
+    Response,
+    OptionsError,
+)
+from .templates import Template
+from .plugins import pm
+import click
+from typing import Dict, List
+import json
+import os
+import pathlib
+
+__all__ = [
+    "hookimpl",
+    "get_model",
+    "get_key",
+    "LogMessage",
+    "Model",
+    "Options",
+    "OptionsError",
+    "Prompt",
+    "Response",
+    "Template",
+]
+
+
+def get_plugins():
+    plugins = []
+    plugin_to_distinfo = dict(pm.list_plugin_distinfo())
+    for plugin in pm.get_plugins():
+        plugin_info = {
+            "name": plugin.__name__,
+            "hooks": [h.name for h in pm.get_hookcallers(plugin)],
+        }
+        distinfo = plugin_to_distinfo.get(plugin)
+        if distinfo:
+            plugin_info["version"] = distinfo.version
+            plugin_info["name"] = distinfo.project_name
+        plugins.append(plugin_info)
+    return plugins
+
+
+def get_models_with_aliases() -> List["ModelWithAliases"]:
+    model_aliases = []
+
+    def register(model, aliases=None):
+        model_aliases.append(ModelWithAliases(model, aliases or set()))
+
+    pm.hook.register_models(register=register)
+    return model_aliases
+
+
+def get_model_aliases() -> Dict[str, Model]:
+    model_aliases = {}
+    for model_with_aliases in get_models_with_aliases():
+        for alias in model_with_aliases.aliases:
+            model_aliases[alias] = model_with_aliases.model
+        model_aliases[model_with_aliases.model.model_id] = model_with_aliases.model
+    return model_aliases
+
+
+class UnknownModelError(KeyError):
+    pass
+
+
+def get_model(name):
+    aliases = get_model_aliases()
+    try:
+        return aliases[name]
+    except KeyError:
+        raise UnknownModelError(name)
+
+
+def get_key(key_arg, default_key, env_var=None):
+    keys = load_keys()
+    if key_arg in keys:
+        return keys[key_arg]
+    if key_arg:
+        return key_arg
+    if env_var and os.environ.get(env_var):
+        return os.environ[env_var]
+    return keys.get(default_key)
+
+
+def load_keys():
+    path = user_dir() / "keys.json"
+    if path.exists():
+        return json.loads(path.read_text())
+    else:
+        return {}
+
+
+def user_dir():
+    llm_user_path = os.environ.get("LLM_USER_PATH")
+    if llm_user_path:
+        return pathlib.Path(llm_user_path)
+    return pathlib.Path(click.get_app_dir("io.datasette.llm"))
diff --git a/llm/cli.py b/llm/cli.py
index 97a84491..e8c8842e 100644
--- a/llm/cli.py
+++ b/llm/cli.py
@@ -1,20 +1,24 @@
 import click
 from click_default_group import DefaultGroup
-import datetime
 import json
-from llm import Template
+from llm import (
+    Template,
+    get_key,
+    get_plugins,
+    get_model,
+    get_model_aliases,
+    get_models_with_aliases,
+    user_dir,
+)
+
 from .migrations import migrate
-from .plugins import pm, get_plugins
-import openai
-import os
+from .plugins import pm
 import pathlib
 import pydantic
 from runpy import run_module
 import shutil
 import sqlite_utils
-from string import Template as StringTemplate
 import sys
-import time
 import warnings
 import yaml
 
@@ -22,16 +26,6 @@
 
 DEFAULT_MODEL = "gpt-3.5-turbo"
 
-MODEL_ALIASES = {
-    "4": "gpt-4",
-    "gpt4": "gpt-4",
-    "4-32k": "gpt-4-32k",
-    "chatgpt": "gpt-3.5-turbo",
-    "3.5": "gpt-3.5-turbo",
-    "chatgpt-16k": "gpt-3.5-turbo-16k",
-    "3.5-16k": "gpt-3.5-turbo-16k",
-}
-
 DEFAULT_TEMPLATE = "prompt: "
 
 
@@ -61,8 +55,16 @@ def cli():
 
 @cli.command(name="prompt")
 @click.argument("prompt", required=False)
-@click.option("--system", help="System prompt to use")
-@click.option("-m", "--model", help="Model to use")
+@click.option("-s", "--system", help="System prompt to use")
+@click.option("model_id", "-m", "--model", help="Model to use")
+@click.option(
+    "options",
+    "-o",
+    "--option",
+    type=(str, str),
+    multiple=True,
+    help="key/value options for the model",
+)
 @click.option("-t", "--template", help="Template to use")
 @click.option(
     "-p",
@@ -92,7 +94,8 @@ def cli():
 def prompt(
     prompt,
     system,
-    model,
+    model_id,
+    options,
     template,
     param,
     no_stream,
@@ -107,34 +110,43 @@ def prompt(
 
     Documentation: https://llm.datasette.io/en/stable/usage.html
     """
-    if prompt is None:
-        if template:
-            # If running a template only consume from stdin if it has data
-            if not sys.stdin.isatty():
+    model_aliases = get_model_aliases()
+
+    def read_prompt():
+        nonlocal prompt
+        if prompt is None:
+            if template:
+                # If running a template only consume from stdin if it has data
+                if not sys.stdin.isatty():
+                    prompt = sys.stdin.read()
+            elif not save:
+                # Hang waiting for input to stdin (unless --save)
                 prompt = sys.stdin.read()
-        elif not save:
-            # Hang waiting for input to stdin (unless --save)
-            prompt = sys.stdin.read()
+        return prompt
 
     if save:
         # We are saving their prompt/system/etc to a new template
         # Fields to save: prompt, system, model - and more in the future
-        bad_options = []
+        disallowed_options = []
         for option, var in (
             ("--template", template),
             ("--continue", _continue),
             ("--chat", chat_id),
         ):
             if var:
-                bad_options.append(option)
-        if bad_options:
+                disallowed_options.append(option)
+        if disallowed_options:
             raise click.ClickException(
-                "--save cannot be used with {}".format(", ".join(bad_options))
+                "--save cannot be used with {}".format(", ".join(disallowed_options))
             )
         path = template_dir() / f"{save}.yaml"
         to_save = {}
-        if model:
-            to_save["model"] = MODEL_ALIASES.get(model, model)
+        if model_id:
+            try:
+                to_save["model"] = model_aliases[model_id].model_id
+            except KeyError:
+                raise click.ClickException("'{}' is not a known model".format(model_id))
+        prompt = read_prompt()
         if prompt:
             to_save["prompt"] = prompt
         if system:
@@ -151,79 +163,100 @@ def prompt(
         )
         return
 
-    openai.api_key = get_key(key, "openai", "OPENAI_API_KEY")
-
     if template:
         params = dict(param)
         # Cannot be used with system
         if system:
             raise click.ClickException("Cannot use -t/--template and --system together")
         template_obj = load_template(template)
+        prompt = read_prompt()
         try:
-            prompt, system = template_obj.execute(prompt, params)
+            prompt, system = template_obj.evaluate(prompt, params)
         except Template.MissingVariables as ex:
             raise click.ClickException(str(ex))
-        if model is None and template_obj.model:
-            model = template_obj.model
-
-    messages = []
-    if _continue:
-        _continue = -1
-        if chat_id:
-            raise click.ClickException("Cannot use --continue and --chat together")
-    else:
-        _continue = chat_id
-    chat_id, history = get_history(_continue)
+        if model_id is None and template_obj.model:
+            model_id = template_obj.model
+
     history_model = None
-    if history:
-        for entry in history:
-            if entry.get("system"):
-                messages.append({"role": "system", "content": entry["system"]})
-            messages.append({"role": "user", "content": entry["prompt"]})
-            messages.append({"role": "assistant", "content": entry["response"]})
-            history_model = entry["model"]
-    if system:
-        messages.append({"role": "system", "content": system})
-    messages.append({"role": "user", "content": prompt})
-    if model is None:
-        model = history_model or DEFAULT_MODEL
-    else:
-        # Resolve model aliases
-        model = MODEL_ALIASES.get(model, model)
+
+    if _continue or chat_id:
+        raise click.ClickException("--continue mode is not yet implemented")
+    # TODO: Re-introduce --continue mode
+    # messages = []
+    # if _continue:
+    #     _continue = -1
+    #     if chat_id:
+    #         raise click.ClickException("Cannot use --continue and --chat together")
+    # else:
+    #     _continue = chat_id
+    # chat_id, history = get_history(_continue)
+    # history_model = None
+    # if history:
+    #     for entry in history:
+    #         if entry.get("system"):
+    #             messages.append({"role": "system", "content": entry["system"]})
+    #         messages.append({"role": "user", "content": entry["prompt"]})
+    #         messages.append({"role": "assistant", "content": entry["response"]})
+    #         history_model = entry["model"]
+
+    # Figure out which model we are using
+    if model_id is None:
+        model_id = history_model or get_default_model()
+
+    # Now resolve the model
     try:
-        debug = {}
-        if no_stream:
-            start = time.time()
-            response = openai.ChatCompletion.create(
-                model=model,
-                messages=messages,
+        model = model_aliases[model_id]
+    except KeyError:
+        raise click.ClickException("'{}' is not a known model".format(model_id))
+
+    # Provide the API key, if one is needed and has been provided
+    if model.needs_key:
+        model.key = get_key(key, model.needs_key, model.key_env_var)
+
+    # Validate options
+    validated_options = {}
+    if options:
+        # Validate with pydantic
+        try:
+            validated_options = dict(
+                (key, value)
+                for key, value in model.Options(**dict(options))
+                if value is not None
             )
-            debug["model"] = response.model
-            debug["usage"] = response.usage
-            content = response.choices[0].message.content
-            log(no_log, system, prompt, content, model, chat_id, debug, start)
-            print(content)
-        else:
-            start = time.time()
-            response = []
-            for chunk in openai.ChatCompletion.create(
-                model=model,
-                messages=messages,
-                stream=True,
-            ):
-                debug["model"] = chunk.model
-                content = chunk["choices"][0].get("delta", {}).get("content")
-                if content is not None:
-                    response.append(content)
-                    print(content, end="")
-                    sys.stdout.flush()
+        except pydantic.ValidationError as ex:
+            raise click.ClickException(render_errors(ex.errors()))
+
+    should_stream = model.can_stream and not no_stream
+    if not should_stream:
+        validated_options["stream"] = False
+
+    prompt = read_prompt()
+
+    try:
+        response = model.prompt(prompt, system, **validated_options)
+        if should_stream:
+            for chunk in response:
+                print(chunk, end="")
+                sys.stdout.flush()
             print("")
-            log(no_log, system, prompt, "".join(response), model, chat_id, debug, start)
-    except openai.error.AuthenticationError as ex:
-        raise click.ClickException("{}: {}".format(ex.error.type, ex.error.code))
-    except openai.error.OpenAIError as ex:
+        else:
+            print(response.text())
+    except Exception as ex:
         raise click.ClickException(str(ex))
 
+    # Log to the database
+    if no_log:
+        return
+    log_path = logs_db_path()
+    if not log_path.exists():
+        return
+    db = sqlite_utils.Database(log_path)
+    migrate(db)
+
+    response.log_to_db(db)
+
+    # TODO: Figure out OpenAI exception handling
+
 
 @cli.command()
 def init_db():
@@ -232,7 +265,7 @@ def init_db():
 
     All subsequent prompts will be logged to this database.
     """
-    path = log_db_path()
+    path = logs_db_path()
     if path.exists():
         return
     # Ensure directory exists
@@ -249,15 +282,7 @@ def keys():
 @keys.command(name="path")
 def keys_path_command():
     "Output the path to the keys.json file"
-    click.echo(keys_path())
-
-
-def keys_path():
-    llm_keys_path = os.environ.get("LLM_KEYS_PATH")
-    if llm_keys_path:
-        return pathlib.Path(llm_keys_path)
-    else:
-        return user_dir() / "keys.json"
+    click.echo(user_dir() / "keys.json")
 
 
 @keys.command(name="set")
@@ -274,7 +299,7 @@ def set_(name, value):
         Enter key: ...
     """
     default = {"// Note": "This file stores secret API credentials. Do not share!"}
-    path = pathlib.Path(keys_path())
+    path = user_dir() / "keys.json"
     path.parent.mkdir(parents=True, exist_ok=True)
     if not path.exists():
         path.write_text(json.dumps(default))
@@ -298,7 +323,7 @@ def logs():
 @logs.command(name="path")
 def logs_path():
     "Output the path to the logs.db file"
-    click.echo(log_db_path())
+    click.echo(logs_db_path())
 
 
 @logs.command(name="list")
@@ -317,19 +342,54 @@ def logs_path():
 @click.option("-t", "--truncate", is_flag=True, help="Truncate long strings in output")
 def logs_list(count, path, truncate):
     "Show recent logged prompts and their responses"
-    path = pathlib.Path(path or log_db_path())
+    path = pathlib.Path(path or logs_db_path())
     if not path.exists():
         raise click.ClickException("No log database found at {}".format(path))
     db = sqlite_utils.Database(path)
     migrate(db)
-    rows = list(db["log"].rows_where(order_by="-id", limit=count or None))
-    if truncate:
-        for row in rows:
+    rows = list(db["logs"].rows_where(order_by="-id", limit=count or None))
+    for row in rows:
+        if truncate:
             row["prompt"] = _truncate_string(row["prompt"])
             row["response"] = _truncate_string(row["response"])
+        # Decode all JSON keys
+        for key in row:
+            if key.endswith("_json") and row[key] is not None:
+                row[key] = json.loads(row[key])
     click.echo(json.dumps(list(rows), indent=2))
 
 
+@cli.group()
+def models():
+    "Manage available models"
+
+
+@models.command(name="list")
+def models_list():
+    "List available models"
+    for model_with_aliases in get_models_with_aliases():
+        extra = ""
+        if model_with_aliases.aliases:
+            extra = " (aliases: {})".format(", ".join(model_with_aliases.aliases))
+        output = str(model_with_aliases.model) + extra
+        click.echo(output)
+
+
+@models.command(name="default")
+@click.argument("model", required=False)
+def models_default(model):
+    "Show or set the default model"
+    if not model:
+        click.echo(get_default_model())
+        return
+    # Validate it is a known model
+    try:
+        model = get_model(model)
+        set_default_model(model.model_id)
+    except KeyError:
+        raise click.ClickException("Unknown model: {}".format(model))
+
+
 @cli.group()
 def templates():
     "Manage stored prompt templates"
@@ -410,15 +470,23 @@ def templates_path():
 
 
 @cli.command()
-@click.argument("packages", nargs=-1, required=True)
+@click.argument("packages", nargs=-1, required=False)
 @click.option(
     "-U", "--upgrade", is_flag=True, help="Upgrade packages to latest version"
 )
-def install(packages, upgrade):
+@click.option(
+    "-e",
+    "--editable",
+    type=click.Path(readable=True, exists=True, dir_okay=True, file_okay=False),
+    help="Install a project in editable mode from this path",
+)
+def install(packages, upgrade, editable):
     """Install packages from PyPI into the same environment as LLM"""
     args = ["pip", "install"]
     if upgrade:
         args += ["--upgrade"]
+    if editable:
+        args += ["--editable", str(editable)]
     args += list(packages)
     sys.argv = args
     run_module("pip", run_name="__main__")
@@ -434,11 +502,7 @@ def uninstall(packages, yes):
 
 
 def template_dir():
-    llm_templates_path = os.environ.get("LLM_TEMPLATES_PATH")
-    if llm_templates_path:
-        path = pathlib.Path(llm_templates_path)
-    else:
-        path = user_dir() / "templates"
+    path = user_dir() / "templates"
     path.mkdir(parents=True, exist_ok=True)
     return path
 
@@ -449,67 +513,21 @@ def _truncate_string(s, max_length=100):
     return s
 
 
-def get_key(key_arg, default_key, env_var=None):
-    keys = load_keys()
-    if key_arg in keys:
-        return keys[key_arg]
-    if env_var and os.environ.get(env_var):
-        return os.environ[env_var]
-    if key_arg:
-        return key_arg
-    default = keys.get(default_key)
-    if not default:
-        message = "No key found - add one using 'llm keys set {}'".format(default_key)
-        if env_var:
-            message += " or set the {} environment variable".format(env_var)
-        raise click.ClickException(message)
-    return default
-
-
-def load_keys():
-    path = pathlib.Path(keys_path())
+def get_default_model():
+    path = user_dir() / "default_model.txt"
     if path.exists():
-        return json.loads(path.read_text())
+        return path.read_text().strip()
     else:
-        return {}
+        return DEFAULT_MODEL
 
 
-def user_dir():
-    return pathlib.Path(click.get_app_dir("io.datasette.llm"))
+def set_default_model(model):
+    path = user_dir() / "default_model.txt"
+    path.write_text(model)
 
 
-def log_db_path():
-    llm_log_path = os.environ.get("LLM_LOG_PATH")
-    if llm_log_path:
-        return pathlib.Path(llm_log_path)
-    else:
-        return user_dir() / "logs.db"
-
-
-def log(no_log, system, prompt, response, model, chat_id=None, debug=None, start=None):
-    duration_ms = None
-    if start is not None:
-        end = time.time()
-        duration_ms = int((end - start) * 1000)
-    if no_log:
-        return
-    log_path = log_db_path()
-    if not log_path.exists():
-        return
-    db = sqlite_utils.Database(log_path)
-    migrate(db)
-    db["log"].insert(
-        {
-            "system": system,
-            "prompt": prompt,
-            "chat_id": chat_id,
-            "response": response,
-            "model": model,
-            "timestamp": str(datetime.datetime.utcnow()),
-            "debug": debug,
-            "duration_ms": duration_ms,
-        },
-    )
+def logs_db_path():
+    return user_dir() / "logs.db"
 
 
 def load_template(name):
@@ -524,18 +542,17 @@ def load_template(name):
         return Template(name=name, prompt=loaded)
     loaded["name"] = name
     try:
-        return Template.parse_obj(loaded)
-    except pydantic.ValidationError as e:
-        msg = "A validation error occurred:"
-        for error in e.errors():
-            msg += f"\n  {error['loc'][0]}: {error['msg']}"
+        return Template.model_validate(loaded)
+    except pydantic.ValidationError as ex:
+        msg = "A validation error occurred:\n"
+        msg += render_errors(ex.errors())
         raise click.ClickException(msg)
 
 
 def get_history(chat_id):
     if chat_id is None:
         return None, []
-    log_path = log_db_path()
+    log_path = logs_db_path()
     if not log_path.exists():
         raise click.ClickException(
             "This feature requires logging. Run `llm init-db` to create logs.db"
@@ -544,15 +561,23 @@ def get_history(chat_id):
     migrate(db)
     if chat_id == -1:
         # Return the most recent chat
-        last_row = list(db["log"].rows_where(order_by="-id", limit=1))
+        last_row = list(db["logs"].rows_where(order_by="-id", limit=1))
         if last_row:
             chat_id = last_row[0].get("chat_id") or last_row[0].get("id")
         else:  # Database is empty
             return None, []
-    rows = db["log"].rows_where(
+    rows = db["logs"].rows_where(
         "id = ? or chat_id = ?", [chat_id, chat_id], order_by="id"
     )
     return chat_id, rows
 
 
+def render_errors(errors):
+    output = []
+    for error in errors:
+        output.append(", ".join(error["loc"]))
+        output.append("  " + error["msg"])
+    return "\n".join(output)
+
+
 pm.hook.register_commands(cli=cli)
diff --git a/llm/default_plugins/__init__.py b/llm/default_plugins/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/llm/default_plugins/openai_models.py b/llm/default_plugins/openai_models.py
new file mode 100644
index 00000000..8388c517
--- /dev/null
+++ b/llm/default_plugins/openai_models.py
@@ -0,0 +1,172 @@
+from llm import Model, hookimpl
+import llm
+from llm.utils import dicts_to_table_string
+import click
+import datetime
+import openai
+from pydantic import field_validator
+import requests
+from typing import List, Optional, Union
+import json
+
+
+@hookimpl
+def register_models(register):
+    register(Chat("gpt-3.5-turbo"), aliases=("3.5", "chatgpt"))
+    register(Chat("gpt-3.5-turbo-16k"), aliases=("chatgpt-16k", "3.5-16k"))
+    register(Chat("gpt-4"), aliases=("4", "gpt4"))
+    register(Chat("gpt-4-32k"), aliases=("4-32k",))
+
+
+@hookimpl
+def register_commands(cli):
+    @cli.group(name="openai")
+    def openai_():
+        "Commands for working directly with the OpenAI API"
+
+    @openai_.command()
+    @click.option("json_", "--json", is_flag=True, help="Output as JSON")
+    @click.option("--key", help="OpenAI API key")
+    def models(json_, key):
+        "List models available to you from the OpenAI API"
+        from llm.cli import get_key
+
+        api_key = get_key(key, "openai", "OPENAI_API_KEY")
+        response = requests.get(
+            "https://api.openai.com/v1/models",
+            headers={"Authorization": f"Bearer {api_key}"},
+        )
+        if response.status_code != 200:
+            raise click.ClickException(
+                f"Error {response.status_code} from OpenAI API: {response.text}"
+            )
+        models = response.json()["data"]
+        if json_:
+            click.echo(json.dumps(models, indent=4))
+        else:
+            to_print = []
+            for model in models:
+                # Print id, owned_by, root, created as ISO 8601
+                created_str = datetime.datetime.utcfromtimestamp(
+                    model["created"]
+                ).isoformat()
+                to_print.append(
+                    {
+                        "id": model["id"],
+                        "owned_by": model["owned_by"],
+                        "created": created_str,
+                    }
+                )
+            done = dicts_to_table_string("id owned_by created".split(), to_print)
+            print("\n".join(done))
+
+
+class Chat(Model):
+    needs_key = "openai"
+    key_env_var = "OPENAI_API_KEY"
+    can_stream: bool = True
+
+    class Options(llm.Options):
+        temperature: Optional[float] = None
+        max_tokens: Optional[int] = None
+        top_p: Optional[float] = None
+        frequency_penalty: Optional[float] = None
+        presence_penalty: Optional[float] = None
+        stop: Optional[str] = None
+        logit_bias: Optional[Union[dict, str]] = None
+
+        @field_validator("logit_bias")
+        def validate_logit_bias(cls, logit_bias):
+            if logit_bias is None:
+                return None
+
+            if isinstance(logit_bias, str):
+                try:
+                    logit_bias = json.loads(logit_bias)
+                except json.JSONDecodeError:
+                    raise ValueError("Invalid JSON in logit_bias string")
+
+            validated_logit_bias = {}
+            for key, value in logit_bias.items():
+                try:
+                    int_key = int(key)
+                    int_value = int(value)
+                    if -100 <= int_value <= 100:
+                        validated_logit_bias[int_key] = int_value
+                    else:
+                        raise ValueError("Value must be between -100 and 100")
+                except ValueError:
+                    raise ValueError("Invalid key-value pair in logit_bias dictionary")
+
+            return validated_logit_bias
+
+    class Response(llm.Response):
+        def __init__(self, prompt, model, stream, key):
+            super().__init__(prompt, model, stream)
+            self.key = key
+
+    def __init__(self, model_id, key=None):
+        self.model_id = model_id
+        self.key = key
+
+    def __str__(self):
+        return "OpenAI Chat: {}".format(self.model_id)
+
+    def execute(self, prompt, stream, response):
+        messages = []
+        if prompt.system:
+            messages.append({"role": "system", "content": prompt.system})
+        messages.append({"role": "user", "content": prompt.prompt})
+        openai.api_key = self.key
+        response._prompt_json = {"messages": messages}
+        if stream:
+            completion = openai.ChatCompletion.create(
+                model=prompt.model.model_id,
+                messages=messages,
+                stream=True,
+                **not_nulls(prompt.options),
+            )
+            chunks = []
+            for chunk in completion:
+                chunks.append(chunk)
+                content = chunk["choices"][0].get("delta", {}).get("content")
+                if content is not None:
+                    yield content
+            response._response_json = combine_chunks(chunks)
+        else:
+            completion = openai.ChatCompletion.create(
+                model=prompt.model.model_id,
+                messages=messages,
+                stream=False,
+            )
+            response._response_json = completion.to_dict_recursive()
+            yield completion.choices[0].message.content
+
+
+def not_nulls(data) -> dict:
+    return {key: value for key, value in data if value is not None}
+
+
+def combine_chunks(chunks: List[dict]) -> dict:
+    content = ""
+    role = None
+
+    for item in chunks:
+        for choice in item["choices"]:
+            if "role" in choice["delta"]:
+                role = choice["delta"]["role"]
+            if "content" in choice["delta"]:
+                content += choice["delta"]["content"]
+            if choice["finish_reason"] is not None:
+                finish_reason = choice["finish_reason"]
+
+    return {
+        "id": chunks[0]["id"],
+        "object": chunks[0]["object"],
+        "model": chunks[0]["model"],
+        "created": chunks[0]["created"],
+        "index": chunks[0]["choices"][0]["index"],
+        "role": role,
+        "content": content,
+        "finish_reason": finish_reason,
+    }
diff --git a/llm/errors.py b/llm/errors.py
new file mode 100644
index 00000000..16014389
--- /dev/null
+++ b/llm/errors.py
@@ -0,0 +1,3 @@
+class NeedsKeyException(Exception):
+    "Model needs an API key which has not been provided"
+    pass
diff --git a/llm/hookspecs.py b/llm/hookspecs.py
index 88422de7..d3a00d4d 100644
--- a/llm/hookspecs.py
+++ b/llm/hookspecs.py
@@ -8,3 +8,8 @@
 @hookspec
 def register_commands(cli):
     """Register additional CLI commands, e.g. 'llm mycommand ...'"""
+
+
+@hookspec
+def register_models(register):
+    "Return a list of model instances representing LLM models that can be called"
diff --git a/llm/migrations.py b/llm/migrations.py
index 4d76439c..ab47ac22 100644
--- a/llm/migrations.py
+++ b/llm/migrations.py
@@ -1,6 +1,7 @@
 import datetime
+from typing import Callable, List
 
-MIGRATIONS = []
+MIGRATIONS: List[Callable] = []
 migration = MIGRATIONS.append
 
 
@@ -33,7 +34,7 @@ def m001_initial(db):
     # Ensure the original table design exists, so other migrations can run
     if db["log"].exists():
         # It needs to have the chat_id column
-        if not "chat_id" in db["log"].columns_dict:
+        if "chat_id" not in db["log"].columns_dict:
             db["log"].add_column("chat_id")
         return
     db["log"].create(
@@ -84,3 +85,76 @@ def m004_drop_provider(db):
 def m005_debug(db):
     db["log"].add_column("debug", str)
     db["log"].add_column("duration_ms", int)
+
+
+@migration
+def m006_new_logs_table(db):
+    columns = db["log"].columns_dict
+    for column, type in (
+        ("options_json", str),
+        ("prompt_json", str),
+        ("response_json", str),
+        ("reply_to_id", int),
+    ):
+        # It's possible people running development code like myself
+        # might have accidentally created these columns already
+        if column not in columns:
+            db["log"].add_column(column, type)
+
+    # Use .transform() to rename options and timestamp_utc, and set new order
+    db["log"].transform(
+        column_order=(
+            "id",
+            "model",
+            "prompt",
+            "system",
+            "prompt_json",
+            "options_json",
+            "response",
+            "response_json",
+            "reply_to_id",
+            "chat_id",
+            "duration_ms",
+            "timestamp_utc",
+        ),
+        rename={
+            "timestamp": "timestamp_utc",
+            "options": "options_json",
+        },
+    )
+
+
+@migration
+def m007_finish_logs_table(db):
+    db["log"].transform(
+        drop={"debug"},
+        rename={"timestamp_utc": "datetime_utc"},
+    )
+    with db.conn:
+        db.execute("alter table log rename to logs")
+
+
+@migration
+def m008_reply_to_id_foreign_key(db):
+    db["logs"].add_foreign_key("reply_to_id", "logs", "id")
+
+
+@migration
+def m008_fix_column_order_in_logs(db):
+    # reply_to_id ended up at the end after foreign key added
+    db["logs"].transform(
+        column_order=(
+            "id",
+            "model",
+            "prompt",
+            "system",
+            "prompt_json",
+            "options_json",
+            "response",
+            "response_json",
+            "reply_to_id",
+            "chat_id",
+            "duration_ms",
+            "timestamp_utc",
+        ),
+    )
diff --git a/llm/models.py b/llm/models.py
new file mode 100644
index 00000000..1dddec02
--- /dev/null
+++ b/llm/models.py
@@ -0,0 +1,218 @@
+from dataclasses import dataclass, asdict
+import datetime
+from .errors import NeedsKeyException
+import time
+from typing import Any, Callable, Dict, Iterator, List, Optional, Set, Union
+from abc import ABC, abstractmethod
+import os
+from pydantic import ConfigDict, BaseModel
+
+
+@dataclass
+class Prompt:
+    prompt: str
+    model: "Model"
+    system: Optional[str]
+    prompt_json: Optional[str]
+    options: "Options"
+
+    def __init__(self, prompt, model, system=None, prompt_json=None, options=None):
+        self.prompt = prompt
+        self.model = model
+        self.system = system
+        self.prompt_json = prompt_json
+        self.options = options or {}
+
+
+class OptionsError(Exception):
+    pass
+
+
+@dataclass
+class LogMessage:
+    model: str  # Actually the model.model_id string
+    prompt: str  # Simplified string version of prompt
+    system: Optional[str]  # Simplified string of system prompt
+    prompt_json: Optional[Dict[str, Any]]  # Detailed JSON of prompt
+    options_json: Dict[str, Any]  # Any options e.g. temperature
+    response: str  # Simplified string version of response
+    response_json: Optional[Dict[str, Any]]  # Detailed JSON of response
+    reply_to_id: Optional[int]  # ID of message this is a reply to
+    chat_id: Optional[
+        int
+    ]  # ID of chat this is a part of (ID of first message in thread)
+
+
+class Response(ABC):
+    def __init__(self, prompt: Prompt, model: "Model", stream: bool):
+        self.prompt = prompt
+        self._prompt_json = None
+        self.model = model
+        self.stream = stream
+        self._chunks: List[str] = []
+        self._done = False
+        self._response_json = None
+
+    def reply(self, prompt, system=None, **options):
+        new_prompt = [self.prompt.prompt, self.text(), prompt]
+        return self.model.response(
+            Prompt(
+                "\n".join(new_prompt),
+                system=system or self.prompt.system or None,
+                model=self.model,
+                options=options,
+            ),
+            stream=self.stream,
+        )
+
+    def __iter__(self) -> Iterator[str]:
+        self._start = time.monotonic()
+        self._start_utcnow = datetime.datetime.utcnow()
+        if self._done:
+            return self._chunks
+        for chunk in self.model.execute(self.prompt, stream=self.stream, response=self):
+            yield chunk
+            self._chunks.append(chunk)
+        self._end = time.monotonic()
+        self._done = True
+
+    def _force(self):
+        if not self._done:
+            list(self)
+
+    def text(self) -> str:
+        self._force()
+        return "".join(self._chunks)
+
+    def json(self) -> Optional[Dict[str, Any]]:
+        self._force()
+        return self._response_json
+
+    def duration_ms(self) -> int:
+        self._force()
+        return int((self._end - self._start) * 1000)
+
+    def datetime_utc(self) -> str:
+        self._force()
+        return self._start_utcnow.isoformat()
+
+    def log_message(self) -> LogMessage:
+        return LogMessage(
+            model=self.prompt.model.model_id,
+            prompt=self.prompt.prompt,
+            system=self.prompt.system,
+            prompt_json=self._prompt_json,
+            options_json={
+                key: value
+                for key, value in self.prompt.options.model_dump().items()
+                if value is not None
+            },
+            response=self.text(),
+            response_json=self.json(),
+            reply_to_id=None,  # TODO
+            chat_id=None,  # TODO
+        )
+
+    def log_to_db(self, db):
+        message = self.log_message()
+        message_dict = asdict(message)
+        message_dict["duration_ms"] = self.duration_ms()
+        message_dict["datetime_utc"] = self.datetime_utc()
+        db["logs"].insert(message_dict, pk="id")
+
+
+class Options(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+
+_Options = Options
+
+
+class Model(ABC):
+    model_id: str
+    key: Optional[str] = None
+    needs_key: Optional[str] = None
+    key_env_var: Optional[str] = None
+    can_stream: bool = False
+
+    class Options(_Options):
+        pass
+
+    def get_key(self):
+        if self.needs_key is None:
+            return None
+        if self.key is not None:
+            return self.key
+        if self.key_env_var is not None:
+            key = os.environ.get(self.key_env_var)
+            if key:
+                return key
+
+        message = "No key found - add one using 'llm keys set {}'".format(
+            self.needs_key
+        )
+        if self.key_env_var:
+            message += " or set the {} environment variable".format(self.key_env_var)
+        raise NeedsKeyException(message)
+
+    @abstractmethod
+    def execute(
+        self, prompt: Prompt, stream: bool, response: Response
+    ) -> Iterator[str]:
+        """
+        Execute a prompt and yield chunks of text, or yield a single big chunk.
+        Any additional useful information about the execution should be assigned to the response.
+        """
+        pass
+
+    def prompt(
+        self,
+        prompt: Optional[str],
+        system: Optional[str] = None,
+        stream: bool = True,
+        **options
+    ):
+        return self.response(
+            Prompt(prompt, system=system, model=self, options=self.Options(**options)),
+            stream=stream,
+        )
+
+    def chain(
+        self,
+        prompt: Union[Prompt, str],
+        system: Optional[str] = None,
+        stream: bool = True,
+        proceed: Optional[Callable] = None,
+        **options
+    ):
+        if proceed is None:
+
+            def proceed(response):
+                return None
+
+        while True:
+            if isinstance(prompt, str):
+                prompt = Prompt(
+                    prompt, model=self, system=system, options=self.Options(**options)
+                )
+            response = self.response(
+                prompt,
+                stream=stream,
+            )
+            yield response
+            next_prompt = proceed(response)
+            if not next_prompt:
+                break
+            prompt = next_prompt
+
+    def response(self, prompt: Prompt, stream: bool = True) -> Response:
+        return Response(prompt, self, stream)
+
+    def __str__(self) -> str:
+        return "{}: {}".format(self.__class__.__name__, self.model_id)
+
+
+@dataclass
+class ModelWithAliases:
+    model: Model
+    aliases: Set[str]
diff --git a/llm/plugins.py b/llm/plugins.py
index 97e3b825..230b41c3 100644
--- a/llm/plugins.py
+++ b/llm/plugins.py
@@ -1,7 +1,10 @@
+import importlib
 import pluggy
 import sys
 from . import hookspecs
 
+DEFAULT_PLUGINS = ("llm.default_plugins.openai_models",)
+
 pm = pluggy.PluginManager("llm")
 pm.add_hookspecs(hookspecs)
 
@@ -9,18 +12,6 @@
     # Only load plugins if not running tests
     pm.load_setuptools_entrypoints("llm")
 
-
-def get_plugins():
-    plugins = []
-    plugin_to_distinfo = dict(pm.list_plugin_distinfo())
-    for plugin in pm.get_plugins():
-        plugin_info = {
-            "name": plugin.__name__,
-            "hooks": [h.name for h in pm.get_hookcallers(plugin)],
-        }
-        distinfo = plugin_to_distinfo.get(plugin)
-        if distinfo:
-            plugin_info["version"] = distinfo.version
-            plugin_info["name"] = distinfo.project_name
-        plugins.append(plugin_info)
-    return plugins
+for plugin in DEFAULT_PLUGINS:
+    mod = importlib.import_module(plugin)
+    pm.register(mod, plugin)
diff --git a/llm/templates.py b/llm/templates.py
new file mode 100644
index 00000000..4a18a44e
--- /dev/null
+++ b/llm/templates.py
@@ -0,0 +1,55 @@
+from pydantic import ConfigDict, BaseModel
+import string
+from typing import Optional, Any, Dict, List, Tuple
+
+
+class Template(BaseModel):
+    name: str
+    prompt: Optional[str] = None
+    system: Optional[str] = None
+    model: Optional[str] = None
+    defaults: Optional[Dict[str, Any]] = None
+    model_config = ConfigDict(extra="forbid")
+
+    class MissingVariables(Exception):
+        pass
+
+    def evaluate(
+        self, input: str, params: Optional[Dict[str, Any]] = None
+    ) -> Tuple[Optional[str], Optional[str]]:
+        params = params or {}
+        params["input"] = input
+        if self.defaults:
+            for k, v in self.defaults.items():
+                if k not in params:
+                    params[k] = v
+        prompt: Optional[str] = None
+        system: Optional[str] = None
+        if not self.prompt:
+            system = self.interpolate(self.system, params)
+            prompt = input
+        else:
+            prompt = self.interpolate(self.prompt, params)
+            system = self.interpolate(self.system, params)
+        return prompt, system
+
+    @classmethod
+    def interpolate(cls, text: Optional[str], params: Dict[str, Any]) -> Optional[str]:
+        if not text:
+            return text
+        # Confirm all variables in text are provided
+        string_template = string.Template(text)
+        vars = cls.extract_vars(string_template)
+        missing = [p for p in vars if p not in params]
+        if missing:
+            raise cls.MissingVariables(
+                "Missing variables: {}".format(", ".join(missing))
+            )
+        return string_template.substitute(**params)
+
+    @staticmethod
+    def extract_vars(string_template: string.Template) -> List[str]:
+        return [
+            match.group("named")
+            for match in string_template.pattern.finditer(string_template.template)
+        ]
diff --git a/llm/utils.py b/llm/utils.py
new file mode 100644
index 00000000..cf9a685a
--- /dev/null
+++ b/llm/utils.py
@@ -0,0 +1,25 @@
+from typing import List, Dict
+
+
+def dicts_to_table_string(
+    headings: List[str], dicts: List[Dict[str, str]]
+) -> List[str]:
+    max_lengths = [len(h) for h in headings]
+
+    # Compute maximum length for each column
+    for d in dicts:
+        for i, h in enumerate(headings):
+            if h in d and len(str(d[h])) > max_lengths[i]:
+                max_lengths[i] = len(str(d[h]))
+
+    # Generate formatted table strings
+    res = []
+    res.append("    ".join(h.ljust(max_lengths[i]) for i, h in enumerate(headings)))
+
+    for d in dicts:
+        row = []
+        for i, h in enumerate(headings):
+            row.append(str(d.get(h, "")).ljust(max_lengths[i]))
+        res.append("    ".join(row))
+
+    return res
diff --git a/mypy.ini b/mypy.ini
new file mode 100644
index 00000000..d025b7cf
--- /dev/null
+++ b/mypy.ini
@@ -0,0 +1,8 @@
+[mypy]
+
+[mypy-pluggy.*]
+ignore_missing_imports = True
+
+[mypy-click_default_group.*]
+ignore_missing_imports = True
+
diff --git a/ruff.toml b/ruff.toml
new file mode 100644
index 00000000..7a3f6a3f
--- /dev/null
+++ b/ruff.toml
@@ -0,0 +1,1 @@
+line-length = 160
diff --git a/setup.py b/setup.py
index e27f35ce..c126240c 100644
--- a/setup.py
+++ b/setup.py
@@ -1,4 +1,4 @@
-from setuptools import setup
+from setuptools import setup, find_packages
 import os
 
 VERSION = "0.4.1"
@@ -27,7 +27,7 @@ def get_long_description():
     },
     license="Apache License, Version 2.0",
     version=VERSION,
-    packages=["llm"],
+    packages=find_packages(),
     entry_points="""
         [console_scripts]
         llm=llm.cli:cli
@@ -37,10 +37,22 @@ def get_long_description():
         "openai",
         "click-default-group-wheel",
         "sqlite-utils",
-        "pydantic<2.0",
+        "pydantic>=2.0.0",
         "PyYAML",
         "pluggy",
     ],
-    extras_require={"test": ["pytest", "requests-mock"]},
+    extras_require={
+        "test": [
+            "pytest",
+            "requests-mock",
+            "cogapp",
+            "mypy",
+            "black",
+            "ruff",
+            "types-click",
+            "types-PyYAML",
+            "types-requests",
+        ]
+    },
     python_requires=">=3.7",
 )


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index be12df66..aeca90c2 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -21,11 +21,19 @@ jobs:
         cache-dependency-path: setup.py
     - name: Install dependencies
       run: |
-        pip install '.[test]'
+        pip install -e '.[test]'
     - name: Run tests
       run: |
         pytest
     - name: Check if cog needs to be run
       run: |
-        pip install -r docs/requirements.txt
         cog --check docs/*.md
+    - name: Run Black
+      run: |
+        black --check .
+    - name: Run mypy
+      run: |
+        mypy llm
+    - name: Run ruff
+      run: |
+        ruff .
diff --git a/tests/conftest.py b/tests/conftest.py
index 78d3afa5..06f0f6e7 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -8,27 +8,22 @@ def pytest_configure(config):
 
 
 @pytest.fixture
-def log_path(tmpdir):
-    return tmpdir / "logs.db"
+def user_path(tmpdir):
+    dir = tmpdir / "llm.datasette.io"
+    dir.mkdir()
+    return dir
 
 
 @pytest.fixture
-def keys_path(tmpdir):
-    return tmpdir / "keys.json"
-
-
-@pytest.fixture
-def templates_path(tmpdir):
-    path = tmpdir / "templates"
-    path.mkdir()
-    return path
+def templates_path(user_path):
+    dir = user_path / "templates"
+    dir.mkdir()
+    return dir
 
 
 @pytest.fixture(autouse=True)
-def env_setup(monkeypatch, log_path, keys_path, templates_path):
-    monkeypatch.setenv("LLM_KEYS_PATH", str(keys_path))
-    monkeypatch.setenv("LLM_LOG_PATH", str(log_path))
-    monkeypatch.setenv("LLM_TEMPLATES_PATH", str(templates_path))
+def env_setup(monkeypatch, user_path):
+    monkeypatch.setenv("LLM_USER_PATH", str(user_path))
 
 
 @pytest.fixture
diff --git a/tests/test_cli_openai_models.py b/tests/test_cli_openai_models.py
new file mode 100644
index 00000000..fd5ff07a
--- /dev/null
+++ b/tests/test_cli_openai_models.py
@@ -0,0 +1,39 @@
+from click.testing import CliRunner
+from llm.cli import cli
+import pytest
+
+
+@pytest.fixture
+def mocked_models(requests_mock):
+    requests_mock.get(
+        "https://api.openai.com/v1/models",
+        json={
+            "data": [
+                {
+                    "id": "ada:2020-05-03",
+                    "object": "model",
+                    "created": 1588537600,
+                    "owned_by": "openai",
+                },
+                {
+                    "id": "babbage:2020-05-03",
+                    "object": "model",
+                    "created": 1588537600,
+                    "owned_by": "openai",
+                },
+            ]
+        },
+        headers={"Content-Type": "application/json"},
+    )
+    return requests_mock
+
+
+def test_openai_models(mocked_models):
+    runner = CliRunner()
+    result = runner.invoke(cli, ["openai", "models", "--key", "x"])
+    assert result.exit_code == 0
+    assert result.output == (
+        "id                    owned_by    created            \n"
+        "ada:2020-05-03        openai      2020-05-03T20:26:40\n"
+        "babbage:2020-05-03    openai      2020-05-03T20:26:40\n"
+    )
diff --git a/tests/test_keys.py b/tests/test_keys.py
index 0d84de9c..a36fac70 100644
--- a/tests/test_keys.py
+++ b/tests/test_keys.py
@@ -1,30 +1,31 @@
 from click.testing import CliRunner
 import json
 from llm.cli import cli
+import pathlib
 import pytest
 
 
-@pytest.mark.parametrize("env", ({}, {"LLM_KEYS_PATH": "/tmp/foo.json"}))
-def test_keys_path(monkeypatch, env, keys_path):
+@pytest.mark.parametrize("env", ({}, {"LLM_USER_PATH": "/tmp/llm-keys-test"}))
+def test_keys_in_user_path(monkeypatch, env, user_path):
     for key, value in env.items():
         monkeypatch.setenv(key, value)
     runner = CliRunner()
     result = runner.invoke(cli, ["keys", "path"])
     assert result.exit_code == 0
     if env:
-        expected = env["LLM_KEYS_PATH"]
+        expected = env["LLM_USER_PATH"] + "/keys.json"
     else:
-        expected = keys_path
+        expected = user_path + "/keys.json"
     assert result.output.strip() == expected
 
 
 def test_keys_set(monkeypatch, tmpdir):
-    keys_path = str(tmpdir / "keys.json")
-    monkeypatch.setenv("LLM_KEYS_PATH", keys_path)
+    user_path = str(tmpdir / "user/keys")
+    monkeypatch.setenv("LLM_USER_PATH", user_path)
     runner = CliRunner()
     result = runner.invoke(cli, ["keys", "set", "openai"], input="foo")
     assert result.exit_code == 0
-    content = open(keys_path).read()
+    content = open(user_path + "/keys.json").read()
     assert json.loads(content) == {
         "// Note": "This file stores secret API credentials. Do not share!",
         "openai": "foo",
@@ -32,7 +33,9 @@ def test_keys_set(monkeypatch, tmpdir):
 
 
 def test_uses_correct_key(mocked_openai, monkeypatch, tmpdir):
-    keys_path = tmpdir / "keys.json"
+    user_dir = tmpdir / "user-dir"
+    pathlib.Path(user_dir).mkdir()
+    keys_path = user_dir / "keys.json"
     keys_path.write_text(
         json.dumps(
             {
@@ -42,7 +45,7 @@ def test_uses_correct_key(mocked_openai, monkeypatch, tmpdir):
         ),
         "utf-8",
     )
-    monkeypatch.setenv("LLM_KEYS_PATH", str(keys_path))
+    monkeypatch.setenv("LLM_USER_PATH", str(user_dir))
     monkeypatch.setenv("OPENAI_API_KEY", "from-env")
 
     def assert_key(key):
diff --git a/tests/test_llm.py b/tests/test_llm.py
index 1faf32b4..b300839d 100644
--- a/tests/test_llm.py
+++ b/tests/test_llm.py
@@ -17,10 +17,11 @@ def test_version():
 
 
 @pytest.mark.parametrize("n", (None, 0, 2))
-def test_logs(n, log_path):
-    db = sqlite_utils.Database(str(log_path))
+def test_logs(n, user_path):
+    log_path = str(user_path / "logs.db")
+    db = sqlite_utils.Database(log_path)
     migrate(db)
-    db["log"].insert_all(
+    db["logs"].insert_all(
         {
             "system": "system",
             "prompt": "prompt",
@@ -45,26 +46,27 @@ def test_logs(n, log_path):
     assert len(logs) == expected_length
 
 
-@pytest.mark.parametrize("env", ({}, {"LLM_LOG_PATH": "/tmp/logs.db"}))
-def test_logs_path(monkeypatch, env, log_path):
+@pytest.mark.parametrize("env", ({}, {"LLM_USER_PATH": "/tmp/llm-user-path"}))
+def test_logs_path(monkeypatch, env, user_path):
     for key, value in env.items():
         monkeypatch.setenv(key, value)
     runner = CliRunner()
     result = runner.invoke(cli, ["logs", "path"])
     assert result.exit_code == 0
     if env:
-        expected = env["LLM_LOG_PATH"]
+        expected = env["LLM_USER_PATH"] + "/logs.db"
     else:
-        expected = str(log_path)
+        expected = str(user_path) + "/logs.db"
     assert result.output.strip() == expected
 
 
 @mock.patch.dict(os.environ, {"OPENAI_API_KEY": "X"})
 @pytest.mark.parametrize("use_stdin", (True, False))
-def test_llm_default_prompt(mocked_openai, use_stdin, log_path):
+def test_llm_default_prompt(mocked_openai, use_stdin, user_path):
     # Reset the log_path database
+    log_path = user_path / "logs.db"
     log_db = sqlite_utils.Database(str(log_path))
-    log_db["log"].delete_where()
+    log_db["logs"].delete_where()
     runner = CliRunner()
     prompt = "three names for a pet pelican"
     input = None
@@ -77,8 +79,9 @@ def test_llm_default_prompt(mocked_openai, use_stdin, log_path):
     assert result.exit_code == 0
     assert result.output == "Bob, Alice, Eve\n"
     assert mocked_openai.last_request.headers["Authorization"] == "Bearer X"
+
     # Was it logged?
-    rows = list(log_db["log"].rows)
+    rows = list(log_db["logs"].rows)
     assert len(rows) == 1
     expected = {
         "model": "gpt-3.5-turbo",
@@ -87,4 +90,50 @@ def test_llm_default_prompt(mocked_openai, use_stdin, log_path):
         "response": "Bob, Alice, Eve",
         "chat_id": None,
     }
-    assert expected.items() <= rows[0].items()
+    expected = {
+        "model": "gpt-3.5-turbo",
+        "prompt": "three names for a pet pelican",
+        "system": None,
+        "options_json": "{}",
+        "response": "Bob, Alice, Eve",
+        "reply_to_id": None,
+        "chat_id": None,
+    }
+    row = rows[0]
+    assert expected.items() <= row.items()
+    assert isinstance(row["duration_ms"], int)
+    assert isinstance(row["datetime_utc"], str)
+    assert json.loads(row["prompt_json"]) == {
+        "messages": [{"role": "user", "content": "three names for a pet pelican"}]
+    }
+    assert json.loads(row["response_json"]) == {
+        "model": "gpt-3.5-turbo",
+        "usage": {},
+        "choices": [{"message": {"content": "Bob, Alice, Eve"}}],
+    }
+
+    # Test "llm logs"
+    log_result = runner.invoke(cli, ["logs", "-n", "1"], catch_exceptions=False)
+    log_json = json.loads(log_result.output)
+    assert (
+        log_json[0].items()
+        >= {
+            "model": "gpt-3.5-turbo",
+            "prompt": "three names for a pet pelican",
+            "system": None,
+            "prompt_json": {
+                "messages": [
+                    {"role": "user", "content": "three names for a pet pelican"}
+                ]
+            },
+            "options_json": {},
+            "response": "Bob, Alice, Eve",
+            "response_json": {
+                "model": "gpt-3.5-turbo",
+                "usage": {},
+                "choices": [{"message": {"content": "Bob, Alice, Eve"}}],
+            },
+            "reply_to_id": None,
+            "chat_id": None,
+        }.items()
+    )
diff --git a/tests/test_migrate.py b/tests/test_migrate.py
index 9450af78..64ffa396 100644
--- a/tests/test_migrate.py
+++ b/tests/test_migrate.py
@@ -2,21 +2,38 @@
 import sqlite_utils
 
 
+EXPECTED = {
+    "id": int,
+    "model": str,
+    "prompt": str,
+    "system": str,
+    "prompt_json": str,
+    "options_json": str,
+    "response": str,
+    "response_json": str,
+    "reply_to_id": int,
+    "chat_id": int,
+    "duration_ms": int,
+    "datetime_utc": str,
+}
+
+
 def test_migrate_blank():
     db = sqlite_utils.Database(memory=True)
     migrate(db)
-    assert set(db.table_names()) == {"_llm_migrations", "log"}
-    assert db["log"].columns_dict == {
-        "id": int,
-        "model": str,
-        "timestamp": str,
-        "prompt": str,
-        "system": str,
-        "response": str,
-        "chat_id": int,
-        "debug": str,
-        "duration_ms": int,
-    }
+    assert set(db.table_names()) == {"_llm_migrations", "logs"}
+    assert db["logs"].columns_dict == EXPECTED
+
+    foreign_keys = db["logs"].foreign_keys
+    for expected_fk in (
+        sqlite_utils.db.ForeignKey(
+            table="logs", column="reply_to_id", other_table="logs", other_column="id"
+        ),
+        sqlite_utils.db.ForeignKey(
+            table="logs", column="chat_id", other_table="logs", other_column="id"
+        ),
+    ):
+        assert expected_fk in foreign_keys
 
 
 def test_migrate_from_original_schema():
@@ -33,15 +50,5 @@ def test_migrate_from_original_schema():
         },
     )
     migrate(db)
-    assert set(db.table_names()) == {"_llm_migrations", "log"}
-    assert db["log"].columns_dict == {
-        "id": int,
-        "model": str,
-        "timestamp": str,
-        "prompt": str,
-        "system": str,
-        "response": str,
-        "chat_id": int,
-        "debug": str,
-        "duration_ms": int,
-    }
+    assert set(db.table_names()) == {"_llm_migrations", "logs"}
+    assert db["logs"].columns_dict == EXPECTED
diff --git a/tests/test_plugins.py b/tests/test_plugins.py
index 029608c8..ff742c29 100644
--- a/tests/test_plugins.py
+++ b/tests/test_plugins.py
@@ -1,13 +1,17 @@
 from click.testing import CliRunner
 import click
 import importlib
+import llm
 from llm import cli, hookimpl, plugins
-import pytest
 
 
 def test_register_commands():
     importlib.reload(cli)
-    assert plugins.get_plugins() == []
+
+    def plugin_names():
+        return [plugin["name"] for plugin in llm.get_plugins()]
+
+    assert "HelloWorldPlugin" not in plugin_names()
 
     class HelloWorldPlugin:
         __name__ = "HelloWorldPlugin"
@@ -23,9 +27,7 @@ def hello_world():
         plugins.pm.register(HelloWorldPlugin(), name="HelloWorldPlugin")
         importlib.reload(cli)
 
-        assert plugins.get_plugins() == [
-            {"name": "HelloWorldPlugin", "hooks": ["register_commands"]}
-        ]
+        assert "HelloWorldPlugin" in plugin_names()
 
         runner = CliRunner()
         result = runner.invoke(cli.cli, ["hello-world"])
@@ -35,4 +37,4 @@ def hello_world():
     finally:
         plugins.pm.unregister(name="HelloWorldPlugin")
         importlib.reload(cli)
-        assert plugins.get_plugins() == []
+        assert "HelloWorldPlugin" not in plugin_names()
diff --git a/tests/test_templates.py b/tests/test_templates.py
index 02f43d43..08e6c424 100644
--- a/tests/test_templates.py
+++ b/tests/test_templates.py
@@ -27,16 +27,16 @@
         ),
     ),
 )
-def test_template_execute(
+def test_template_evaluate(
     prompt, system, defaults, params, expected_prompt, expected_system, expected_error
 ):
     t = Template(name="t", prompt=prompt, system=system, defaults=defaults)
     if expected_error:
         with pytest.raises(Template.MissingVariables) as ex:
-            prompt, system = t.execute("input", params)
+            prompt, system = t.evaluate("input", params)
         assert ex.value.args[0] == expected_error
     else:
-        prompt, system = t.execute("input", params)
+        prompt, system = t.evaluate("input", params)
         assert prompt == expected_prompt
         assert system == expected_system
 
@@ -132,7 +132,7 @@ def test_templates_prompt_save(templates_path, args, expected_prompt, expected_e
         ),
         (
             "boo",
-            ["--system", "s"],
+            ["-s", "s"],
             None,
             None,
             "Error: Cannot use -t/--template and --system together",
@@ -174,6 +174,7 @@ def test_template_basic(
         assert mocked_openai.last_request.json() == {
             "model": expected_model,
             "messages": [{"role": "user", "content": expected_input}],
+            "stream": False,
         }
     else:
         assert result.exit_code == 1