Repository: tensorzero/tensorzero
Pull request #2022: Fetch large fixtures from s3 in ui tests
Author: login: Aaron1011
Created: 2025-05-02T18:43:14Z | Merged: 2025-05-09T16:39:12Z

Each of these fixture files has 10 million rows - this will help us to test that the ui remains responsive when querying from a large ClickHouse databsae

The code to generate these files will be in a separate PR

<!--
Thank you for contributing to TensorZero!

Before submitting your PR, make sure you've read **[Contributing to TensorZero](https://github.com/tensorzero/tensorzero/blob/main/CONTRIBUTING.md)**.

In particular, make sure you've run `pre-commit` and any tests relevant to your changes (including E2E tests for changes to the gateway).

By submitting this PR, unless otherwise specified, you agree to license your contributions under the **[Apache 2.0 license](https://github.com/tensorzero/tensorzero/blob/main/LICENSE)**.
-->

<!-- ELLIPSIS_HIDDEN -->


----

> [!IMPORTANT]
> Fetch large fixtures from S3 in UI tests to test UI responsiveness with large datasets, updating CI/CD and Docker configurations to support this.
> 
>   - **Behavior**:
>     - Fetch large fixture files from S3 in UI tests to simulate large ClickHouse database queries.
>     - Update `ui-tests` job in `.github/workflows/general.yml` to use `namespace-profile-tensorzero-2x8` for caching S3 fixtures.
>     - Add `download-fixtures.py` to handle downloading large fixtures from S3.
>   - **Docker and CI/CD**:
>     - Modify `Dockerfile` to include AWS CLI and `parquet-tools` for handling large fixture files.
>     - Update `docker-compose.yml` and `docker-compose.e2e.yml` to build fixtures service with S3 capabilities.
>     - Extend `load_fixtures.sh` to download and insert large fixture files into ClickHouse.
>   - **Fixture Management**:
>     - Add `check-fixtures.sh` to verify fixture counts and check for duplicates.
>     - Introduce `upload-fixtures.sh` for uploading new large fixtures to S3.
>     - Update `README.md` with instructions for managing large fixtures.
> 
> <sup>This description was created by </sup>[<img alt="Ellipsis" src="https://img.shields.io/badge/Ellipsis-blue?color=175173">](https://www.ellipsis.dev?ref=tensorzero%2Ftensorzero&utm_source=github&utm_medium=referral)<sup> for 37c4c6a174590dab929bb1605c374839d50b1d0f. You can [customize](https://app.ellipsis.dev/tensorzero/settings/summaries) this summary. It will automatically update as commits are pushed.</sup>


<!-- ELLIPSIS_HIDDEN -->


Diff:
diff --git a/.github/workflows/general.yml b/.github/workflows/general.yml
index 1d3dd7b8e25..37bb6ece395 100644
--- a/.github/workflows/general.yml
+++ b/.github/workflows/general.yml
@@ -436,7 +436,8 @@ jobs:
         run: cat e2e_logs.txt
 
   ui-tests:
-    runs-on: ubuntu-latest
+    # We run on Namespace so that we have cache volumes for our s3-fixtures directory
+    runs-on: namespace-profile-tensorzero-2x8
     permissions:
       # Permission to checkout the repository
       contents: read
@@ -469,6 +470,9 @@ jobs:
         uses: namespacelabs/nscloud-setup-buildx-action@v0
         continue-on-error: ${{ github.event.pull_request.head.repo.full_name != github.repository || github.actor == 'dependabot[bot]' }}
 
+      - name: Install uv
+        run: curl -LsSf https://astral.sh/uv/0.6.17/install.sh | sh
+
       - name: Install pnpm
         uses: pnpm/action-setup@v4
         with:
@@ -479,6 +483,12 @@ jobs:
         with:
           node-version: "22.9.0"
 
+      - name: Cache s3-fixtures
+        uses: namespacelabs/nscloud-cache-action@v1
+        with:
+          path: |
+            ./ui/fixtures/s3-fixtures
+
       - name: Install wasm-pack
         run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
 
@@ -508,6 +518,11 @@ jobs:
           TENSORZERO_GATEWAY_URL: http://localhost:3000
         run: pnpm test
 
+      - name: Print ui tests logs
+        if: always()
+        working-directory: ui
+        run: docker compose -f fixtures/docker-compose.yml logs -t
+
   ui-tests-e2e:
     runs-on: namespace-profile-tensorzero-8x16
     # We currently only run this job when we have secrets available, since we need to use an S3 object_store
@@ -532,6 +547,9 @@ jobs:
       # We allow the namespace builder setup to fail on Dependabot PRs and PRs from forks
       # (where the oidc token is not available)
 
+      - name: Install uv
+        run: curl -LsSf https://astral.sh/uv/0.6.17/install.sh | sh
+
       - name: Install pnpm
         uses: pnpm/action-setup@v4
         with:
@@ -546,6 +564,12 @@ jobs:
         working-directory: ui
         run: pnpm install --frozen-lockfile
 
+      - name: Cache s3-fixtures
+        uses: namespacelabs/nscloud-cache-action@v1
+        with:
+          path: |
+            ./ui/fixtures/s3-fixtures
+
       - name: Build ui container
         # Enable feedback in the UI for this job so the playwright tests can use it
         run: docker buildx build -f ui/Dockerfile . -t tensorzero/ui
diff --git a/.gitignore b/.gitignore
index 31f4a88bfe8..12020a373b6 100644
--- a/.gitignore
+++ b/.gitignore
@@ -42,3 +42,5 @@ node_modules/
 
 # Claude Code
 CLAUDE.md
+
+ui/fixtures/s3-fixtures
diff --git a/ci/check-docker-compose.sh b/ci/check-docker-compose.sh
index 31e6de74998..b2a274781a6 100755
--- a/ci/check-docker-compose.sh
+++ b/ci/check-docker-compose.sh
@@ -7,7 +7,7 @@ if [[ "$dir_path" == */cursor ]]; then
 fi
 
 cd "$dir_path"
-docker compose -f "$1" up --wait --wait-timeout 30
+docker compose -f "$1" up --wait --wait-timeout 120
 status=$?
 if [ $status -ne 0 ]; then
   echo "Docker Compose failed for $1 with status $status"
diff --git a/tensorzero-internal/src/clickhouse/migration_manager/migrations/migration_0023.rs b/tensorzero-internal/src/clickhouse/migration_manager/migrations/migration_0023.rs
index 487288bc051..7b6e116dd90 100644
--- a/tensorzero-internal/src/clickhouse/migration_manager/migrations/migration_0023.rs
+++ b/tensorzero-internal/src/clickhouse/migration_manager/migrations/migration_0023.rs
@@ -5,11 +5,16 @@ use async_trait::async_trait;
 
 use super::check_table_exists;
 
+/// NOTE: This migration is BANNED in favor of migration_0028.
+///
 /// This migration adds a table StaticEvaluationHumanFeedback that stores human feedback in an easy-to-reference format.
 /// This is technically an auxiliary table as the primary store is still the various feedback tables.
 /// We also create two materialized views that automatically write to StaticEvaluationHumanFeedback when
 /// FloatMetricFeedback and BooleanMetricFeedback are updated with new feedback that contains both
 /// tensorzero::datapoint_id and tensorzero::human_feedback tags.
+///
+/// NOTE: The views in this table are StaticEvaluationHumanFeedbackFloatView and StaticEvaluationHumanFeedbackBooleanView.
+/// Thew views created by Migration 0028 are StaticEvaluationFloatHumanFeedbackView and StaticEvaluationBooleanHumanFeedbackView.
 pub struct Migration0023<'a> {
     pub clickhouse: &'a ClickHouseConnectionInfo,
 }
diff --git a/tensorzero-internal/src/clickhouse/migration_manager/migrations/migration_0027.rs b/tensorzero-internal/src/clickhouse/migration_manager/migrations/migration_0027.rs
new file mode 100644
index 00000000000..6a528217e9f
--- /dev/null
+++ b/tensorzero-internal/src/clickhouse/migration_manager/migrations/migration_0027.rs
@@ -0,0 +1,174 @@
+use async_trait::async_trait;
+
+use super::{check_index_exists, check_table_exists};
+use crate::clickhouse::migration_manager::migration_trait::Migration;
+use crate::clickhouse::ClickHouseConnectionInfo;
+use crate::error::{Error, ErrorDetails};
+
+/// This migration adds an index by `inference_id` to the `TagInference`,
+/// `ChatInference`, and `JsonInference` tables.
+/// This allows us to efficiently query for all inferences by id and/or tag values for a given key for a given inference.
+/// It also adds an index by `id` to the `ChatInferenceDatapoint` and `JsonInferenceDatapoint` tables.
+pub struct Migration0027<'a> {
+    pub clickhouse: &'a ClickHouseConnectionInfo,
+}
+
+#[async_trait]
+impl Migration for Migration0027<'_> {
+    async fn can_apply(&self) -> Result<(), Error> {
+        if !check_table_exists(self.clickhouse, "TagInference", "0027").await? {
+            return Err(ErrorDetails::ClickHouseMigration {
+                id: "0027".to_string(),
+                message: "TagInference table does not exist".to_string(),
+            }
+            .into());
+        }
+        if !check_table_exists(self.clickhouse, "ChatInference", "0027").await? {
+            return Err(ErrorDetails::ClickHouseMigration {
+                id: "0027".to_string(),
+                message: "ChatInference table does not exist".to_string(),
+            }
+            .into());
+        }
+        if !check_table_exists(self.clickhouse, "JsonInference", "0027").await? {
+            return Err(ErrorDetails::ClickHouseMigration {
+                id: "0027".to_string(),
+                message: "JsonInference table does not exist".to_string(),
+            }
+            .into());
+        }
+        if !check_table_exists(self.clickhouse, "ChatInferenceDatapoint", "0027").await? {
+            return Err(ErrorDetails::ClickHouseMigration {
+                id: "0027".to_string(),
+                message: "ChatInferenceDatapoint table does not exist".to_string(),
+            }
+            .into());
+        }
+        if !check_table_exists(self.clickhouse, "JsonInferenceDatapoint", "0027").await? {
+            return Err(ErrorDetails::ClickHouseMigration {
+                id: "0027".to_string(),
+                message: "JsonInferenceDatapoint table does not exist".to_string(),
+            }
+            .into());
+        }
+        Ok(())
+    }
+
+    async fn should_apply(&self) -> Result<bool, Error> {
+        let index_exists =
+            check_index_exists(self.clickhouse, "TagInference", "inference_id_index").await?;
+        let chat_index_exists =
+            check_index_exists(self.clickhouse, "ChatInference", "inference_id_index").await?;
+        let json_index_exists =
+            check_index_exists(self.clickhouse, "JsonInference", "inference_id_index").await?;
+        let chat_datapoint_index_exists =
+            check_index_exists(self.clickhouse, "ChatInferenceDatapoint", "id_index").await?;
+        let json_datapoint_index_exists =
+            check_index_exists(self.clickhouse, "JsonInferenceDatapoint", "id_index").await?;
+        Ok(!index_exists
+            || !chat_index_exists
+            || !json_index_exists
+            || !chat_datapoint_index_exists
+            || !json_datapoint_index_exists)
+    }
+
+    async fn apply(&self, _clean_start: bool) -> Result<(), Error> {
+        let create_index_query = r#"
+            ALTER TABLE TagInference ADD INDEX IF NOT EXISTS inference_id_index inference_id TYPE bloom_filter GRANULARITY 1;
+        "#;
+        let _ = self
+            .clickhouse
+            .run_query_synchronous(create_index_query.to_string(), None)
+            .await?;
+
+        let materialize_index_query = r#"
+            ALTER TABLE TagInference MATERIALIZE INDEX inference_id_index;
+        "#;
+        let _ = self
+            .clickhouse
+            .run_query_synchronous(materialize_index_query.to_string(), None)
+            .await?;
+
+        let create_index_query = r#"
+            ALTER TABLE ChatInference ADD INDEX IF NOT EXISTS inference_id_index id TYPE bloom_filter GRANULARITY 1;
+        "#;
+        let _ = self
+            .clickhouse
+            .run_query_synchronous(create_index_query.to_string(), None)
+            .await?;
+
+        let materialize_index_query = r#"
+            ALTER TABLE ChatInference MATERIALIZE INDEX inference_id_index;
+        "#;
+        let _ = self
+            .clickhouse
+            .run_query_synchronous(materialize_index_query.to_string(), None)
+            .await?;
+
+        let create_index_query = r#"
+            ALTER TABLE JsonInference ADD INDEX IF NOT EXISTS inference_id_index id TYPE bloom_filter GRANULARITY 1;
+        "#;
+        let _ = self
+            .clickhouse
+            .run_query_synchronous(create_index_query.to_string(), None)
+            .await?;
+
+        let materialize_index_query = r#"
+            ALTER TABLE JsonInference MATERIALIZE INDEX inference_id_index;
+        "#;
+        let _ = self
+            .clickhouse
+            .run_query_synchronous(materialize_index_query.to_string(), None)
+            .await?;
+
+        let create_index_query = r#"
+            ALTER TABLE ChatInferenceDatapoint ADD INDEX IF NOT EXISTS id_index id TYPE bloom_filter GRANULARITY 1;
+        "#;
+        let _ = self
+            .clickhouse
+            .run_query_synchronous(create_index_query.to_string(), None)
+            .await?;
+
+        let materialize_index_query = r#"
+            ALTER TABLE ChatInferenceDatapoint MATERIALIZE INDEX id_index;
+        "#;
+        let _ = self
+            .clickhouse
+            .run_query_synchronous(materialize_index_query.to_string(), None)
+            .await?;
+
+        let create_index_query = r#"
+            ALTER TABLE JsonInferenceDatapoint ADD INDEX IF NOT EXISTS id_index id TYPE bloom_filter GRANULARITY 1;
+        "#;
+        let _ = self
+            .clickhouse
+            .run_query_synchronous(create_index_query.to_string(), None)
+            .await?;
+
+        let materialize_index_query = r#"
+            ALTER TABLE JsonInferenceDatapoint MATERIALIZE INDEX id_index;
+        "#;
+        let _ = self
+            .clickhouse
+            .run_query_synchronous(materialize_index_query.to_string(), None)
+            .await?;
+
+        Ok(())
+    }
+
+    fn rollback_instructions(&self) -> String {
+        r#"
+        ALTER TABLE TagInference DROP INDEX IF EXISTS inference_id_index;
+        ALTER TABLE ChatInference DROP INDEX IF EXISTS inference_id_index;
+        ALTER TABLE JsonInference DROP INDEX IF EXISTS inference_id_index;
+        ALTER TABLE ChatInferenceDatapoint DROP INDEX IF EXISTS id_index;
+        ALTER TABLE JsonInferenceDatapoint DROP INDEX IF EXISTS id_index;
+        "#
+        .to_string()
+    }
+
+    async fn has_succeeded(&self) -> Result<bool, Error> {
+        let should_apply = self.should_apply().await?;
+        Ok(!should_apply)
+    }
+}
diff --git a/tensorzero-internal/src/clickhouse/migration_manager/migrations/migration_0028.rs b/tensorzero-internal/src/clickhouse/migration_manager/migrations/migration_0028.rs
new file mode 100644
index 00000000000..8fe100746b3
--- /dev/null
+++ b/tensorzero-internal/src/clickhouse/migration_manager/migrations/migration_0028.rs
@@ -0,0 +1,391 @@
+use std::time::Duration;
+
+use crate::clickhouse::migration_manager::migration_trait::Migration;
+use crate::clickhouse::ClickHouseConnectionInfo;
+use crate::error::{Error, ErrorDetails};
+use async_trait::async_trait;
+
+use super::check_table_exists;
+
+/// NOTE: This migration supersedes migration_0023.
+/// Migration 0023 was inefficient due to its use of joins.
+/// In this migration, we create the same table StaticEvaluationHumanFeedback if it doesn't exist.
+/// We then drop the old materialized view and create new ones that are more efficient.
+/// This migration adds a table StaticEvaluationHumanFeedback that stores human feedback in an easy-to-reference format.
+/// This is technically an auxiliary table as the primary store is still the various feedback tables.
+/// We also create two materialized views that automatically write to StaticEvaluationHumanFeedback when
+/// FloatMetricFeedback and BooleanMetricFeedback are updated with new feedback that contains both
+/// tensorzero::datapoint_id and tensorzero::human_feedback tags.
+///
+/// In particular, since we don't care if there are duplicate rows and we need the migration to work with concurrent gateways,
+/// we do a full cutover to the new view, meaning that if we determine that this migration needs to run and we're not clean starting,
+/// we:
+///   1. Fix a timestamp for the migration
+///   2. Drop the old views if they exist
+///   3. Create the new views with a timestamp of the migration timestamp + 10 seconds
+///   4. Wait 10 seconds
+///   5. Insert the data from the missing 10 seconds into the new table.
+///
+///
+/// NOTE: The views created by this migration are StaticEvaluationFloatHumanFeedbackView and StaticEvaluationBooleanHumanFeedbackView.
+/// The views created by Migration 0023 are StaticEvaluationHumanFeedbackFloatView and StaticEvaluationHumanFeedbackBooleanView.s
+///
+/// NOTE: The two views created by this migration are required to be separate as ClickHouse only triggers materialized views
+/// on the first table to appear.
+pub struct Migration0028<'a> {
+    pub clickhouse: &'a ClickHouseConnectionInfo,
+}
+
+const MIGRATION_ID: &str = "0028";
+
+#[async_trait]
+impl Migration for Migration0028<'_> {
+    async fn can_apply(&self) -> Result<(), Error> {
+        // As long as FloatMetricFeedback and BooleanMetricFeedback exist, we can apply this migration
+        if !check_table_exists(self.clickhouse, "FloatMetricFeedback", MIGRATION_ID).await? {
+            return Err(Error::new(ErrorDetails::ClickHouseMigration {
+                id: MIGRATION_ID.to_string(),
+                message: "FloatMetricFeedback table does not exist".to_string(),
+            }));
+        }
+        if !check_table_exists(self.clickhouse, "BooleanMetricFeedback", MIGRATION_ID).await? {
+            return Err(Error::new(ErrorDetails::ClickHouseMigration {
+                id: MIGRATION_ID.to_string(),
+                message: "BooleanMetricFeedback table does not exist".to_string(),
+            }));
+        }
+        Ok(())
+    }
+
+    async fn should_apply(&self) -> Result<bool, Error> {
+        let human_feedback_table_exists = check_table_exists(
+            self.clickhouse,
+            "StaticEvaluationHumanFeedback",
+            MIGRATION_ID,
+        )
+        .await?;
+        let float_materialized_view_exists = check_table_exists(
+            self.clickhouse,
+            "StaticEvaluationFloatHumanFeedbackView",
+            MIGRATION_ID,
+        )
+        .await?;
+        let boolean_materialized_view_exists = check_table_exists(
+            self.clickhouse,
+            "StaticEvaluationBooleanHumanFeedbackView",
+            MIGRATION_ID,
+        )
+        .await?;
+        Ok(!human_feedback_table_exists
+            || !float_materialized_view_exists
+            || !boolean_materialized_view_exists)
+    }
+
+    async fn apply(&self, clean_start: bool) -> Result<(), Error> {
+        let view_offset = Duration::from_secs(10);
+        let current_time = std::time::SystemTime::now();
+        let view_timestamp = (current_time
+            .duration_since(std::time::UNIX_EPOCH)
+            .map_err(|e| {
+                Error::new(ErrorDetails::ClickHouseMigration {
+                    id: MIGRATION_ID.to_string(),
+                    message: e.to_string(),
+                })
+            })?
+            + view_offset)
+            .as_secs();
+        let view_timestamp_where_clause = if !clean_start {
+            format!("AND UUIDv7ToDateTime(feedback_id) >= toDateTime(toUnixTimestamp({view_timestamp}))")
+        } else {
+            String::new()
+        };
+        self.clickhouse
+            .run_query_synchronous(
+                r#"CREATE TABLE IF NOT EXISTS StaticEvaluationHumanFeedback (
+                    metric_name LowCardinality(String),
+                    datapoint_id UUID,
+                    output String,
+                    value String,  -- JSON encoded value of the feedback
+                    feedback_id UUID,
+                    evaluator_inference_id UUID,
+                    timestamp DateTime MATERIALIZED UUIDv7ToDateTime(feedback_id)
+                ) ENGINE = MergeTree()
+                ORDER BY (metric_name, datapoint_id, output)
+                SETTINGS index_granularity = 256 -- We use a small index granularity to improve lookup performance
+            "#.to_string(),
+                None,
+            )
+            .await?;
+
+        // Since there cannot have been any StaticEvaluationHumanFeedback rows before this migration runs,
+        // we can just create the materialized views in place.
+
+        // Create the materialized view for FloatMetricFeedback
+        let query = format!(
+            r#"
+            CREATE MATERIALIZED VIEW IF NOT EXISTS StaticEvaluationFloatHumanFeedbackView
+            TO StaticEvaluationHumanFeedback
+            AS
+                WITH float_human_feedback AS (
+                    SELECT
+                        metric_name,
+                        toUUID(tags['tensorzero::datapoint_id']) AS datapoint_id,
+                        toJSONString(value) AS value,
+                        id AS feedback_id,
+                        target_id,
+                        toUUID(tags['tensorzero::evaluator_inference_id']) AS evaluator_inference_id
+                        -- we enforce in the feedback endpoint that this is present
+                        -- if the tensorzero::datapoint_id is present and the tensorzero::human_feedback
+                        -- tag is present.
+                    FROM FloatMetricFeedback
+                    WHERE
+                        mapContains(tags, 'tensorzero::human_feedback')
+                        AND mapContains(tags, 'tensorzero::datapoint_id')
+                        {view_timestamp_where_clause}
+
+                ),
+                inference_by_id AS (
+                    SELECT id_uint, function_name, function_type, variant_name, episode_id FROM InferenceById
+                    WHERE id_uint IN (
+                        SELECT toUInt128(target_id) FROM float_human_feedback
+                    )
+                ),
+                chat_inference AS (
+                    SELECT function_name, variant_name, episode_id, id, output FROM ChatInference
+                    WHERE (function_name, variant_name, episode_id) IN (
+                        SELECT function_name, variant_name, episode_id 
+                        FROM inference_by_id
+                    )
+                ),
+                json_inference AS (
+                    SELECT function_name, variant_name, episode_id, id, output FROM JsonInference
+                    WHERE (function_name, variant_name, episode_id) IN (
+                        SELECT function_name, variant_name, episode_id 
+                        FROM inference_by_id
+                    )
+                )
+            SELECT
+                f.metric_name as metric_name,
+                f.datapoint_id as datapoint_id,
+                -- Select output based on function_type determined via InferenceById join
+                if(i.function_type = 'chat', ci.output, ji.output) AS output,
+                f.value AS value,
+                f.feedback_id AS feedback_id,
+                f.evaluator_inference_id AS evaluator_inference_id
+            FROM float_human_feedback f
+            INNER JOIN inference_by_id i ON
+                toUInt128(f.target_id) = i.id_uint
+            LEFT JOIN chat_inference ci ON
+                i.function_type = 'chat' AND
+                i.function_name = ci.function_name AND
+                i.variant_name = ci.variant_name AND
+                i.episode_id = ci.episode_id AND
+                f.target_id = ci.id
+            LEFT JOIN json_inference ji ON
+                i.function_type = 'json' AND
+                i.function_name = ji.function_name AND
+                i.variant_name = ji.variant_name AND
+                i.episode_id = ji.episode_id AND
+                f.target_id = ji.id;
+        "#
+        );
+        self.clickhouse
+            .run_query_synchronous(query.to_string(), None)
+            .await?;
+
+        // Create the materialized view for BooleanMetricFeedback
+        let query = format!(
+            r#"
+            CREATE MATERIALIZED VIEW IF NOT EXISTS StaticEvaluationBooleanHumanFeedbackView
+            TO StaticEvaluationHumanFeedback
+            AS
+                WITH boolean_human_feedback AS (
+                    SELECT
+                       metric_name,
+                       toUUID(tags['tensorzero::datapoint_id']) AS datapoint_id,
+                       toJSONString(value) AS value,
+                       id AS feedback_id,
+                       target_id,
+                       toUUID(tags['tensorzero::evaluator_inference_id']) AS evaluator_inference_id
+                       -- we enforce in the feedback endpoint that this is present
+                       -- if the tensorzero::datapoint_id is present and the tensorzero::human_feedback
+                       -- tag is present.
+                   FROM BooleanMetricFeedback
+                   WHERE
+                       mapContains(tags, 'tensorzero::human_feedback')
+                       AND mapContains(tags, 'tensorzero::datapoint_id')
+                       {view_timestamp_where_clause}
+
+                ),
+                inference_by_id AS (
+                    SELECT id_uint, function_name, function_type, variant_name, episode_id FROM InferenceById
+                    WHERE id_uint IN (
+                        SELECT toUInt128(target_id) FROM boolean_human_feedback
+                    )
+                ),
+                chat_inference AS (
+                    SELECT function_name, variant_name, episode_id, id, output FROM ChatInference
+                    WHERE (function_name, variant_name, episode_id) IN (
+                        SELECT function_name, variant_name, episode_id 
+                        FROM inference_by_id
+                    )
+                ),
+                json_inference AS (
+                    SELECT function_name, variant_name, episode_id, id, output FROM JsonInference
+                    WHERE (function_name, variant_name, episode_id) IN (
+                        SELECT function_name, variant_name, episode_id 
+                        FROM inference_by_id
+                    )
+                )
+            SELECT
+                f.metric_name as metric_name,
+                f.datapoint_id as datapoint_id,
+                -- Select output based on function_type determined via InferenceById join
+                if(i.function_type = 'chat', ci.output, ji.output) AS output,
+                f.value AS value,
+                f.feedback_id AS feedback_id,
+                f.evaluator_inference_id AS evaluator_inference_id
+            FROM boolean_human_feedback f
+            INNER JOIN inference_by_id i ON
+                toUInt128(f.target_id) = i.id_uint
+            LEFT JOIN chat_inference ci ON
+                i.function_type = 'chat' AND
+                i.function_name = ci.function_name AND
+                i.variant_name = ci.variant_name AND
+                i.episode_id = ci.episode_id AND
+                f.target_id = ci.id
+            LEFT JOIN json_inference ji ON
+                i.function_type = 'json' AND
+                i.function_name = ji.function_name AND
+                i.variant_name = ji.variant_name AND
+                i.episode_id = ji.episode_id AND
+                f.target_id = ji.id;
+        "#
+        );
+
+        self.clickhouse
+            .run_query_synchronous(query.to_string(), None)
+            .await?;
+
+        if !clean_start {
+            // Sleep for the duration specified by view_offset to allow the materialized views to catch up
+            tokio::time::sleep(view_offset).await;
+            let current_timestamp = current_time
+                .duration_since(std::time::UNIX_EPOCH)
+                .map_err(|e| {
+                    Error::new(ErrorDetails::ClickHouseMigration {
+                        id: MIGRATION_ID.to_string(),
+                        message: e.to_string(),
+                    })
+                })?
+                .as_secs();
+            let insert_timestamp_where_clause = format!(
+                r#"
+                AND UUIDv7ToDateTime(feedback_id) < toDateTime(toUnixTimestamp({view_timestamp}))
+                AND UUIDv7ToDateTime(feedback_id) >= toDateTime(toUnixTimestamp({current_timestamp}))"#
+            );
+
+            let query = format!(
+                r#"
+                WITH human_feedback AS (
+                    SELECT
+                        metric_name,
+                        toUUID(tags['tensorzero::datapoint_id']) AS datapoint_id,
+                        toJSONString(value) AS value,
+                        id AS feedback_id,
+                        target_id,
+                        toUUID(tags['tensorzero::evaluator_inference_id']) AS evaluator_inference_id
+                        -- we enforce in the feedback endpoint that this is present
+                        -- if the tensorzero::datapoint_id is present and the tensorzero::human_feedback
+                        -- tag is present.
+                    FROM FloatMetricFeedback
+                    WHERE
+                        mapContains(tags, 'tensorzero::human_feedback')
+                        AND mapContains(tags, 'tensorzero::datapoint_id')
+                        {insert_timestamp_where_clause}
+                    UNION ALL
+                    SELECT
+                       metric_name,
+                       toUUID(tags['tensorzero::datapoint_id']) AS datapoint_id,
+                       toJSONString(value) AS value,
+                       id AS feedback_id,
+                       target_id,
+                       toUUID(tags['tensorzero::evaluator_inference_id']) AS evaluator_inference_id
+                       -- we enforce in the feedback endpoint that this is present
+                       -- if the tensorzero::datapoint_id is present and the tensorzero::human_feedback
+                       -- tag is present.
+                    FROM BooleanMetricFeedback
+                    WHERE
+                       mapContains(tags, 'tensorzero::human_feedback')
+                       AND mapContains(tags, 'tensorzero::datapoint_id')
+                       {insert_timestamp_where_clause}
+
+                ),
+                inference_by_id AS (
+                    SELECT id_uint, function_name, function_type, variant_name, episode_id FROM InferenceById
+                    WHERE id_uint IN (
+                        SELECT toUInt128(target_id) FROM human_feedback
+                    )
+                ),
+                chat_inference AS (
+                    SELECT function_name, variant_name, episode_id, id, output FROM ChatInference
+                    WHERE (function_name, variant_name, episode_id) IN (
+                        SELECT function_name, variant_name, episode_id 
+                        FROM inference_by_id
+                    )
+                ),
+                json_inference AS (
+                    SELECT function_name, variant_name, episode_id, id, output FROM JsonInference
+                    WHERE (function_name, variant_name, episode_id) IN (
+                        SELECT function_name, variant_name, episode_id 
+                        FROM inference_by_id
+                    )
+                )
+            INSERT INTO StaticEvaluationHumanFeedback
+            SELECT
+                f.metric_name as metric_name,
+                f.datapoint_id as datapoint_id,
+                -- Select output based on function_type determined via InferenceById join
+                if(i.function_type = 'chat', ci.output, ji.output) AS output,
+                f.value AS value,
+                f.feedback_id AS feedback_id,
+                f.evaluator_inference_id AS evaluator_inference_id
+            FROM human_feedback f
+            INNER JOIN inference_by_id i ON
+                toUInt128(f.target_id) = i.id_uint
+            LEFT JOIN chat_inference ci ON
+                i.function_type = 'chat' AND
+                i.function_name = ci.function_name AND
+                i.variant_name = ci.variant_name AND
+                i.episode_id = ci.episode_id AND
+                f.target_id = ci.id
+            LEFT JOIN json_inference ji ON
+                i.function_type = 'json' AND
+                i.function_name = ji.function_name AND
+                i.variant_name = ji.variant_name AND
+                i.episode_id = ji.episode_id AND
+                f.target_id = ji.id;
+        "#
+            );
+            self.clickhouse
+                .run_query_synchronous(query.to_string(), None)
+                .await?;
+        }
+
+        Ok(())
+    }
+
+    fn rollback_instructions(&self) -> String {
+        r#"
+        DROP MATERIALIZED VIEW IF EXISTS StaticEvaluationFloatHumanFeedbackView;
+        DROP MATERIALIZED VIEW IF EXISTS StaticEvaluationBooleanHumanFeedbackView;
+        DROP TABLE IF EXISTS StaticEvaluationHumanFeedback;
+        "#
+        .to_string()
+    }
+
+    async fn has_succeeded(&self) -> Result<bool, Error> {
+        let should_apply = self.should_apply().await?;
+        Ok(!should_apply)
+    }
+}
diff --git a/tensorzero-internal/src/clickhouse/migration_manager/migrations/mod.rs b/tensorzero-internal/src/clickhouse/migration_manager/migrations/mod.rs
index 2bad9f94254..83ff88bbe8f 100644
--- a/tensorzero-internal/src/clickhouse/migration_manager/migrations/mod.rs
+++ b/tensorzero-internal/src/clickhouse/migration_manager/migrations/mod.rs
@@ -21,10 +21,11 @@ pub mod migration_0019;
 pub mod migration_0020;
 pub mod migration_0021;
 pub mod migration_0022;
-pub mod migration_0023;
 pub mod migration_0024;
 pub mod migration_0025;
 pub mod migration_0026;
+pub mod migration_0027;
+pub mod migration_0028;
 
 /// Returns true if the table exists, false if it does not
 /// Errors if the query fails
@@ -164,3 +165,13 @@ async fn get_table_engine(
     let result = clickhouse.run_query_synchronous(query, None).await?;
     Ok(result.trim().to_string())
 }
+
+async fn check_index_exists(
+    clickhouse: &ClickHouseConnectionInfo,
+    table: &str,
+    index: &str,
+) -> Result<bool, Error> {
+    let query = format!("SELECT 1 FROM system.data_skipping_indices WHERE database='{}' AND table='{}' AND name='{}'", clickhouse.database(), table, index);
+    let result = clickhouse.run_query_synchronous(query, None).await?;
+    Ok(result.trim() == "1")
+}
diff --git a/tensorzero-internal/src/clickhouse/migration_manager/mod.rs b/tensorzero-internal/src/clickhouse/migration_manager/mod.rs
index 7ecc10e72bc..3acf5be5194 100644
--- a/tensorzero-internal/src/clickhouse/migration_manager/mod.rs
+++ b/tensorzero-internal/src/clickhouse/migration_manager/mod.rs
@@ -22,10 +22,11 @@ use migrations::migration_0019::Migration0019;
 use migrations::migration_0020::Migration0020;
 use migrations::migration_0021::Migration0021;
 use migrations::migration_0022::Migration0022;
-use migrations::migration_0023::Migration0023;
 use migrations::migration_0024::Migration0024;
 use migrations::migration_0025::Migration0025;
 use migrations::migration_0026::Migration0026;
+use migrations::migration_0027::Migration0027;
+use migrations::migration_0028::Migration0028;
 
 /// Constructs (but does not run) a vector of all our database migrations.
 /// This is the single source of truth for all migration - it's used during startup to migrate
@@ -64,10 +65,11 @@ pub fn make_all_migrations(
         Box::new(Migration0020 { clickhouse }),
         Box::new(Migration0021 { clickhouse }),
         Box::new(Migration0022 { clickhouse }),
-        Box::new(Migration0023 { clickhouse }),
         Box::new(Migration0024 { clickhouse }),
         Box::new(Migration0025 { clickhouse }),
         Box::new(Migration0026 { clickhouse }),
+        Box::new(Migration0027 { clickhouse }),
+        Box::new(Migration0028 { clickhouse }),
     ]
 }
 
diff --git a/tensorzero-internal/src/clickhouse/mod.rs b/tensorzero-internal/src/clickhouse/mod.rs
index 50fe88bb370..8470af43037 100644
--- a/tensorzero-internal/src/clickhouse/mod.rs
+++ b/tensorzero-internal/src/clickhouse/mod.rs
@@ -14,6 +14,7 @@ pub mod migration_manager;
 #[cfg(any(test, feature = "e2e_tests"))]
 pub mod test_helpers;
 
+use crate::error::DisplayOrDebugGateway;
 use crate::error::{Error, ErrorDetails};
 
 #[derive(Debug, Clone)]
@@ -233,14 +234,14 @@ impl ClickHouseConnectionInfo {
                     .await
                     .map_err(|e| {
                         Error::new(ErrorDetails::ClickHouseQuery {
-                            message: e.to_string(),
+                            message: DisplayOrDebugGateway::new(e).to_string(),
                         })
                     })?;
                 let status = response.status();
 
                 let response_body = response.text().await.map_err(|e| {
                     Error::new(ErrorDetails::ClickHouseQuery {
-                        message: e.to_string(),
+                        message: DisplayOrDebugGateway::new(e).to_string(),
                     })
                 })?;
 
diff --git a/tensorzero-internal/tests/e2e/clickhouse.rs b/tensorzero-internal/tests/e2e/clickhouse.rs
index cd4eda09946..0052854a3dc 100644
--- a/tensorzero-internal/tests/e2e/clickhouse.rs
+++ b/tensorzero-internal/tests/e2e/clickhouse.rs
@@ -19,6 +19,7 @@ use tensorzero_internal::clickhouse::migration_manager::migrations::migration_00
 use tensorzero_internal::clickhouse::migration_manager::migrations::migration_0009::Migration0009;
 use tensorzero_internal::clickhouse::migration_manager::migrations::migration_0011::Migration0011;
 use tensorzero_internal::clickhouse::migration_manager::migrations::migration_0013::Migration0013;
+
 use tensorzero_internal::clickhouse::migration_manager::{self, make_all_migrations};
 use tensorzero_internal::clickhouse::test_helpers::{get_clickhouse, CLICKHOUSE_URL};
 use tensorzero_internal::clickhouse::ClickHouseConnectionInfo;
@@ -150,7 +151,7 @@ async fn test_clickhouse_migration_manager() {
         // will throw an error if it doesn't.
         // This must be an array literal, so that the macro can generate a function
         // for each element in the array.
-        [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
+        [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]
     );
     run_all(&migrations).await;
 
diff --git a/ui/app/utils/clickhouse/datasets.test.ts b/ui/app/utils/clickhouse/datasets.test.ts
index 2064fd3d3d4..0f43d3b1e07 100644
--- a/ui/app/utils/clickhouse/datasets.test.ts
+++ b/ui/app/utils/clickhouse/datasets.test.ts
@@ -26,7 +26,7 @@ describe("countRowsForDataset", () => {
       output_source: "none",
     });
     const rows = await countRowsForDataset(dataset_params);
-    expect(rows).toBe(804);
+    expect(rows).toBe(803);
   });
 
   test("returns the correct number of rows for a specific variant", async () => {
diff --git a/ui/app/utils/clickhouse/evaluations.server.ts b/ui/app/utils/clickhouse/evaluations.server.ts
index 3ed1eef7040..2a2c40e012c 100644
--- a/ui/app/utils/clickhouse/evaluations.server.ts
+++ b/ui/app/utils/clickhouse/evaluations.server.ts
@@ -34,7 +34,7 @@ export async function getEvaluationRunInfos(
         '%Y-%m-%dT%H:%i:%SZ'
       ) as most_recent_inference_date
     FROM
-      TagInference AS run_tag
+      TagInference AS run_tag FINAL
     WHERE
       run_tag.key = 'tensorzero::evaluation_run_id'
       AND run_tag.value IN ({evaluation_run_ids:Array(String)})
@@ -66,22 +66,22 @@ export async function getEvaluationRunInfosForDatapoint(
   const function_type = config.functions[function_name].type;
   const inference_table_name =
     function_type === "chat" ? "ChatInference" : "JsonInference";
+
   const query = `
-    SELECT any(i.tags['tensorzero::evaluation_run_id']) as evaluation_run_id, any(i.variant_name) as variant_name,
-      formatDateTime(
-        max(UUIDv7ToDateTime(inference_id)),
-        '%Y-%m-%dT%H:%i:%SZ'
-      ) as most_recent_inference_date
-    FROM TagInference t
-    INNER JOIN {inference_table_name:Identifier} i
-      ON t.inference_id = i.id
-      AND i.function_name = {function_name:String}
-      AND i.variant_name = t.variant_name
-      AND i.episode_id = t.episode_id
-    WHERE t.key = 'tensorzero::datapoint_id'
-      AND t.value = {datapoint_id:String}
+    WITH datapoint_inference_ids AS (
+      SELECT inference_id FROM TagInference FINAL WHERE key = 'tensorzero::datapoint_id' AND value = {datapoint_id:String}
+    )
+    SELECT any(tags['tensorzero::evaluation_run_id']) as evaluation_run_id,
+           any(variant_name) as variant_name,
+           formatDateTime(
+             max(UUIDv7ToDateTime(id)),
+             '%Y-%m-%dT%H:%i:%SZ'
+           ) as most_recent_inference_date
+    FROM {inference_table_name:Identifier}
+      WHERE id IN (SELECT inference_id FROM datapoint_inference_ids)
+      AND function_name = {function_name:String}
     GROUP BY
-      i.tags['tensorzero::evaluation_run_id']
+      tags['tensorzero::evaluation_run_id']
   `;
   const result = await clickhouseClient.query({
     query,
@@ -157,21 +157,29 @@ async function parseEvaluationResultWithVariant(
   }
 }
 
-const getEvaluationResultDatapointIdsQuery = `
-  SELECT DISTINCT dp.id as dp_id
-  FROM {datapoint_table_name:Identifier} dp
-  INNER JOIN TagInference datapoint_tag
-    ON dp.id = toUUIDOrNull(datapoint_tag.value)
-    AND datapoint_tag.key = 'tensorzero::datapoint_id'
-    AND datapoint_tag.function_name = {function_name:String}
-  INNER JOIN {inference_table_name:Identifier} ci
-    ON ci.id = datapoint_tag.inference_id
-    AND ci.function_name = {function_name:String}
-  WHERE dp.function_name = {function_name:String}
-    AND (ci.tags['tensorzero::evaluation_run_id'] IS NULL OR
-         ci.tags['tensorzero::evaluation_run_id'] IN ({evaluation_run_ids:Array(String)}))
-  ORDER BY toUInt128(toUUID(dp.id)) DESC
+const getEvaluationResultDatapointIdQuery = (
+  limit?: number,
+  offset?: number,
+) => {
+  return `
+    all_inference_ids AS (
+      SELECT DISTINCT inference_id
+      FROM TagInference FINAL WHERE key = 'tensorzero::evaluation_run_id'
+      AND function_name = {function_name:String}
+      AND value IN ({evaluation_run_ids:Array(String)})
+    ),
+    all_datapoint_ids AS (
+      SELECT DISTINCT value as datapoint_id
+      FROM TagInference FINAL
+      WHERE key = 'tensorzero::datapoint_id'
+      AND function_name = {function_name:String}
+      AND inference_id IN (SELECT inference_id FROM all_inference_ids)
+      ORDER BY toUInt128(toUUID(datapoint_id)) DESC
+      ${limit ? `LIMIT ${limit}` : ""}
+      ${offset ? `OFFSET ${offset}` : ""}
+    )
 `;
+};
 
 export async function getEvaluationResults(
   function_name: string,
@@ -188,11 +196,45 @@ export async function getEvaluationResults(
   const inference_table_name =
     function_type === "chat" ? "ChatInference" : "JsonInference";
   const query = `
+  WITH ${getEvaluationResultDatapointIdQuery(limit, offset)},
+    filtered_dp AS (
+      SELECT * FROM {datapoint_table_name:Identifier} FINAL
+      WHERE function_name = {function_name:String}
+      -- We don't have the dataset_name here but there is an index on datapoint_ids so this should not be a full scan
+      AND id IN (SELECT datapoint_id FROM all_datapoint_ids)
+    ),
+    filtered_inference AS (
+      SELECT * FROM {inference_table_name:Identifier}
+      WHERE id IN (SELECT inference_id FROM all_inference_ids)
+      AND function_name = {function_name:String}
+    ),
+    filtered_feedback AS (
+      SELECT metric_name,
+             argMax(toString(value), timestamp) as value,
+             argMax(tags['tensorzero::evaluator_inference_id'], timestamp) as evaluator_inference_id,
+             argMax(id, timestamp) as feedback_id,
+             target_id
+      FROM BooleanMetricFeedback
+      WHERE metric_name IN ({metric_names:Array(String)})
+      AND target_id IN (SELECT inference_id FROM all_inference_ids)
+      GROUP BY target_id, metric_name -- for the argMax
+      UNION ALL
+      SELECT metric_name,
+             argMax(toString(value), timestamp) as value,
+             argMax(tags['tensorzero::evaluator_inference_id'], timestamp) as evaluator_inference_id,
+             argMax(id, timestamp) as feedback_id,
+             target_id
+      FROM FloatMetricFeedback
+      WHERE metric_name IN ({metric_names:Array(String)})
+      AND target_id IN (SELECT inference_id FROM all_inference_ids)
+      GROUP BY target_id, metric_name -- for the argMax
+    )
   SELECT
     dp.input as input,
     dp.id as datapoint_id,
     dp.output as reference_output,
     ci.output as generated_output,
+    ci.function_name as function_name,
     ci.tags['tensorzero::evaluation_run_id'] as evaluation_run_id,
     ci.tags['tensorzero::dataset_name'] as dataset_name,
     if(length(feedback.evaluator_inference_id) > 0, feedback.evaluator_inference_id, null) as evaluator_inference_id,
@@ -200,45 +242,13 @@ export async function getEvaluationResults(
     feedback.metric_name as metric_name,
     feedback.value as metric_value,
     feedback.feedback_id as feedback_id
-  FROM (
-    ${getEvaluationResultDatapointIdsQuery}
-    LIMIT {limit:UInt32}
-    OFFSET {offset:UInt32}
-  ) paginated_dp
-  INNER JOIN {datapoint_table_name:Identifier} dp
-    ON dp.id = paginated_dp.dp_id
-  INNER JOIN TagInference datapoint_tag FINAL
-    ON dp.id = toUUIDOrNull(datapoint_tag.value)
-    AND datapoint_tag.key = 'tensorzero::datapoint_id'
-    AND datapoint_tag.function_name = {function_name:String}
-  INNER JOIN {inference_table_name:Identifier} ci
-    ON datapoint_tag.inference_id = ci.id
-    AND ci.function_name = {function_name:String}
-    AND ci.variant_name = datapoint_tag.variant_name
-    AND ci.episode_id = datapoint_tag.episode_id
-  INNER JOIN (
-    SELECT target_id,
-           metric_name,
-           argMax(toString(value), timestamp) as value,
-           argMax(tags['tensorzero::evaluator_inference_id'], timestamp) as evaluator_inference_id,
-           argMax(id, timestamp) as feedback_id
-    FROM BooleanMetricFeedback
-    WHERE metric_name IN ({metric_names:Array(String)})
-    GROUP BY target_id, metric_name -- for the argMax
-    UNION ALL
-    SELECT target_id,
-           metric_name,
-           argMax(toString(value), timestamp) as value,
-           argMax(tags['tensorzero::evaluator_inference_id'], timestamp) as evaluator_inference_id,
-           argMax(id, timestamp) as feedback_id
-    FROM FloatMetricFeedback
-    WHERE metric_name IN ({metric_names:Array(String)})
-    GROUP BY target_id, metric_name -- for the argMax
-  ) feedback
+  FROM filtered_dp dp
+  INNER JOIN filtered_inference ci
+    ON toUUIDOrNull(ci.tags['tensorzero::datapoint_id']) = dp.id
+  INNER JOIN filtered_feedback feedback
     ON feedback.target_id = ci.id
-  WHERE
-    (ci.tags['tensorzero::evaluation_run_id'] IS NULL OR
-     ci.tags['tensorzero::evaluation_run_id'] IN ({evaluation_run_ids:Array(String)}))
+  ORDER BY toUInt128(datapoint_id) DESC
+
   `;
 
   const result = await clickhouseClient.query({
@@ -276,48 +286,48 @@ export async function getEvaluationStatistics(
   const inference_table_name =
     function_type === "chat" ? "ChatInference" : "JsonInference";
   const query = `
+  WITH ${getEvaluationResultDatapointIdQuery()},
+    filtered_inference AS (
+      SELECT * FROM {inference_table_name:Identifier}
+      WHERE id IN (SELECT inference_id FROM all_inference_ids)
+      AND function_name = {function_name:String}
+    ),
+   filtered_feedback AS (
+      SELECT metric_name,
+             argMax(value, timestamp) as value,
+             argMax(tags['tensorzero::evaluator_inference_id'], timestamp) as evaluator_inference_id,
+             argMax(id, timestamp) as feedback_id,
+             target_id
+      FROM BooleanMetricFeedback
+      WHERE metric_name IN ({metric_names:Array(String)})
+      AND target_id IN (SELECT inference_id FROM all_inference_ids)
+      GROUP BY target_id, metric_name -- for the argMax
+      UNION ALL
+      SELECT metric_name,
+             argMax(value, timestamp) as value,
+             argMax(tags['tensorzero::evaluator_inference_id'], timestamp) as evaluator_inference_id,
+             argMax(id, timestamp) as feedback_id,
+             target_id
+      FROM FloatMetricFeedback
+      WHERE metric_name IN ({metric_names:Array(String)})
+      AND target_id IN (SELECT inference_id FROM all_inference_ids)
+      GROUP BY target_id, metric_name -- for the argMax
+    )
   SELECT
-    ci.tags['tensorzero::evaluation_run_id'] AS evaluation_run_id,
-    feedback.metric_name AS metric_name,
+    filtered_inference.tags['tensorzero::evaluation_run_id'] AS evaluation_run_id,
+    filtered_feedback.metric_name AS metric_name,
     toUInt32(count()) AS datapoint_count,
-    avg(toFloat64(feedback.value)) AS mean_metric,
-    stddevSamp(toFloat64(feedback.value)) / sqrt(count()) AS stderr_metric
-  FROM (
-    ${getEvaluationResultDatapointIdsQuery}
-  ) dp
-  INNER JOIN TagInference datapoint_tag FINAL
-    ON dp_id = toUUIDOrNull(datapoint_tag.value)
-    AND datapoint_tag.key = 'tensorzero::datapoint_id'
-    AND datapoint_tag.function_name = {function_name:String}
-  INNER JOIN {inference_table_name:Identifier} ci
-    ON datapoint_tag.inference_id = ci.id
-    AND ci.function_name = {function_name:String}
-    AND ci.variant_name = datapoint_tag.variant_name
-    AND ci.episode_id = datapoint_tag.episode_id
-  INNER JOIN (
-    SELECT target_id,
-           metric_name,
-           argMax(value, timestamp) as value
-    FROM BooleanMetricFeedback
-    WHERE metric_name IN ({metric_names:Array(String)})
-    GROUP BY target_id, metric_name -- for the argMax
-    UNION ALL
-    SELECT target_id,
-           metric_name,
-           argMax(value, timestamp) as value
-    FROM FloatMetricFeedback
-    WHERE metric_name IN ({metric_names:Array(String)})
-    GROUP BY target_id, metric_name -- for the argMax
-  ) feedback
-    ON feedback.target_id = ci.id
-  WHERE
-     ci.tags['tensorzero::evaluation_run_id'] IN ({evaluation_run_ids:Array(String)})
-    AND feedback.value IS NOT NULL
+    avg(toFloat64(filtered_feedback.value)) AS mean_metric,
+    stddevSamp(toFloat64(filtered_feedback.value)) / sqrt(count()) AS stderr_metric
+  FROM filtered_inference
+  INNER JOIN filtered_feedback
+    ON filtered_feedback.target_id = filtered_inference.id
+    AND filtered_feedback.value IS NOT NULL
   GROUP BY
-    ci.tags['tensorzero::evaluation_run_id'],
-    feedback.metric_name
+    filtered_inference.tags['tensorzero::evaluation_run_id'],
+    filtered_feedback.metric_name
   ORDER BY
-    toUInt128(toUUID(ci.tags['tensorzero::evaluation_run_id'])) DESC
+    toUInt128(toUUID(filtered_inference.tags['tensorzero::evaluation_run_id'])) DESC
   `;
 
   const result = await clickhouseClient.query({
@@ -348,10 +358,9 @@ export async function countDatapointsForEvaluation(
     function_type === "chat" ? "ChatInference" : "JsonInference";
 
   const query = `
+      WITH ${getEvaluationResultDatapointIdQuery()}
       SELECT toUInt32(count()) as count
-      FROM (
-        ${getEvaluationResultDatapointIdsQuery}
-      )
+      FROM all_datapoint_ids
   `;
 
   const result = await clickhouseClient.query({
@@ -400,7 +409,7 @@ export async function getEvaluationRunInfo(
             any(function_name) AS inference_function_name,
             any(variant_name) AS variant_name,
             max(toUInt128(inference_id)) AS max_inference_id
-        FROM TagInference
+        FROM TagInference FINAL
         WHERE key IN ('tensorzero::evaluation_run_id', 'tensorzero::evaluation_name', 'tensorzero::dataset_name')
         GROUP BY inference_id
     )
@@ -438,7 +447,7 @@ export async function searchEvaluationRuns(
         AND value = {evaluation_name:String}
       )
     SELECT DISTINCT value as evaluation_run_id, variant_name
-    FROM TagInference
+    FROM TagInference FINAL
     WHERE key = 'tensorzero::evaluation_run_id'
       AND function_name = {function_name:String}
       AND inference_id IN (SELECT inference_id FROM evaluation_inference_ids)
@@ -487,52 +496,63 @@ export async function getEvaluationsForDatapoint(
     getEvaluatorMetricName(evaluation_name, evaluatorName),
   );
   const query = `
-  SELECT
-    inference.input as input,
-    dp.id as datapoint_id,
-    dp.output as reference_output,
-    inference.id as inference_id,
-    inference.output as generated_output,
-    inference.tags['tensorzero::evaluation_run_id'] as evaluation_run_id,
-    inference.variant_name as variant_name,
-    inference.tags['tensorzero::dataset_name'] as dataset_name,
-    if(length(feedback.evaluator_inference_id) > 0, feedback.evaluator_inference_id, null) as evaluator_inference_id,
-    feedback.metric_name as metric_name,
-    feedback.value as metric_value,
-    feedback.feedback_id as feedback_id
-  FROM TagInference datapoint_tag FINAL
-  JOIN {inference_table_name:Identifier} inference
-    ON datapoint_tag.inference_id = inference.id
-    AND inference.function_name = {function_name:String}
-    AND inference.variant_name = datapoint_tag.variant_name
-    AND inference.episode_id = datapoint_tag.episode_id
-  JOIN {datapoint_table_name:Identifier} dp
-    ON dp.id = toUUIDOrNull(datapoint_tag.value)
-    AND dp.function_name = {function_name:String}
-    AND dp.dataset_name = inference.tags['tensorzero::dataset_name']
-  LEFT JOIN (
-    SELECT target_id,
-           metric_name,
-           argMax(toString(value), timestamp) as value,
-           argMax(tags['tensorzero::evaluator_inference_id'], timestamp) as evaluator_inference_id,
-           argMax(id, timestamp) as feedback_id
-    FROM BooleanMetricFeedback
-    WHERE metric_name IN ({metric_names:Array(String)})
-    GROUP BY target_id, metric_name -- for the argMax
-    UNION ALL
-    SELECT target_id,
-           metric_name,
-           argMax(toString(value), timestamp) as value,
-           argMax(tags['tensorzero::evaluator_inference_id'], timestamp) as evaluator_inference_id,
-           argMax(id, timestamp) as feedback_id
-    FROM FloatMetricFeedback
-    WHERE metric_name IN ({metric_names:Array(String)})
-    GROUP BY target_id, metric_name -- for the argMax
-  ) feedback
-    ON feedback.target_id = inference.id
-  WHERE datapoint_tag.key = 'tensorzero::datapoint_id'
-    AND datapoint_tag.value = {datapoint_id:String}
-    AND inference.tags['tensorzero::evaluation_run_id'] IN ({evaluation_run_ids:Array(String)})
+   WITH all_inference_ids AS (
+      SELECT DISTINCT inference_id
+      FROM TagInference FINAL WHERE key = 'tensorzero::datapoint_id'
+      AND value = {datapoint_id:String}
+      AND function_name = {function_name:String}
+    ),
+    filtered_inference AS (
+      SELECT * FROM {inference_table_name:Identifier}
+      WHERE id IN (SELECT inference_id FROM all_inference_ids)
+      AND function_name = {function_name:String}
+      AND tags['tensorzero::evaluation_run_id'] IN ({evaluation_run_ids:Array(String)})
+    ),
+    filtered_feedback AS (
+      SELECT target_id,
+            metric_name,
+            argMax(toString(value), timestamp) as value,
+            argMax(tags['tensorzero::evaluator_inference_id'], timestamp) as evaluator_inference_id,
+            argMax(id, timestamp) as feedback_id
+      FROM BooleanMetricFeedback
+      WHERE metric_name IN ({metric_names:Array(String)})
+      AND target_id IN (SELECT inference_id FROM all_inference_ids)
+      GROUP BY target_id, metric_name -- for the argMax
+      UNION ALL
+      SELECT target_id,
+            metric_name,
+            argMax(toString(value), timestamp) as value,
+            argMax(tags['tensorzero::evaluator_inference_id'], timestamp) as evaluator_inference_id,
+            argMax(id, timestamp) as feedback_id
+      FROM FloatMetricFeedback
+      WHERE metric_name IN ({metric_names:Array(String)})
+      AND target_id IN (SELECT inference_id FROM all_inference_ids)
+      GROUP BY target_id, metric_name -- for the argMax
+    ),
+    filtered_datapoint AS (
+      SELECT * FROM {datapoint_table_name:Identifier}
+      WHERE id = {datapoint_id:UUID}
+      AND function_name = {function_name:String}
+    )
+    SELECT
+      filtered_inference.input as input,
+      filtered_inference.tags['tensorzero::datapoint_id'] as datapoint_id,
+      filtered_datapoint.output as reference_output,
+      filtered_inference.id as inference_id,
+      filtered_inference.output as generated_output,
+      filtered_inference.tags['tensorzero::evaluation_run_id'] as evaluation_run_id,
+      filtered_inference.variant_name as variant_name,
+      filtered_inference.tags['tensorzero::dataset_name'] as dataset_name,
+      if(length(filtered_feedback.evaluator_inference_id) > 0, filtered_feedback.evaluator_inference_id, null) as evaluator_inference_id,
+      filtered_feedback.metric_name as metric_name,
+      filtered_feedback.value as metric_value,
+      filtered_feedback.feedback_id as feedback_id
+    FROM filtered_inference
+    INNER JOIN filtered_datapoint
+      ON filtered_datapoint.id = toUUIDOrNull(filtered_inference.tags['tensorzero::datapoint_id'])
+    INNER JOIN filtered_feedback
+      ON filtered_feedback.target_id = filtered_inference.id
+
   `;
 
   const result = await clickhouseClient.query({
diff --git a/ui/app/utils/clickhouse/evaluations.test.ts b/ui/app/utils/clickhouse/evaluations.test.ts
index 84e1032f687..0ec09d8f438 100644
--- a/ui/app/utils/clickhouse/evaluations.test.ts
+++ b/ui/app/utils/clickhouse/evaluations.test.ts
@@ -2,12 +2,12 @@ import { describe, expect, test } from "vitest";
 import {
   countDatapointsForEvaluation,
   countTotalEvaluationRuns,
-  getEvaluationResults,
   getEvaluationRunInfo,
   getEvaluationRunInfos,
   getEvaluationRunInfosForDatapoint,
   getEvaluationsForDatapoint,
   getEvaluationStatistics,
+  getEvaluationResults,
   searchEvaluationRuns,
 } from "./evaluations.server";
 import type { ChatEvaluationResultWithVariant } from "./evaluations";
@@ -346,16 +346,16 @@ describe("getEvaluationStatistics", () => {
     expect(statistics[2].metric_name).toBe(
       "tensorzero::evaluation_name::entity_extraction::evaluator_name::exact_match",
     );
-    expect(statistics[2].datapoint_count).toBe(41);
-    expect(statistics[2].mean_metric).toBeCloseTo(0.5122);
+    expect(statistics[2].datapoint_count).toBe(42);
+    expect(statistics[2].mean_metric).toBeCloseTo(0.524);
     expect(statistics[2].stderr_metric).toBeCloseTo(0.08);
 
     expect(statistics[3].evaluation_run_id).toBe(evaluation_run_id2);
     expect(statistics[3].metric_name).toBe(
       "tensorzero::evaluation_name::entity_extraction::evaluator_name::count_sports",
     );
-    expect(statistics[3].datapoint_count).toBe(41);
-    expect(statistics[3].mean_metric).toBeCloseTo(0.7805);
+    expect(statistics[3].datapoint_count).toBe(42);
+    expect(statistics[3].mean_metric).toBeCloseTo(0.762);
     expect(statistics[3].stderr_metric).toBeCloseTo(0.0665);
   });
 });
@@ -569,14 +569,14 @@ describe("getEvaluationRunInfosForDatapoint", () => {
 
   test("should return correct evaluation run info for ragged haiku datapoint", async () => {
     const evaluationRunInfos = await getEvaluationRunInfosForDatapoint(
-      "01963691-7489-74b3-837f-e386de14c5f9",
+      "0196374a-d03f-7420-9da5-1561cba71ddb",
       "write_haiku",
     );
 
     const expected = {
-      evaluation_run_id: "01963691-9d3c-7793-a8be-3937ebb849c1",
+      evaluation_run_id: "0196374b-04a3-7013-9049-e59ed5fe3f74",
       variant_name: "better_prompt_haiku_3_5",
-      most_recent_inference_date: "2025-04-14T23:10:35Z",
+      most_recent_inference_date: "2025-04-15T02:33:05Z",
     };
     expect(evaluationRunInfos).toHaveLength(1); // Ensure exactly one item
     expect(evaluationRunInfos).toEqual(expect.arrayContaining([expected]));
diff --git a/ui/app/utils/clickhouse/function.test.ts b/ui/app/utils/clickhouse/function.test.ts
index 07dfed3c3bf..731a92e31aa 100644
--- a/ui/app/utils/clickhouse/function.test.ts
+++ b/ui/app/utils/clickhouse/function.test.ts
@@ -590,7 +590,7 @@ describe("getVariantCounts", () => {
         variant_name: "initial_prompt_gpt4o_mini",
       },
       {
-        count: 156,
+        count: 155,
         last_used: "2025-04-15T02:33:07.000Z",
         variant_name: "better_prompt_haiku_3_5",
       },
diff --git a/ui/app/utils/clickhouse/inference.test.ts b/ui/app/utils/clickhouse/inference.test.ts
index fc054b2fb5b..8b92feeecb4 100644
--- a/ui/app/utils/clickhouse/inference.test.ts
+++ b/ui/app/utils/clickhouse/inference.test.ts
@@ -38,7 +38,7 @@ test("countInferencesForFunction returns correct counts", async () => {
     tool_choice: "none",
     parallel_tool_calls: false,
   });
-  expect(chatCount).toBe(804);
+  expect(chatCount).toBe(803);
 });
 
 // Test countInferencesForVariant
@@ -82,107 +82,95 @@ test("queryInferenceTable", async () => {
   expect(inferences2.length).toBe(10);
 });
 
-test("queryInferenceTable pages through all results correctly using before", async () => {
+test("queryInferenceTable pagination samples front and near-end pages correctly", async () => {
   const PAGE_SIZE = 100;
-  let currentPage = await queryInferenceTable({
-    page_size: PAGE_SIZE,
-  });
-
-  // Keep track of how many full pages we've seen
-  let numFullPages = 0;
-  let totalElements = 0;
-
-  while (currentPage.length === PAGE_SIZE) {
-    totalElements += currentPage.length;
 
-    // Verify each page is the correct size
-    expect(currentPage.length).toBe(PAGE_SIZE);
-    // Verify IDs are in descending order within each page
-    for (let i = 1; i < currentPage.length; i++) {
-      expect(currentPage[i - 1].id > currentPage[i].id).toBe(true);
-    }
-
-    // Get next page using last item's ID as cursor
-    currentPage = await queryInferenceTable({
-      before: currentPage[currentPage.length - 1].id,
-      page_size: PAGE_SIZE,
-    });
-
-    numFullPages++;
+  // --- Front of the table (most recent entries) ---
+  const firstPage = await queryInferenceTable({ page_size: PAGE_SIZE });
+  expect(firstPage.length).toBe(PAGE_SIZE);
+  for (let i = 1; i < firstPage.length; i++) {
+    expect(firstPage[i - 1].id > firstPage[i].id).toBe(true);
   }
 
-  // Add the remaining elements from the last page
-  totalElements += currentPage.length;
-
-  // The last page should have fewer items than PAGE_SIZE
-  // (unless the total happens to be exactly divisible by PAGE_SIZE)
-  expect(currentPage.length).toBeLessThanOrEqual(PAGE_SIZE);
-
-  // Verify total number of elements
-  expect(totalElements).toBe(8143);
-
-  // We should have seen at least one full page
-  expect(numFullPages).toBeGreaterThan(0);
-});
+  const secondPage = await queryInferenceTable({
+    before: firstPage[firstPage.length - 1].id,
+    page_size: PAGE_SIZE,
+  });
+  expect(secondPage.length).toBe(PAGE_SIZE);
+  for (let i = 1; i < secondPage.length; i++) {
+    expect(secondPage[i - 1].id > secondPage[i].id).toBe(true);
+  }
 
-test("queryInferenceTable pages through all results correctly using after", async () => {
-  const PAGE_SIZE = 100;
+  // --- Near the end of the table (oldest entries) ---
+  const bounds = await queryInferenceTableBounds();
+  // bounds.last_id is the earliest (oldest) inference ID
+  const lastID = bounds.last_id!;
 
-  // First get to the last page to find the earliest ID
-  let currentPage = await queryInferenceTable({
+  const endPage1 = await queryInferenceTable({
+    before: lastID,
     page_size: PAGE_SIZE,
   });
-
-  while (currentPage.length === PAGE_SIZE) {
-    currentPage = await queryInferenceTable({
-      before: currentPage[currentPage.length - 1].id,
-      page_size: PAGE_SIZE,
-    });
+  expect(endPage1.length).toBeGreaterThan(0);
+  for (let i = 1; i < endPage1.length; i++) {
+    expect(endPage1[i - 1].id > endPage1[i].id).toBe(true);
   }
 
-  // Now we have the earliest ID, let's page forward using after
-  const firstId = currentPage[currentPage.length - 1].id;
-  currentPage = await queryInferenceTable({
-    after: firstId,
+  const endPage2 = await queryInferenceTable({
+    before: endPage1[endPage1.length - 1].id,
     page_size: PAGE_SIZE,
   });
+  // this may be empty if there are no more older entries
+  expect(endPage2.length).toBeGreaterThanOrEqual(0);
+  for (let i = 1; i < endPage2.length; i++) {
+    expect(endPage2[i - 1].id > endPage2[i].id).toBe(true);
+  }
 
-  // Keep track of how many full pages we've seen
-  let numFullPages = 0;
-  let totalElements = 0;
-
-  while (currentPage.length === PAGE_SIZE) {
-    totalElements += currentPage.length;
+  // Try to grab the last page by after
+  const lastPageByAfter = await queryInferenceTable({
+    after: endPage1[endPage1.length - 1].id,
+    page_size: PAGE_SIZE,
+  });
+  expect(lastPageByAfter.length).toBe(PAGE_SIZE);
+  for (let i = 1; i < lastPageByAfter.length; i++) {
+    expect(lastPageByAfter[i - 1].id > lastPageByAfter[i].id).toBe(true);
+  }
+});
 
-    // Verify each page is the correct size
-    expect(currentPage.length).toBe(PAGE_SIZE);
+test("queryInferenceTable pages through results correctly using after with inference ID", async () => {
+  const PAGE_SIZE = 20;
+  // Get a limited sample of inferences instead of the full table
+  const sampleSize = 100;
 
-    // Verify IDs are in descending order within each page
-    for (let i = 1; i < currentPage.length; i++) {
-      expect(currentPage[i - 1].id > currentPage[i].id).toBe(true);
-    }
+  // Now, page forward using after through our sample
+  let results: Awaited<ReturnType<typeof queryInferenceTable>> = [];
+  let after: string | undefined = undefined;
+  let pageCount = 0;
 
-    // Get next page using first item's ID as cursor
-    currentPage = await queryInferenceTable({
-      after: currentPage[0].id,
+  // Only page through our sample size
+  while (results.length < sampleSize) {
+    const page = await queryInferenceTable({
       page_size: PAGE_SIZE,
+      after,
     });
+    if (page.length === 0) break;
 
-    numFullPages++;
-  }
+    // IDs should be in descending order within the page
+    for (let i = 1; i < page.length; i++) {
+      expect(page[i - 1].id > page[i].id).toBe(true);
+    }
 
-  // Add the remaining elements from the last page
-  totalElements += currentPage.length;
+    results = results.concat(page);
+    after = page[page.length - 1].id;
+    pageCount++;
 
-  // The last page should have fewer items than PAGE_SIZE
-  // (unless the total happens to be exactly divisible by PAGE_SIZE)
-  expect(currentPage.length).toBeLessThanOrEqual(PAGE_SIZE);
+    if (page.length < PAGE_SIZE) break;
 
-  // Verify total number of elements matches the previous test
-  expect(totalElements).toBe(8142); // One less than with before because we excluded the first ID
+    // Safety check to avoid infinite loops
+    if (pageCount > 10) break;
+  }
 
-  // We should have seen at least one full page
-  expect(numFullPages).toBeGreaterThan(0);
+  // Should have paged at least once
+  expect(pageCount).toBeGreaterThanOrEqual(1);
 });
 
 test("queryInferenceTable after future timestamp is empty", async () => {
@@ -205,113 +193,108 @@ test("queryInferenceTable before past timestamp is empty", async () => {
   expect(inferences.length).toBe(0);
 });
 
-test("queryInferenceTableByEpisodeId pages through all results correctly using before with episode_id", async () => {
+test("queryInferenceTableByEpisodeId pages through initial and final pages correctly using before with episode_id", async () => {
   const PAGE_SIZE = 20;
-  let currentPage = await queryInferenceTableByEpisodeId({
+  const episodeId = "01942e26-618b-7b80-b492-34bed9f6d872";
+
+  // First page
+  const firstPage = await queryInferenceTableByEpisodeId({
     page_size: PAGE_SIZE,
-    episode_id: "01942e26-618b-7b80-b492-34bed9f6d872",
+    episode_id: episodeId,
   });
-
-  // Keep track of how many full pages we've seen
-  let numFullPages = 0;
-  let totalElements = 0;
-
-  while (currentPage.length === PAGE_SIZE) {
-    totalElements += currentPage.length;
-
-    // Verify each page is the correct size
-    expect(currentPage.length).toBe(PAGE_SIZE);
-    // Verify IDs are in descending order within each page
-    for (let i = 1; i < currentPage.length; i++) {
-      expect(currentPage[i - 1].id > currentPage[i].id).toBe(true);
-    }
-
-    // Get next page using last item's ID as cursor
-    currentPage = await queryInferenceTableByEpisodeId({
-      before: currentPage[currentPage.length - 1].id,
-      page_size: PAGE_SIZE,
-      episode_id: "01942e26-618b-7b80-b492-34bed9f6d872",
-    });
-
-    numFullPages++;
+  // Should be exactly PAGE_SIZE
+  expect(firstPage.length).toBe(PAGE_SIZE);
+  // IDs in descending order
+  for (let i = 1; i < firstPage.length; i++) {
+    expect(firstPage[i - 1].id > firstPage[i].id).toBe(true);
   }
 
-  // Add the remaining elements from the last page
-  totalElements += currentPage.length;
-
-  // The last page should have fewer items than PAGE_SIZE
-  // (unless the total happens to be exactly divisible by PAGE_SIZE)
-  expect(currentPage.length).toBeLessThanOrEqual(PAGE_SIZE);
-
-  // Verify total number of elements
-  expect(totalElements).toBe(35);
+  // Next (and in this case final) page
+  const lastPage = await queryInferenceTableByEpisodeId({
+    before: firstPage[firstPage.length - 1].id,
+    page_size: PAGE_SIZE,
+    episode_id: episodeId,
+  });
+  // Should be smaller or equal to PAGE_SIZE (here 15)
+  expect(lastPage.length).toBeLessThanOrEqual(PAGE_SIZE);
+  // IDs in descending order
+  for (let i = 1; i < lastPage.length; i++) {
+    expect(lastPage[i - 1].id > lastPage[i].id).toBe(true);
+  }
 
-  // We should have seen at least one full page
-  expect(numFullPages).toBeGreaterThan(0);
+  // Verify total elements matches expected count
+  expect(firstPage.length + lastPage.length).toBe(35);
 });
 
-test("queryInferenceTableByEpisodeId pages through all results correctly using after with episode_id", async () => {
+test("queryInferenceTableByEpisodeId pages through a sample of results correctly using after with episode_id", async () => {
   const PAGE_SIZE = 20;
+  const episodeId = "01942e26-469a-7553-af00-cb0495dc7bb5";
 
-  // First get to the last page to find the earliest ID
-  let currentPage = await queryInferenceTableByEpisodeId({
+  // Get the first page
+  const firstPage = await queryInferenceTableByEpisodeId({
     page_size: PAGE_SIZE,
-    episode_id: "01942e26-469a-7553-af00-cb0495dc7bb5",
+    episode_id: episodeId,
   });
 
-  while (currentPage.length === PAGE_SIZE) {
-    currentPage = await queryInferenceTableByEpisodeId({
-      before: currentPage[currentPage.length - 1].id,
-      page_size: PAGE_SIZE,
-      episode_id: "01942e26-469a-7553-af00-cb0495dc7bb5",
-    });
+  // Verify first page properties
+  expect(firstPage.length).toBe(PAGE_SIZE);
+  for (let i = 1; i < firstPage.length; i++) {
+    expect(firstPage[i - 1].id > firstPage[i].id).toBe(true);
   }
 
-  // Now we have the earliest ID, let's page forward using after
-  const firstId = currentPage[currentPage.length - 1].id;
-  currentPage = await queryInferenceTableByEpisodeId({
-    after: firstId,
+  // Get the second page using the last ID of the first page
+  const secondPage = await queryInferenceTableByEpisodeId({
+    before: firstPage[firstPage.length - 1].id,
     page_size: PAGE_SIZE,
-    episode_id: "01942e26-469a-7553-af00-cb0495dc7bb5",
+    episode_id: episodeId,
   });
 
-  // Keep track of how many full pages we've seen
-  let numFullPages = 0;
-  let totalElements = 0;
+  // Verify second page properties
+  expect(secondPage.length).toBe(PAGE_SIZE);
+  for (let i = 1; i < secondPage.length; i++) {
+    expect(secondPage[i - 1].id > secondPage[i].id).toBe(true);
+  }
 
-  while (currentPage.length === PAGE_SIZE) {
-    totalElements += currentPage.length;
+  // Now test paging forward using after
+  // Get a page starting after the first item of the second page
+  const forwardPage = await queryInferenceTableByEpisodeId({
+    after: secondPage[0].id,
+    page_size: PAGE_SIZE,
+    episode_id: episodeId,
+  });
 
-    // Verify each page is the correct size
-    expect(currentPage.length).toBe(PAGE_SIZE);
+  // Verify forward paging works
+  expect(forwardPage.length).toBeGreaterThan(0);
+  expect(forwardPage.length).toBeLessThanOrEqual(PAGE_SIZE);
 
-    // Verify IDs are in descending order within each page
-    for (let i = 1; i < currentPage.length; i++) {
-      expect(currentPage[i - 1].id > currentPage[i].id).toBe(true);
-    }
+  // Verify IDs are in descending order
+  for (let i = 1; i < forwardPage.length; i++) {
+    expect(forwardPage[i - 1].id > forwardPage[i].id).toBe(true);
+  }
 
-    // Get next page using first item's ID as cursor
-    currentPage = await queryInferenceTableByEpisodeId({
-      after: currentPage[0].id,
+  // Get the last page to verify we can reach the end
+  let lastPage = firstPage;
+  let nextPage = secondPage;
+
+  // Just get a couple more pages to avoid too many queries
+  for (let i = 0; i < 2 && nextPage.length === PAGE_SIZE; i++) {
+    lastPage = nextPage;
+    nextPage = await queryInferenceTableByEpisodeId({
+      before: lastPage[lastPage.length - 1].id,
       page_size: PAGE_SIZE,
-      episode_id: "01942e26-469a-7553-af00-cb0495dc7bb5",
+      episode_id: episodeId,
     });
-
-    numFullPages++;
   }
 
-  // Add the remaining elements from the last page
-  totalElements += currentPage.length;
-
-  // The last page should have fewer items than PAGE_SIZE
-  // (unless the total happens to be exactly divisible by PAGE_SIZE)
-  expect(currentPage.length).toBeLessThanOrEqual(PAGE_SIZE);
-
-  // Verify total number of elements matches the previous test
-  expect(totalElements).toBe(42);
+  // Verify we can get to the end if needed
+  if (nextPage.length < PAGE_SIZE) {
+    // We reached the last page
+    expect(nextPage.length).toBeLessThan(PAGE_SIZE);
+  }
 
-  // We should have seen at least two full pages
-  expect(numFullPages).toBeGreaterThan(1);
+  // Verify total count is as expected (42 from previous test)
+  const totalCount = await countInferencesForEpisode(episodeId);
+  expect(totalCount).toBe(43);
 });
 
 // queryInferenceTableBounds and queryEpisodeTableBounds are the same because the early inferences are in singleton episodes.
@@ -471,113 +454,6 @@ test("queryEpisodeTable", async () => {
   }
 });
 
-test("queryEpisodeTable pages through all results correctly using before", async () => {
-  const PAGE_SIZE = 100;
-  let currentPage = await queryEpisodeTable({
-    page_size: PAGE_SIZE,
-  });
-
-  // Keep track of how many full pages we've seen
-  let numFullPages = 0;
-  let totalElements = 0;
-
-  while (currentPage.length === PAGE_SIZE) {
-    totalElements += currentPage.length;
-
-    // Verify each page is the correct size
-    expect(currentPage.length).toBe(PAGE_SIZE);
-    // Verify IDs are in descending order within each page
-    for (let i = 1; i < currentPage.length; i++) {
-      expect(
-        currentPage[i - 1].last_inference_id > currentPage[i].last_inference_id,
-      ).toBe(true);
-    }
-
-    // Get next page using last item's ID as cursor
-    currentPage = await queryEpisodeTable({
-      before: currentPage[currentPage.length - 1].last_inference_id,
-      page_size: PAGE_SIZE,
-    });
-
-    numFullPages++;
-  }
-
-  // Add the remaining elements from the last page
-  totalElements += currentPage.length;
-
-  // The last page should have fewer items than PAGE_SIZE
-  // (unless the total happens to be exactly divisible by PAGE_SIZE)
-  expect(currentPage.length).toBeLessThanOrEqual(PAGE_SIZE);
-
-  // Verify total number of elements
-  expect(totalElements).toBe(2250);
-
-  // We should have seen at least 9 full pages
-  expect(numFullPages).toBeGreaterThan(8);
-});
-
-test("queryEpisodeTable pages through all results correctly using after", async () => {
-  const PAGE_SIZE = 100;
-
-  // First get to the last page to find the earliest ID
-  let currentPage = await queryEpisodeTable({
-    page_size: PAGE_SIZE,
-  });
-
-  while (currentPage.length === PAGE_SIZE) {
-    currentPage = await queryEpisodeTable({
-      before: currentPage[currentPage.length - 1].last_inference_id,
-      page_size: PAGE_SIZE,
-    });
-  }
-
-  // Now we have the earliest ID, let's page forward using after
-  const firstId = currentPage[currentPage.length - 1].last_inference_id;
-  currentPage = await queryEpisodeTable({
-    after: firstId,
-    page_size: PAGE_SIZE,
-  });
-
-  // Keep track of how many full pages we've seen
-  let numFullPages = 0;
-  let totalElements = 0;
-
-  while (currentPage.length === PAGE_SIZE) {
-    totalElements += currentPage.length;
-
-    // Verify each page is the correct size
-    expect(currentPage.length).toBe(PAGE_SIZE);
-
-    // Verify IDs are in descending order within each page
-    for (let i = 1; i < currentPage.length; i++) {
-      expect(
-        currentPage[i - 1].last_inference_id > currentPage[i].last_inference_id,
-      ).toBe(true);
-    }
-
-    // Get next page using first item's ID as cursor
-    currentPage = await queryEpisodeTable({
-      after: currentPage[0].last_inference_id,
-      page_size: PAGE_SIZE,
-    });
-
-    numFullPages++;
-  }
-
-  // Add the remaining elements from the last page
-  totalElements += currentPage.length;
-
-  // The last page should have fewer items than PAGE_SIZE
-  // (unless the total happens to be exactly divisible by PAGE_SIZE)
-  expect(currentPage.length).toBeLessThanOrEqual(PAGE_SIZE);
-
-  // Verify total number of elements matches the previous test
-  expect(totalElements).toBe(2249); // One less than with before because we excluded the first ID
-
-  // We should have seen at least 9 full pages
-  expect(numFullPages).toBeGreaterThan(8);
-});
-
 test("countInferencesForEpisode", async () => {
   const count = await countInferencesForEpisode(
     "01942e26-549f-7153-ac56-dd1d23d30f8c",
@@ -640,7 +516,7 @@ test("countInferencesByFunction", async () => {
         max_timestamp: "2025-04-15T02:33:10Z",
       },
       {
-        count: 804,
+        count: 803,
         function_name: "write_haiku",
         max_timestamp: "2025-04-15T02:33:07Z",
       },
diff --git a/ui/app/utils/clickhouse/inference.ts b/ui/app/utils/clickhouse/inference.ts
index 94677a27b84..3090c23f09d 100644
--- a/ui/app/utils/clickhouse/inference.ts
+++ b/ui/app/utils/clickhouse/inference.ts
@@ -253,6 +253,8 @@ export async function queryEpisodeTable(params: {
   };
   if (!before && !after) {
     // No before/after => just the most recent page_size items
+    // The weird 'WHERE' clause lowers the peak memory usage (and also seems to speed up the query),
+    // though it unfortunately still does a full table scan.
     query = `
       SELECT
         uint_to_uuid(episode_id_uint) as episode_id,
@@ -261,6 +263,7 @@ export async function queryEpisodeTable(params: {
         formatDateTime(max(UUIDv7ToDateTime(uint_to_uuid(id_uint))), '%Y-%m-%dT%H:%i:%SZ') as end_time,
         uint_to_uuid(max(id_uint)) as last_inference_id
       FROM InferenceByEpisodeId FINAL
+      WHERE episode_id_uint in (SELECT DISTINCT ON (episode_id) toUInt128(episode_id) FROM InferenceById FINAL ORDER BY id_uint DESC LIMIT {page_size:UInt32})
       GROUP BY episode_id
       ORDER BY toUInt128(last_inference_id) DESC
       LIMIT {page_size:UInt32}
diff --git a/ui/app/utils/config/index.test.ts b/ui/app/utils/config/index.test.ts
index 33d17e3a557..98e31749022 100644
--- a/ui/app/utils/config/index.test.ts
+++ b/ui/app/utils/config/index.test.ts
@@ -48,7 +48,9 @@ test("parse e2e config", async () => {
   const gcpVertexGeminiProVariant =
     jsonTestFunction.variants["gcp-vertex-gemini-pro"];
   if (gcpVertexGeminiProVariant.type === "chat_completion") {
-    expect(gcpVertexGeminiProVariant.model).toBe("gemini-2.5-pro-preview-05-06");
+    expect(gcpVertexGeminiProVariant.model).toBe(
+      "gemini-2.5-pro-preview-05-06",
+    );
   } else {
     throw new Error("GCP Vertex Gemini Pro variant not found");
   }
diff --git a/ui/e2e_tests/dynamic_evaluations.projects.$project_name.spec.ts b/ui/e2e_tests/dynamic_evaluations.projects.$project_name.spec.ts
index b2c1a79e4bf..4063cf1ea07 100644
--- a/ui/e2e_tests/dynamic_evaluations.projects.$project_name.spec.ts
+++ b/ui/e2e_tests/dynamic_evaluations.projects.$project_name.spec.ts
@@ -11,6 +11,7 @@ test("dynamic evaluation project page should render and show correct information
   ).toBeVisible();
   // Click on the run selector
   await page.getByText("Select dynamic evaluation runs to compare...").click();
+  await page.waitForTimeout(1000);
   // Check that the run selector has the correct runs
   await expect(page.getByText("aac7e7")).toBeVisible();
   await expect(page.getByText("8fddbd")).toBeVisible();
diff --git a/ui/fixtures/Dockerfile b/ui/fixtures/Dockerfile
new file mode 100644
index 00000000000..fa1ee5f25c7
--- /dev/null
+++ b/ui/fixtures/Dockerfile
@@ -0,0 +1,7 @@
+# This Dockerfile is used to load fixtures into a (separate) ClickHouse server
+# We need the AWS CLI available to download our large fixtures from s3
+FROM clickhouse/clickhouse-server:24.12-alpine
+
+RUN apk add --no-cache python3 py3-pip aws-cli
+RUN pip install parquet-tools --break-system-packages
+COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
\ No newline at end of file
diff --git a/ui/fixtures/README.md b/ui/fixtures/README.md
new file mode 100644
index 00000000000..c28669c4135
--- /dev/null
+++ b/ui/fixtures/README.md
@@ -0,0 +1,16 @@
+# TensorZero fixtures
+
+Most of our fixtures are stored in this directory, with the exception of some large (>100MB) files
+
+## Pulling fixtures from S3
+
+Our large fixtures are stored in an S3-compatible object store (currently Cloudflare R2)
+They can be manually downloaded with `uv run ./download-fixtures.py`
+
+## Writing new large fixtures
+
+Large fixtures should _not_ be committed to the repository. Instead:
+
+1. Add the new fixtures to `./s3-fixtures`
+2. Run `./upload-fixtures.sh`
+3. List the new fixtures files in `uv run ./download-fixtures.py`
diff --git a/ui/fixtures/chat_inference_examples.jsonl b/ui/fixtures/chat_inference_examples.jsonl
index 3bceab72165..e75e0d58e33 100644
--- a/ui/fixtures/chat_inference_examples.jsonl
+++ b/ui/fixtures/chat_inference_examples.jsonl
@@ -3224,7 +3224,6 @@
 {"id":"01963691-b32b-7970-8b61-55579f8a1f4b","function_name":"write_haiku","variant_name":"better_prompt_haiku_3_5","episode_id":"01963691-b32b-7970-8b61-5560d9ccfb09","input":"{\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"value\":{\"topic\":\"fuel\"}}]}]}","output":"[{\"type\":\"text\",\"text\":\"Let me think about fuel... I'll want to capture its power, its essence, perhaps with some dramatic imagery. Fuel represents energy, movement, transformation...\\n\\nHmm, I could go with an industrial or elemental approach. Maybe something that suggests both raw power and potential destruction.\\n\\n*thinking through syllable patterns*\\n\\nBurning liquid heart\\nExplosive veins of power\\nIgnite earth's deep breath\\n\\nThis haiku tries to personify fuel as something almost alive - with a \\\"heart\\\" and \\\"veins\\\", suggesting its dynamic, transformative nature. The last line hints at its profound impact on our world, with \\\"earth's deep breath\\\" implying both the geological origins of fossil fuels and their transformative environmental consequences.\\n\\nThe imagery is hyperbolic, treating fuel not just as a substance, but as a living, breathing force of potential and destruction.\"}]","tool_params":"","inference_params":"{\"chat_completion\":{}}","processing_time_ms":11,"tags":{"tensorzero::evaluation_name":"haiku","tensorzero::dataset_name":"foo","tensorzero::datapoint_id":"0193d6c2-7b10-7560-be64-17630b61fe6b","tensorzero::evaluation_run_id":"01963691-9d3c-7793-a8be-3937ebb849c1"},"extra_body":"[]"}
 {"id":"01963691-9e67-7902-8c14-e6f3dc794a3f","function_name":"write_haiku","variant_name":"better_prompt_haiku_3_5","episode_id":"01963691-9e67-7902-8c14-e708df8dc808","input":"{\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"value\":{\"topic\":\"upward\"}}]}]}","output":"[{\"type\":\"text\",\"text\":\"Let me brainstorm some vivid, hyperbolic imagery related to \\\"upward\\\"...\\n\\nI'm thinking about movement, aspiration, climbing, reaching - perhaps something that emphasizes the dramatic vertical trajectory. Maybe I'll use some metaphors of extreme elevation or stretching beyond normal limits.\\n\\n*thinking out loud*\\n- Climbing mountains\\n- Rocket launch\\n- Ambition as a force\\n- Stretching beyond known boundaries\\n\\nAh, here's a haiku that captures the essence of \\\"upward\\\":\\n\\nGravity trembles\\nAmbition pierces the sky\\nDreams burst past horizons\"}]","tool_params":"","inference_params":"{\"chat_completion\":{}}","processing_time_ms":7,"tags":{"tensorzero::dataset_name":"foo","tensorzero::datapoint_id":"01934fc5-ea98-71f0-8191-9fd88f34c28b","tensorzero::evaluation_name":"haiku","tensorzero::evaluation_run_id":"01963691-9d3c-7793-a8be-3937ebb849c1"},"extra_body":"[]"}
 {"id":"01963691-9e67-7902-8c14-e712cada1559","function_name":"write_haiku","variant_name":"better_prompt_haiku_3_5","episode_id":"01963691-9e67-7902-8c14-e723d2d8bc12","input":"{\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"value\":{\"topic\":\"pate\"}}]}]}","output":"[{\"type\":\"text\",\"text\":\"Let me think about this... a haiku about pate (a fancy spreadable meat dish). I want to capture its richness and smooth texture with some dramatic imagery.\\n\\nHmm... perhaps something about its luxurious, dense nature. Maybe a contrast between the delicate spread and its intense flavor. \\n\\nAh, here's a haiku:\\n\\nSilken meat clouds spread\\nTrembling like soft moonlight's breath\\nLiver's velvet kiss\\n\\nI liked the idea of portraying the pate as something ethereal yet intense - using metaphors of clouds, moonlight, and a sensual \\\"velvet kiss\\\" to elevate this rich food item into a poetic experience. The 5-7-5 syllable structure is maintained, and I've used somewhat hyperbolic language to make the description more dramatic.\"}]","tool_params":"","inference_params":"{\"chat_completion\":{}}","processing_time_ms":7,"tags":{"tensorzero::evaluation_name":"haiku","tensorzero::evaluation_run_id":"01963691-9d3c-7793-a8be-3937ebb849c1","tensorzero::datapoint_id":"01942b0c-0e70-78b2-85e7-1d5ef54f793a","tensorzero::dataset_name":"foo"},"extra_body":"[]"}
-{"id":"01963691-9e67-7902-8c14-e734d7d5dfe5","function_name":"write_haiku","variant_name":"better_prompt_haiku_3_5","episode_id":"01963691-9e67-7902-8c14-e74b167260f0","input":"{\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"value\":{\"topic\":\"sheet\"}}]}]}","output":"[{\"type\":\"text\",\"text\":\"Hmm, a haiku about a sheet... Let me think through some imagery.\\n\\nA sheet could be something soft, light, covering, mysterious. Maybe I'll play with the idea of a sheet as something both comforting and slightly eerie - like a ghost or a metaphor for emptiness.\\n\\nThinking about syllables, hyperbole, and vivid imagery...\\n\\nGossamer thin cloth\\nBillows like a phantom's sigh\\nSwallowing moonlight\"}]","tool_params":"","inference_params":"{\"chat_completion\":{}}","processing_time_ms":3557,"tags":{"tensorzero::datapoint_id":"01963691-7489-74b3-837f-e386de14c5f9","tensorzero::evaluation_run_id":"01963691-9d3c-7793-a8be-3937ebb849c1","tensorzero::dataset_name":"foo","tensorzero::evaluation_name":"haiku"},"extra_body":"[]"}
 {"id":"01963691-9e67-7902-8c14-e75a72d53174","function_name":"write_haiku","variant_name":"better_prompt_haiku_3_5","episode_id":"01963691-9e67-7902-8c14-e763058299eb","input":"{\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"value\":{\"topic\":\"literature\"}}]}]}","output":"[{\"type\":\"text\",\"text\":\"Let me brainstorm some imagery...\\n\\nLiterature is such a vast world of imagination and expression. I want to capture its depth, its power to transport, its almost magical quality. Perhaps I'll focus on books as portals, or words as living things...\\n\\n*thinking out loud*\\nMaybe something about how words can bloom like flowers, or how stories can consume us like fire. The syllable constraints of haiku will require something concise yet evocative.\\n\\n*drafting haiku*\\nInk-drenched pages burn\\nWorlds explode from paper's edge\\nUniverses breathe\\n\\n*analyzing the haiku*\\nThis version captures the transformative power of literature - how a simple page can contain entire worlds, how words can feel alive and consuming. The \\\"burn\\\" and \\\"explode\\\" create that hyperbolic sense of literature's intensity, while \\\"universes breathe\\\" suggests the living, generative quality of stories.\\n\\nWould you like me to generate another version?\"}]","tool_params":"","inference_params":"{\"chat_completion\":{}}","processing_time_ms":7,"tags":{"tensorzero::dataset_name":"foo","tensorzero::evaluation_run_id":"01963691-9d3c-7793-a8be-3937ebb849c1","tensorzero::evaluation_name":"haiku","tensorzero::datapoint_id":"0193fa3f-c700-74e1-873e-86861df758e4"},"extra_body":"[]"}
 {"id":"01963691-ae9d-7632-8d43-05326bdca7ac","function_name":"write_haiku","variant_name":"better_prompt_haiku_3_5","episode_id":"01963691-ae9d-7632-8d43-0540d6be2ad8","input":"{\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"value\":{\"topic\":\"fob\"}}]}]}","output":"[{\"type\":\"text\",\"text\":\"Let me think about this... FOB could mean \\\"Fresh Off the Boat\\\", a term often used to describe recent immigrants, or it could refer to a keychain remote or even a military term for \\\"Forward Operating Base\\\". I'll lean into the immigrant experience angle and try to capture something both poignant and slightly exaggerated.\\n\\nHmm... I want to use some vivid imagery that speaks to the experience of cultural transition and feeling out of place.\\n\\nHere's my haiku:\\n\\nDistant shores whisper\\nTsunami of strange customs\\nDrowns my mother's tongue\\n\\nThe haiku uses the hyperbolic image of a \\\"tsunami of strange customs\\\" to represent the overwhelming experience of adapting to a new culture, while also touching on the potential loss of linguistic and cultural heritage. The first line hints at displacement, the second line dramatically illustrates cultural shock, and the third line poignantly suggests the personal cost of immigration.\"}]","tool_params":"","inference_params":"{\"chat_completion\":{}}","processing_time_ms":3,"tags":{"tensorzero::evaluation_run_id":"01963691-9d3c-7793-a8be-3937ebb849c1","tensorzero::datapoint_id":"0193b670-5f30-7851-b204-49715577b880","tensorzero::evaluation_name":"haiku","tensorzero::dataset_name":"foo"},"extra_body":"[]"}
 {"id":"01963691-ad17-79a0-8e60-f5dfefc5ad67","function_name":"write_haiku","variant_name":"better_prompt_haiku_3_5","episode_id":"01963691-ad17-79a0-8e60-f5e10672ecaa","input":"{\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"value\":{\"topic\":\"painting\"}}]}]}","output":"[{\"type\":\"text\",\"text\":\"Let me brainstorm some vivid, exaggerated imagery for a haiku about painting...\\n\\nAh, painting - I'm thinking about the intense passion of creation, the wild sweep of a brush, the explosion of color...\\n\\nHmm, maybe something like:\\n\\nBrush blazes crimson\\nCanvas trembles, torn apart\\nBy storms of color\\n\\nThe haiku captures the dramatic, almost violent act of painting - the brush as a weapon of creativity, the canvas as something that can barely contain the artist's vision. The hyperbole is in the \\\"blazing\\\" brush and the canvas being \\\"torn apart\\\" by color, suggesting painting is not just a gentle act, but a powerful, transformative experience.\"}]","tool_params":"","inference_params":"{\"chat_completion\":{}}","processing_time_ms":7,"tags":{"tensorzero::datapoint_id":"0193aa65-de18-7d12-ade5-d3196e747a59","tensorzero::evaluation_run_id":"01963691-9d3c-7793-a8be-3937ebb849c1","tensorzero::dataset_name":"foo","tensorzero::evaluation_name":"haiku"},"extra_body":"[]"}
diff --git a/ui/fixtures/check-fixtures.sh b/ui/fixtures/check-fixtures.sh
index 059ab5e9fbd..4c0314dbbde 100755
--- a/ui/fixtures/check-fixtures.sh
+++ b/ui/fixtures/check-fixtures.sh
@@ -14,8 +14,8 @@ all_tables["FloatMetricFeedback"]="float_metric_feedback_examples.jsonl"
 all_tables["FloatMetricFeedbackByTargetId"]="float_metric_feedback_examples.jsonl"
 all_tables["CommentFeedback"]="comment_feedback_examples.jsonl"
 all_tables["DemonstrationFeedback"]="demonstration_feedback_examples.jsonl"
-all_tables["ChatInference"]="chat_inference_examples.jsonl"
-all_tables["ModelInference"]="model_inference_examples.jsonl"
+all_tables["ChatInference"]="chat_inference_examples.jsonl ./s3-fixtures/large_chat_inference.parquet"
+all_tables["ModelInference"]="model_inference_examples.jsonl ./s3-fixtures/large_model_inference.parquet"
 all_tables["ChatInferenceDatapoint FINAL"]="chat_inference_datapoint_examples.jsonl"
 all_tables["JsonInferenceDatapoint FINAL"]="json_inference_datapoint_examples.jsonl"
 all_tables["ModelInferenceCache"]="model_inference_cache_examples.jsonl"
@@ -39,18 +39,31 @@ for table in "${!all_tables[@]}"; do
 
     # Get the list of files for this table
     read -r -a files <<< "${all_tables[$table]}"
+    echo "Files: ${files[@]}"
 
     # Count total non-empty lines across all source files
     total_file_count=0
     for file in "${files[@]}"; do
         if [ -f "$file" ]; then
-            file_count=$(grep -v '^[[:space:]]*$' "$file" | wc -l)
-            echo "  - $file: $file_count rows"
-            total_file_count=$((total_file_count + file_count))
+            if [[ "$file" == *.parquet ]]; then
+                # For parquet files, use parquet-tools to count rows
+                if command -v parquet-tools &> /dev/null; then
+                    file_count=$(parquet-tools inspect "$file" | grep "num_rows:" | awk '{print $2}')
+                    echo "  - $file: $file_count rows (parquet)"
+                else
+                    echo "  - WARNING: parquet-tools not installed, cannot count rows in $file"
+                    mismatch=1
+                fi
+            else
+                # For regular text files, count non-empty lines
+                file_count=$(grep -v '^[[:space:]]*$' "$file" | wc -l)
+                echo "  - $file: $file_count rows"
+            fi
         else
-            echo "  - WARNING: File $file not found"
+            echo "  - WARNING: File $file not found in current directory or s3-fixtures/"
             mismatch=1
         fi
+        total_file_count=$(($total_file_count + $file_count))
     done
 
     # Compare counts
diff --git a/ui/fixtures/docker-compose.e2e.yml b/ui/fixtures/docker-compose.e2e.yml
index 7789346ecd5..f33ca1d3ea6 100644
--- a/ui/fixtures/docker-compose.e2e.yml
+++ b/ui/fixtures/docker-compose.e2e.yml
@@ -73,9 +73,11 @@ services:
       timeout: 1s
 
   fixtures:
-    image: clickhouse/clickhouse-server:24.12-alpine
+    build:
+      dockerfile: Dockerfile
     volumes:
       - .:/fixtures
+      - ~/.aws:/root/.aws
     environment:
       CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 1
       CLICKHOUSE_HOST: clickhouse
@@ -91,5 +93,5 @@ services:
       test: ["CMD", "test", "-f", "/tmp/load_complete.marker"]
       interval: 5s
       timeout: 1s
-      retries: 12 # Retry for up to a minute
+      retries: 48 # Retry for up to 4 minutes
       start_period: 5s # Give the script time to start
diff --git a/ui/fixtures/docker-compose.yml b/ui/fixtures/docker-compose.yml
index 8ca36807e82..a3334967a62 100644
--- a/ui/fixtures/docker-compose.yml
+++ b/ui/fixtures/docker-compose.yml
@@ -54,9 +54,11 @@ services:
       timeout: 1s
 
   fixtures:
-    image: clickhouse/clickhouse-server:24.12-alpine
+    build:
+      dockerfile: Dockerfile
     volumes:
-      - .:/fixtures:ro
+      - .:/fixtures
+      - ~/.aws:/root/.aws
     environment:
       CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 1
       CLICKHOUSE_HOST: clickhouse
@@ -68,3 +70,9 @@ services:
     # Keep this running to make 'check-docker-compose.sh' detect that all containers are healthy
     command:
       ["bash", "-c", "cd /fixtures && ./load_fixtures.sh && sleep infinity"]
+    healthcheck:
+      test: ["CMD", "test", "-f", "/tmp/load_complete.marker"]
+      interval: 5s
+      timeout: 1s
+      retries: 48 # Retry for up to 4 minutes
+      start_period: 5s # Give the script time to start
\ No newline at end of file
diff --git a/ui/fixtures/download-fixtures.py b/ui/fixtures/download-fixtures.py
new file mode 100644
index 00000000000..b002745dd92
--- /dev/null
+++ b/ui/fixtures/download-fixtures.py
@@ -0,0 +1,87 @@
+# /// script
+# dependencies = [
+#   "requests",
+# ]
+# ///
+
+
+import hashlib
+import os
+from pathlib import Path
+
+import requests
+
+# Constants
+PART_SIZE = 8388608
+FIXTURES = ["large_chat_inference.parquet", "large_model_inference.parquet"]
+R2_BUCKET = "https://pub-147e9850a60643208c411e70b636e956.r2.dev"
+S3_FIXTURES_DIR = Path("./s3-fixtures")
+
+
+def calculate_etag(file_path):
+    """Calculate S3/R2 style ETag for a file."""
+    file_size = os.path.getsize(file_path)
+    num_parts = (file_size + PART_SIZE - 1) // PART_SIZE
+
+    if num_parts == 1:
+        # Single part upload - just MD5 of the file
+        with open(file_path, "rb") as f:
+            return hashlib.md5(f.read()).hexdigest()
+    else:
+        # Multipart upload - concatenate MD5s of each part
+        md5s = []
+        with open(file_path, "rb") as f:
+            while True:
+                chunk = f.read(PART_SIZE)
+                if not chunk:
+                    break
+                md5s.append(hashlib.md5(chunk).digest())
+
+        # Calculate MD5 of concatenated part MD5s
+        combined_md5 = hashlib.md5(b"".join(md5s)).hexdigest()
+        return f"{combined_md5}-{num_parts}"
+
+
+def get_remote_etag(filename):
+    """Get ETag from R2 bucket."""
+    response = requests.head(f"{R2_BUCKET}/{filename}")
+    return response.headers.get("ETag", "").strip('"')
+
+
+def download_file(filename):
+    """Download file from R2 bucket."""
+    url = f"{R2_BUCKET}/{filename}"
+    response = requests.get(url, stream=True)
+    response.raise_for_status()
+
+    with open(S3_FIXTURES_DIR / filename, "wb") as f:
+        for chunk in response.iter_content(chunk_size=8192):
+            f.write(chunk)
+
+
+def main():
+    # Create s3-fixtures directory if it doesn't exist
+    S3_FIXTURES_DIR.mkdir(exist_ok=True)
+
+    for fixture in FIXTURES:
+        local_file = S3_FIXTURES_DIR / fixture
+
+        if not local_file.exists():
+            print(f"Downloading {fixture} (file doesn't exist locally)")
+            download_file(fixture)
+            continue
+
+        remote_etag = get_remote_etag(fixture)
+        local_etag = calculate_etag(local_file)
+
+        if local_etag != remote_etag:
+            print(f"Downloading {fixture} (ETag mismatch)")
+            print(f"Local ETag: {local_etag}")
+            print(f"Remote ETag: {remote_etag}")
+            download_file(fixture)
+        else:
+            print(f"Skipping {fixture} (up to date)")
+
+
+if __name__ == "__main__":
+    main()
diff --git a/ui/fixtures/load_fixtures.sh b/ui/fixtures/load_fixtures.sh
index 96d2966c2b9..e01af8dac0a 100755
--- a/ui/fixtures/load_fixtures.sh
+++ b/ui/fixtures/load_fixtures.sh
@@ -14,6 +14,9 @@ clickhouse-client --host $CLICKHOUSE_HOST --user chuser --password chpassword --
 clickhouse-client --host $CLICKHOUSE_HOST --user chuser --password chpassword --database tensorzero_ui_fixtures --query "INSERT INTO DynamicEvaluationRun FORMAT JSONEachRow" < dynamic_evaluation_run_examples.jsonl
 clickhouse-client --host $CLICKHOUSE_HOST --user chuser --password chpassword --database tensorzero_ui_fixtures --query "INSERT INTO DynamicEvaluationRunEpisode FORMAT JSONEachRow" < dynamic_evaluation_run_episode_examples.jsonl
 
+uv run ./download-fixtures.py
+clickhouse-client --host $CLICKHOUSE_HOST --user chuser --password chpassword --database tensorzero_ui_fixtures --query "INSERT INTO ChatInference FROM INFILE './s3-fixtures/large_chat_inference.parquet' FORMAT Parquet"
+clickhouse-client --host $CLICKHOUSE_HOST --user chuser --password chpassword --database tensorzero_ui_fixtures --query "INSERT INTO ModelInference FROM INFILE './s3-fixtures/large_model_inference.parquet' FORMAT Parquet"
 
 # Give ClickHouse some time to make the writes visible
 sleep 2
diff --git a/ui/fixtures/s3-fixtures/.gitkeep b/ui/fixtures/s3-fixtures/.gitkeep
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/ui/fixtures/upload-fixtures.sh b/ui/fixtures/upload-fixtures.sh
new file mode 100755
index 00000000000..050c8f8336f
--- /dev/null
+++ b/ui/fixtures/upload-fixtures.sh
@@ -0,0 +1,6 @@
+#!/bin/bash
+set -euxo pipefail
+
+cd $(dirname $0)/s3-fixtures
+
+aws s3 --endpoint-url https://19918a216783f0ac9e052233569aef60.r2.cloudflarestorage.com/ sync . s3://tensorzero-fixtures --checksum-algorithm CRC32
\ No newline at end of file


Reviews:
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

Fetching large S3 fixture files to test UI responsiveness with massive datasets is the primary goal of this PR. Key changes include:
- Switching the UI tests job runner from ubuntu-latest to a namespace-specific runner to leverage cache volumes.
- Adding two caching steps for the ./ui/fixtures/s3-fixtures directory to improve test performance by reducing S3 fetches.

<details>
<summary>Files not reviewed (3)</summary>

* **ui/fixtures/check-fixtures.sh**: Language not supported
* **ui/fixtures/download-fixtures.sh**: Language not supported
* **ui/fixtures/load_fixtures.sh**: Language not supported
</details>

<details>
<summary>Comments suppressed due to low confidence (1)</summary>

**.github/workflows/general.yml:415**
* [nitpick] The caching step for s3-fixtures is added here; please verify if this step is necessary given that a similar caching step is added later to avoid redundancy.
```
+      - name: Cache s3-fixtures
```
</details>


state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T18:43:44Z
user: login: Aaron1011
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T15:04:21Z
user: login: virajmehta
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-05-07T20:45:29Z
user: login: virajmehta
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-05-08T13:55:07Z
user: login: virajmehta
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-05-08T14:01:54Z
user: login: virajmehta
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-09T13:15:10Z
user: login: virajmehta
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-05-09T15:58:12Z
user: login: virajmehta
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-09T16:26:59Z

Review comments:
diff_hunk: @@ -478,6 +485,12 @@ jobs:
         working-directory: ui
         run: pnpm install --frozen-lockfile
 
+      - name: Cache s3-fixtures
+        uses: namespacelabs/nscloud-cache-action@v1
+        with:
+          path: |
+            ./ui/fixtures/s3-fixtures
path: .github/workflows/general.yml
user: login: Copilot
body: [nitpick] A duplicate caching step for s3-fixtures is present later in the job; consider consolidating these steps if they serve the same purpose to simplify the workflow.
```suggestion
# Removed duplicate caching step for s3-fixtures to simplify the workflow.
```
created_at: 2025-05-02T18:43:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -82,107 +82,85 @@ test("queryInferenceTable", async () => {
   expect(inferences2.length).toBe(10);
 });
 
-test("queryInferenceTable pages through all results correctly using before", async () => {
+test("queryInferenceTable pagination samples front and near-end pages correctly", async () => {
   const PAGE_SIZE = 100;
-  let currentPage = await queryInferenceTable({
-    page_size: PAGE_SIZE,
-  });
-
-  // Keep track of how many full pages we've seen
-  let numFullPages = 0;
-  let totalElements = 0;
-
-  while (currentPage.length === PAGE_SIZE) {
-    totalElements += currentPage.length;
-
-    // Verify each page is the correct size
-    expect(currentPage.length).toBe(PAGE_SIZE);
-    // Verify IDs are in descending order within each page
-    for (let i = 1; i < currentPage.length; i++) {
-      expect(currentPage[i - 1].id > currentPage[i].id).toBe(true);
-    }
-
-    // Get next page using last item's ID as cursor
-    currentPage = await queryInferenceTable({
-      before: currentPage[currentPage.length - 1].id,
-      page_size: PAGE_SIZE,
-    });
 
-    numFullPages++;
+  // --- Front of the table (most recent entries) ---
+  const firstPage = await queryInferenceTable({ page_size: PAGE_SIZE });
+  expect(firstPage.length).toBe(PAGE_SIZE);
+  for (let i = 1; i < firstPage.length; i++) {
+    expect(firstPage[i - 1].id > firstPage[i].id).toBe(true);
   }
 
-  // Add the remaining elements from the last page
-  totalElements += currentPage.length;
-
-  // The last page should have fewer items than PAGE_SIZE
-  // (unless the total happens to be exactly divisible by PAGE_SIZE)
-  expect(currentPage.length).toBeLessThanOrEqual(PAGE_SIZE);
-
-  // Verify total number of elements
-  expect(totalElements).toBe(3515);
-
-  // We should have seen at least one full page
-  expect(numFullPages).toBeGreaterThan(0);
-});
+  const secondPage = await queryInferenceTable({
+    before: firstPage[firstPage.length - 1].id,
+    page_size: PAGE_SIZE,
+  });
+  expect(secondPage.length).toBe(PAGE_SIZE);
+  for (let i = 1; i < secondPage.length; i++) {
+    expect(secondPage[i - 1].id > secondPage[i].id).toBe(true);
+  }
 
-test("queryInferenceTable pages through all results correctly using after", async () => {
-  const PAGE_SIZE = 100;
+  // --- Near the end of the table (oldest entries) ---
+  const bounds = await queryInferenceTableBounds();
+  // bounds.last_id is the earliest (oldest) inference ID
+  const lastID = bounds.last_id!;
 
-  // First get to the last page to find the earliest ID
-  let currentPage = await queryInferenceTable({
+  const endPage1 = await queryInferenceTable({
+    before: lastID,
path: ui/app/utils/clickhouse/inference.test.ts
user: login: Aaron1011
body: I think this should be `after`
created_at: 2025-05-07T15:04:21Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,336 @@
+use std::time::Duration;
+
+use crate::clickhouse::migration_manager::migration_trait::Migration;
+use crate::clickhouse::ClickHouseConnectionInfo;
+use crate::error::{Error, ErrorDetails};
+use async_trait::async_trait;
+
+use super::check_table_exists;
+
+/// NOTE: This migration supersedes migration_0023.
+/// Migration 0023 was inefficient due to its use of joins.
+/// In this migration, we create the same table StaticEvaluationHumanFeedback if it doesn't exist.
+/// We then drop the old materialized view and create new ones that are more efficient.
+/// This migration adds a table StaticEvaluationHumanFeedback that stores human feedback in an easy-to-reference format.
+/// This is technically an auxiliary table as the primary store is still the various feedback tables.
+/// We also create two materialized views that automatically write to StaticEvaluationHumanFeedback when
+/// FloatMetricFeedback and BooleanMetricFeedback are updated with new feedback that contains both
+/// tensorzero::datapoint_id and tensorzero::human_feedback tags.
+///
+/// In particular, since we don't care if there are duplicate rows and we need the migration to work with concurrent gateways,
+/// we do a full cutover to the new view, meaning that if we determine that this migration needs to run and we're not clean starting,
+/// we:
+///   1. Fix a timestamp for the migration
+///   2. Drop the old views if they exist
+///   3. Create the new views with a timestamp of the migration timestamp + 10 seconds
+///   4. Wait 10 seconds
+///   5. Insert the data from the missing 10 seconds into the new table.
+pub struct Migration0028<'a> {
+    pub clickhouse: &'a ClickHouseConnectionInfo,
+}
+
+const MIGRATION_ID: &str = "0028";
+
+#[async_trait]
+impl Migration for Migration0028<'_> {
+    async fn can_apply(&self) -> Result<(), Error> {
+        // As long as FloatMetricFeedback and BooleanMetricFeedback exist, we can apply this migration
+        if !check_table_exists(self.clickhouse, "FloatMetricFeedback", MIGRATION_ID).await? {
+            return Err(Error::new(ErrorDetails::ClickHouseMigration {
+                id: MIGRATION_ID.to_string(),
+                message: "FloatMetricFeedback table does not exist".to_string(),
+            }));
+        }
+        if !check_table_exists(self.clickhouse, "BooleanMetricFeedback", MIGRATION_ID).await? {
+            return Err(Error::new(ErrorDetails::ClickHouseMigration {
+                id: MIGRATION_ID.to_string(),
+                message: "BooleanMetricFeedback table does not exist".to_string(),
+            }));
+        }
+        Ok(())
+    }
+
+    async fn should_apply(&self) -> Result<bool, Error> {
+        let human_feedback_table_exists = check_table_exists(
+            self.clickhouse,
+            "StaticEvaluationHumanFeedback",
+            MIGRATION_ID,
+        )
+        .await?;
+        let materialized_view_exists = check_table_exists(
+            self.clickhouse,
+            "StaticEvaluationHumanFeedbackView",
+            MIGRATION_ID,
+        )
+        .await?;
+        Ok(!human_feedback_table_exists || !materialized_view_exists)
+    }
+
+    async fn apply(&self, clean_start: bool) -> Result<(), Error> {
+        let view_offset = Duration::from_secs(10);
+        let current_time = std::time::SystemTime::now();
+        let view_timestamp = (current_time
+            .duration_since(std::time::UNIX_EPOCH)
+            .map_err(|e| {
+                Error::new(ErrorDetails::ClickHouseMigration {
+                    id: MIGRATION_ID.to_string(),
+                    message: e.to_string(),
+                })
+            })?
+            + view_offset)
+            .as_secs();
+        let view_timestamp_where_clause = if !clean_start {
+            format!("AND UUIDv7ToDateTime(feedback_id) >= toDateTime(toUnixTimestamp({view_timestamp}))")
+        } else {
+            String::new()
+        };
+        self.clickhouse
+            .run_query_synchronous(
+                r#"CREATE TABLE IF NOT EXISTS StaticEvaluationHumanFeedback (
+                    metric_name LowCardinality(String),
+                    datapoint_id UUID,
+                    output String,
+                    value String,  -- JSON encoded value of the feedback
+                    feedback_id UUID,
+                    evaluator_inference_id UUID,
+                    timestamp DateTime MATERIALIZED UUIDv7ToDateTime(feedback_id)
+                ) ENGINE = MergeTree()
+                ORDER BY (metric_name, datapoint_id, output)
+                SETTINGS index_granularity = 256 -- We use a small index granularity to improve lookup performance
+            "#.to_string(),
+                None,
+            )
+            .await?;
+
+        // Since there cannot have been any StaticEvaluationHumanFeedback rows before this migration runs,
+        // we can just create the materialized views in place.
+
+        // Create the materialized view for FloatMetricFeedback
+        let query = format!(
+            r#"
+            CREATE MATERIALIZED VIEW IF NOT EXISTS StaticEvaluationHumanFeedbackView
+            TO StaticEvaluationHumanFeedback
+            AS
+                WITH human_feedback AS (
path: tensorzero-internal/src/clickhouse/migration_manager/migrations/migration_0028.rs
user: login: virajmehta
body: TODO: break this into 2 materialized views so they get triggered on writes to Float and BooleanMetricFeedback
created_at: 2025-05-09T13:15:10Z
author_association: MEMBER

PR comments:
user: login: github-actions[bot]
created_at: 2025-05-02T18:43:26Z
body: All contributors have signed the CLA  ✍️ ✅<br/><sub>Posted by the ****CLA Assistant Lite bot****.</sub>
author_association: CONTRIBUTOR
user: login: Aaron1011
created_at: 2025-05-02T18:45:49Z
body: I have read the Contributor License Agreement (CLA) and hereby sign the CLA.
author_association: MEMBER
user: login: virajmehta
created_at: 2025-05-05T18:17:22Z
body: @Aaron1011 I merged main (which fixed one of the bad queries), ran the unit tests, and with the exception of the ones that use pagination it seems like they're all quick even with the larger fixtures. 

Let's chat tomorrow about what we should do before merging but I'm generally happy with progress here and that there may not be too many dragons in our existing UI queries.

We should also add fixtures for JsonInference, the datasets, evals, etc to make sure that all those queries don't suck either.
author_association: MEMBER