DOI: 10.5281/zenodo.16889941
title: LLMShot: Reducing snapshot testing maintenance via LLMs

description: LLMShot

A tool for analyzing UI snapshot differences using LLMs. This project helps identify visual differences between expected and actual UI screenshots, categorize the differences, simulate ignoring them, and generate comprehensive analysis reports.

Accepted Paper https://arxiv.org/abs/2507.10062:

@misc{kaynak2025llmshotreducingsnapshottesting,
      title={LLMShot: Reducing snapshot testing maintenance via LLMs}, 
      author={Ergün Batuhan Kaynak and Mayasah Lami and Sahand Moslemi and Anil Koyuncu},
      year={2025},
      eprint={2507.10062},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2507.10062}, 
} 


Requirements



Python 3.8+

Pillow

NumPy

colorama

Ollama (with gemma3:4b and gemma3:12b models installed)

Xcode 16.3


Requirements Installation





Install required Python packages:

pip install numpy pillow colorama





Install Ollama:





Follow the instructions at ollama.ai to install Ollama




Pull the required models:

For "gemma3:4b":

ollama pull gemma3:4b


For "gemma3:12b":

ollama pull gemma3:12b






General Content

The xcode_project/ directory contains all Xcode-related files including the iOS app and tests for dataset creation. There are also several scripts provided to recreate our experiments:



generate_dataset.sh: Converts Xcode-generated snapshots to a dataset with the expected format and outputs to the dataset/ directory.

process_snapshots.py: Main experiment script that processes data from the dataset/ directory. Runs LLMs (which may take a long time) and outputs results to the analysis_results/ directory.

generate_report.py: Creates an HTML report for analysis results, see Reported Features section. Output results to reports/ directory.


Detailed instructions are provided below for dataset creation, running experiments, and generating reports.

Snapshotting and Dataset Creation

A readily available dataset is given in the dataset/ directory. You can create your own dataset via:



Create a dataset directory in the project root

Execute tests in Xcode

Run ./generate_dataset.sh

Each test directory should contain:



reference.png: Expected UI state

failure.png: Current UI state with potential issues

diff.png: Visualization of differences

metadata.json: Test details and categories




Xcode project was tested with Xcode 16.3.

Extending The Dataset

Our testing framework utilizes XCTest and XCUITest, Apple's native UI testing tools, for simulating UI actions. We also extend the base XCUIElement and XCUIApplication behaviour to allow for commonly used snpshot test actions. These include navigation actions (like vertical scrolling), handling time‑dependent events (such as waiting for loading or obscuring elements, or navigation transitions), and managing launch arguments.

Each test is configured to run the app in the default configuration, then relaunch the app with set of feature flags. Here, both setupAction() and snapshotAction() functions are the same for both test runs, and they are overridden by each individual test to define their own setup and sequence of UI actions.

To extend the dataset, one can



Attach labels to the views using .accessibilityIdentifier(), which can then be interacted with forceTapElement(\_:).

Create new test classses and cases that override setupAction() and snapshotAction() functions to define their own set of UI actions.

Define their own set of test flags in FeatureFlags.swift and subsequently write the logic change in the main program. These flags are then passed during test cases for use during automated snapshot generation.

Run generate_dataset.sh to convert created snapshots to the expected structure for analysis. This script looks for snapshots inside the snapshot save directory (by default /SnapshotInstructorUITests/__Snapshots__/BaseUITest) to find pairs of reference and failure snapshots, then creates their diff.


Please see the existing test code and dataset for examples.

Experiment Execution

We provide a CLI interface for recreating our experiments. To run the tool with no arguments and use the interactive menu:

python process_snapshots.py


This will present a menu with these options:



Select Model: Choose between 4b or 12b models

Run Standard Analysis: Analyze all snapshots and identify differences

Run 'Ignore Reason' Analysis (From Analysis): Ignore the main difference found in the standard analysis

Run 'Ignore Reason' Analysis (From Metadata): Ignore the first category from metadata

Run 'Analyze and Ignore' Analysis: Detect and ignore the main difference in a single step


To run all analyses with both models in batch mode:

python process_snapshots.py --all


This will:



Run all four analysis modes with the 4b model

Run all four analysis modes with the 12b model

Generate a comprehensive report


Report Generation

After running analyses, generate a visual HTML report via:

python generate_report.py


The report will be created in the reports/snapshot_report_<timestamp>.html file and automatically opened in your default browser.

The reported features are as follows:



Metrics Dashboard: See aggregated accuracy and performance metrics across all analysis modes

Test Case Browser: View individual test cases with images and detailed analysis

Visual Comparison: Compare reference and failure images with highlighted differences