WhatDoWeSee-EfficientNet-B0

Tags what is visible in an image with any number of 20 attributes, such as people, text, chart, diagram, table, map or aerial view. A small, fast EfficientNet-B0 model (17 MB ONNX, about 10 ms per image on one CPU thread) for sorting and describing large image collections.

This is v0.3.0 (2026-10-15): trained on 38,386 images where three Albert vision-language models agree. Try it in your browser: WhatDoWeSee Space.

Attributes

An image gets every attribute that is clearly recognisable, also when blurry or in the background; tiny or unrecognisable details don't count. Depicted things count (a drawn dog is Animals).

• People (people): one or more humans, or clearly human faces, hands or bodies
• Animals (animals): animals, birds, fish or insects
• Plants (nature): living green things: trees, plants, flowers, crops; not grass or a lawn as ground, not sea, sky or rocks (those are Landscape)
• Landscape (landscape): an outdoor view of land, sea, sky, mountains, rocks, rivers or a city as a whole
• Buildings / structures (buildings): buildings, interiors where the room or its architecture is clearly visible, and other built structures: bridges, towers, dams, monuments, ruins
• Vehicles (vehicles): cars, bikes, trains, ships, planes or other vehicles
• Food (food): food or drinks
• Text (text): running text, sentences, lists or a page that make up a substantial part of the image; labels, captions, titles and signs don't count
• Handwriting (handwriting): handwritten text, notes or signatures
• Chart (chart): a data plot: values on axes or as proportions (bar, line, pie, scatter, area chart or similar)
• Diagram (diagram): structure or relationships without measured values: a flowchart, schematic, process or relationship diagram; also labelled drawings that show how something is built (anatomy, parts, cross-sections)
• Table (table): text or numbers laid out in rows and columns, with or without grid lines; not the furniture
• Map (map): a geographic map, floor plan or route map
• Logo / icon (logo-icon): a logo, icon, emblem, badge or pictogram
• Formula (formula): mathematical, chemical or other notation
• Screen / UI (screen): an app, website or device interface
• Artwork (artwork): made as art: a painting, illustration, cartoon or sculpture, as the image itself or shown in it; not drawings made to explain (those are Diagram)
• Instruments (instruments): musical instruments: guitar, piano, violin, drums, wind instruments and the like
• Furniture (furniture): chairs, tables, beds, sofas, desks, cabinets, shelves and other furniture (a data table is Table)
• Aerial (aerial): the ground seen from high above: aerial or satellite photos, drone shots, bird's-eye views, also through a plane window, straight down or at an angle; clouds without ground don't count; a drawn map is Map

Usage

Download a fixed version (recommended), so a later release never changes your results:

from huggingface_hub import hf_hub_download

model_path = hf_hub_download("StephanMeijer/WhatDoWeSee-EfficientNet-B0", "model.onnx", revision="v0.3.0")
config_path = hf_hub_download("StephanMeijer/WhatDoWeSee-EfficientNet-B0", "config.json", revision="v0.3.0")

Then:

import json
import numpy as np
import onnxruntime as ort
from PIL import Image

config = json.load(open(config_path))
session = ort.InferenceSession(model_path)
size = config["input"]["shape"][-1]
mean = np.array(config["input"]["mean"], dtype=np.float32)
std = np.array(config["input"]["std"], dtype=np.float32)

image = Image.open("photo.jpg").convert("RGB").resize((size, size), Image.BILINEAR)
x = ((np.asarray(image, dtype=np.float32) / 255 - mean) / std).transpose(2, 0, 1)[None]
probabilities = 1 / (1 + np.exp(-session.run(None, {"pixel_values": x})[0][0]))
present = [label for label, p in zip(config["labels"], probabilities)
           if p >= config["thresholds"][label]]

Preprocessing: RGB, resized to 224×224 (antialiased bilinear, no crop, aspect ratio not kept), scaled to [0, 1], normalised with the ImageNet mean and standard deviation, NCHW float32. Each label has its own threshold in config.json, tuned on a validation split.

Evaluation

On 39 held-out hand-labelled images: mean average precision 0.93, macro F1 0.82 at the tuned thresholds. The set is small: per-label scores are indicative only.

label	name	positives	AP	F1
people	People	3	0.92	0.86
animals	Animals	4	1.00	1.00
nature	Plants	0	-	0.00
landscape	Landscape	1	1.00	0.50
buildings	Buildings / structures	0	-	0.00
vehicles	Vehicles	1	1.00	1.00
food	Food	1	1.00	1.00
text	Text	17	0.99	0.94
handwriting	Handwriting	5	0.91	0.73
chart	Chart	5	1.00	1.00
diagram	Diagram	3	0.92	0.80
table	Table	8	0.97	0.55
map	Map	2	1.00	1.00
logo-icon	Logo / icon	3	0.60	0.50
formula	Formula	0	-	0.00
screen	Screen / UI	2	1.00	1.00
artwork	Artwork	2	0.58	0.50
instruments	Instruments	0	-	0.00
furniture	Furniture	0	-	0.00
aerial	Aerial	1	1.00	1.00

How it compares

Against v0.2 on the same 39 independent hand-labelled images: much better yes/no decisions (macro F1 0.82 against 0.66 at each version's thresholds) and slightly better ranking (mean AP 0.93 against 0.90). Clearly better on Handwriting (AP 0.91 against 0.60), Table (0.97 / 0.85) and People (0.92 / 0.53). Weaker on Logo / icon (0.60 against 1.00). The thresholds are now tuned on 8,104 validation images instead of 95, so they are far more stable. On an earlier set of 33 independent images mean AP was lower than v0.2 (0.86 against 0.92) while macro F1 was higher (0.80 against 0.72): treat ranking as on par with v0.2. Plants, Buildings, Formula, Instruments and Furniture have no test image in the 39; per-label scores rest on 1 to 17 images: indicative only.

Training

• Backbone: efficientnet_b0.ra_in1k (ImageNet-pretrained, via timm), with a new 20-way sigmoid head; 30 epochs, AdamW, binary cross-entropy per label.
• Data: Three vision-language models on the Albert API (Gemma 4 31B, Ministral 3 8B, Mistral Small 3.2 24B) answered the attribute question for about 81,000 images; an attribute counts only when all three picked it. Each image counts under its rarest attribute, at most 2,000 per attribute were kept for training (38,386 images); the 645 hand-labelled images are in the pool with their teacher labels like any other. Thresholds are tuned on the 8,104 teacher-labelled validation images. Test: 39 hand-labelled images that neither this version nor v0.2 trained on or tuned thresholds on. Same weights and thresholds as v0.3.0-rc.3.
• The images come from public collections: photo datasets, document and form scans, charts, diagrams, screenshots, maps, artwork and icons.

Limitations

• Small text-related details (a small table inside a document, a signature) are harder than whole-image subjects.
• The training images come from many public collections, some under research-only or non-commercial terms. Check whether that matters for your use.

Licence

The model weights and config.json are released under the Apache License 2.0, like the EfficientNet-B0 backbone they start from. The licence covers the model, not the images it was trained on.

Versions

Each version on hand-labelled test images it never trained on (mean AP and macro F1). The test sets differ between some versions, so compare versions through the notes of each version:

version	date	test images	mean AP	macro F1
v0.3.0	2026-10-15	39	0.93	0.82	trained on 38,386 images where three Albert vision-language models agree
v0.3.0-rc.3	2026-10-15	95	0.78	0.67	release candidate, trained on 38,386 images where all three Albert vision-language models agree
v0.3.0-rc.2	2026-10-15	38	0.81	0.75	release candidate, v0.3.0-rc.1 with stable thresholds
v0.3.0-rc.1	2026-10-15	38	0.81	0.77	release candidate, trained on 38,710 images labelled by three Albert vision-language models
v0.2	2026-10-14	95	0.84	0.67	trained on 20,551 images labelled by a SigLIP 2 probe plus the 444 hand labels
v0.1	2026-10-14	95	0.80	0.71	baseline, fine-tuned on 444 hand-labelled images