Pengyu965/ChartDete

CACHED: Context-Aware Chart Element Detection (ICDAR 2023)

This is the official model release of CACHED from 2023. The paper was published at ICDAR 2023 (arXiv: May 2023). The weights here are the original checkpoints from the 2023 paper, unchanged. They were uploaded to Hugging Face in October 2026 only to make them easier to download. Before that, they were available only through Google Drive links on GitHub. This is not a new model, a new version, or new work.

• 📄 Paper: Context-Aware Chart Element Detection (ICDAR 2023) · arXiv:2305.04151
• 💻 Code: github.com/pengyu965/ChartDete
• 🤖 Model: Pengyu965/ChartDete (this repo)
• 📚 Dataset: Pengyu965/ChartDete-PMC2022

Pengyu Yan*, Saleem Ahmed*, David Doermann (*equal contribution). University at Buffalo.

Overview

Detecting the basic elements of a chart (axis titles, tick labels, tick marks, legend entries, and so on) is the first step of chart data extraction. Unlike objects in natural images, many chart elements look alike and differ only in their role. A text block can be a tick label, a legend label, or a value label depending on where it sits and what surrounds it. Chart element detection therefore depends heavily on context.

CACHED (Context-Aware CHart Element Detection) adds a local-global context fusion module between the RoI heads of a Cascade R-CNN with a Swin-Transformer backbone:

• Visual context enhancement (VCE) pools the whole feature pyramid into a global feature map and fuses it with each RoI feature (with an SE block), so every region sees the full chart.
• Positional context encoding (PCE) runs a Transformer over the normalized box coordinates of all proposals, so the model learns the layout patterns of charts (for example, x tick labels sit below the x tick marks).
• Focal loss and balanced L1 loss handle the strong class imbalance of chart elements.

The paper also refines the chart element categories into 18 classes and releases an updated PMC dataset in COCO format: Pengyu965/ChartDete-PMC2022.

Checkpoints in this repository

All checkpoints are the original MMDetection 2.x files from the paper. Each folder holds a config and its checkpoint, in the same layout as work_dirs/ in the GitHub repo. The .pth files also contain the optimizer state, which is why they are large.

Folder	File	Model	Classes	Test AP (paper)	Test AP (re-evaluated)
cascade_rcnn_swin-t_fpn_LGF_VCE_PCE_coco_focalsmoothloss/	checkpoint.pth	CACHED (Cascade R-CNN, Swin-T FPN, VCE + PCE, focal + balanced L1 loss)	18	0.729	0.729
cascade_rcnn_swin-t_fpn_LGF_VCE_PCE_coco_focalsmoothloss/	bar_checkpoint.pth	CACHED, fine-tuned for bar detection	1 (bar)	see below	–
cascade_rcnn_swin-t-p4-w7_fpn_1x_coco/	checkpoint.pth	Cascade R-CNN, Swin-T FPN (baseline)	18	0.699	0.699
cascade_rcnn_x101_64x4d_fpn_20e_coco/	checkpoint.pth	Cascade R-CNN, ResNeXt-101 FPN (baseline)	18	0.696	0.696
faster_rcnn_x101_64x4d_fpn_2x_coco/	checkpoint.pth	Faster R-CNN, ResNeXt-101 FPN (baseline)	18	0.665	0.663
detr_finetune/	checkpoint.pth	DETR (baseline)	18	0.536	0.536

"Test AP" is COCO box AP on the test split (ICPR 2022 PMC split 3, 381 images) of the refined PMC dataset. The "re-evaluated" column was measured in October 2026 with the files in this repo and tools/test.py (commands below), to confirm that the uploaded checkpoints reproduce the paper. The intermediate ablation models (PCE only, VCE + PCE without the loss changes) were not released.

The bar detector was fine-tuned from CACHED on the bar-chart subset of PMC. The paper reports it with the chart competition metric (table below). For reference, its COCO box AP on baronly_test.json is 0.692.

Classes

The 18-class model detects, in this order (index = position in the model output):

#	Class	#	Class	#	Class
0	x_title (x-axis title)	6	chart_title	12	legend_area
1	y_title (y-axis title)	7	x_tick (x tick mark)	13	mark_label
2	plot_area	8	y_tick (y tick mark)	14	value_label
3	other	9	legend_patch (legend marker)	15	y_axis_area
4	xlabel (x tick label)	10	legend_label	16	x_axis_area
5	ylabel (y tick label)	11	legend_title	17	tick_grouping

See the dataset card for definitions.

Quick start

These checkpoints need the ChartDete code. It is a fork of MMDetection 2.28 that implements the CACHED modules (CascadeRoIHead_LGF, LocalGlobal_Context_Fuser, BboxEncoder, ...). They cannot be loaded with transformers, upstream MMDetection, or MMDetection 3.x. Hugging Face only hosts the files.

1. Install

The models were built and tested with PyTorch 1.13.1 and CUDA 11.7:

conda create -n ChartDete python=3.8 -y
conda activate ChartDete

pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 --extra-index-url https://download.pytorch.org/whl/cu117
pip install mmcv-full==1.7.2 -f https://download.openmmlab.com/mmcv/dist/cu117/torch1.13/index.html

git clone https://github.com/pengyu965/ChartDete
cd ChartDete
pip install -v -e .
pip install huggingface_hub

Use mmcv-full 1.x (here 1.7.2). The code is not compatible with mmcv 2.x.

2. Detect chart elements

from huggingface_hub import hf_hub_download
from mmdet.apis import init_detector, inference_detector

repo_id = "Pengyu965/ChartDete"
name = "cascade_rcnn_swin-t_fpn_LGF_VCE_PCE_coco_focalsmoothloss"

config = hf_hub_download(repo_id, f"{name}/{name}.py")
checkpoint = hf_hub_download(repo_id, f"{name}/checkpoint.pth")
model = init_detector(config, checkpoint, device="cuda:0")  # or device="cpu"

image = hf_hub_download(repo_id, "assets/sample.jpg")  # or the path to your own chart image
result = inference_detector(model, image)  # one (N, 5) array per class: [x1, y1, x2, y2, score]

for class_name, dets in zip(model.CLASSES, result):
    for x1, y1, x2, y2, score in dets[dets[:, 4] > 0.3]:
        print(f"{class_name:14s} {score:.2f}  [{x1:.0f}, {y1:.0f}, {x2:.0f}, {y2:.0f}]")

model.show_result(image, result, score_thr=0.3, out_file="sample_result.jpg")

3. Detect bars

The bar detector lives in the same folder and outputs a single class, bar:

config = hf_hub_download(repo_id, f"{name}/{name}_bar.py")
checkpoint = hf_hub_download(repo_id, f"{name}/bar_checkpoint.pth")
bar_model = init_detector(config, checkpoint, device="cuda:0")

bars = inference_detector(bar_model, image)[0]  # (N, 5): [x1, y1, x2, y2, score]
print(len(bars[bars[:, 4] > 0.3]), "bars")

4. Use the checkpoints with the repo scripts

To place the checkpoints where the repo configs and scripts expect them (work_dirs/), run this from the repo root:

# CACHED only (element + bar detector, about 3.1 GB)
hf download Pengyu965/ChartDete --include "cascade_rcnn_swin-t_fpn_LGF_VCE_PCE_coco_focalsmoothloss/*" --local-dir work_dirs

# or all checkpoints, including the baselines (about 6.3 GB)
hf download Pengyu965/ChartDete --include "*/*.py" "*/*.pth" --local-dir work_dirs

Then download the test data from Pengyu965/ChartDete-PMC2022 and evaluate. This reproduces the AP in the table above:

hf download Pengyu965/ChartDete-PMC2022 pmc_coco.zip --repo-type dataset --local-dir data/pmc_2022
unzip -oq data/pmc_2022/pmc_coco.zip -d data/pmc_2022/

python tools/test.py \
    work_dirs/cascade_rcnn_swin-t_fpn_LGF_VCE_PCE_coco_focalsmoothloss/cascade_rcnn_swin-t_fpn_LGF_VCE_PCE_coco_focalsmoothloss.py \
    work_dirs/cascade_rcnn_swin-t_fpn_LGF_VCE_PCE_coco_focalsmoothloss/checkpoint.pth --eval bbox

Training follows the usual MMDetection routine (tools/train.py or tools/dist_train.sh with a config from configs/). See the GitHub repo for details.

Results (from the ICDAR 2023 paper)

COCO evaluation on the refined PMC dataset (test split). The first four rows are standard detectors trained by the authors. The last three rows are the CACHED ablation, and the last row is the released CACHED checkpoint.

Model	Context module	Backbone	AP	AP50	AP75	APS	APM	APL
DETR	–	–	0.536	0.762	0.572	0.384	0.555	0.809
Faster R-CNN	–	ResNeXt101-FPN	0.665	0.815	0.737	0.557	0.697	0.827
Cascade R-CNN	–	ResNeXt101-FPN	0.696	0.825	0.759	0.579	0.725	0.899
Cascade R-CNN	–	SwinT-FPN	0.699	0.838	0.772	0.589	0.732	0.885
Cascade R-CNN	PCE	SwinT-FPN	0.708	0.842	0.775	0.591	0.742	0.903
Cascade R-CNN	VCE + PCE	SwinT-FPN	0.713	0.851	0.786	0.597	0.741	0.909
CACHED (Cascade R-CNNL)	VCE + PCE	SwinT-FPN	0.729	0.845	0.790	0.602	0.763	0.939

^L with focal loss and balanced L1 loss.

ICPR 2022 CHART-Infographics competition (original PMC categories; CACHED predictions mapped back to the competition format):

Method	Task 2: text detection (avg. IoU)	Task 3: text role classification (F-measure)
six_seven_four	0.435	–
IIIT_CVIT_Chart_Understanding	0.790	0.821
Ystar	0.810	–
UB-ChartAnalysis	0.820	0.736
CACHED	0.869	0.787

Bar detection on the PMC test set (F-measure at different IoU thresholds, and the chart competition score). The baseline numbers are from Ma et al. (2021).

Model	IoU=0.5	IoU=0.7	IoU=0.9	Score_a
Faster R-CNN + FPN	85.81	78.05	31.30	89.65
Cascade R-CNN + FPN	86.92	83.53	55.32	91.76
CACHED (bar)	89.30	88.73	76.94	93.75

Limitations

• The models were trained on real-world charts from scientific articles (PubMed Central). Accuracy may drop on other chart styles, such as infographics, dashboards, or charts with attached tables.
• The element detector finds chart elements outside the data marks. It does not detect lines, points, or other data marks, and it does not read text (no OCR). Only the bar detector detects data marks, and only bars.
• The code requires MMDetection 2.x with mmcv-full 1.x.

License

The code and checkpoints are released under the MIT license, the same as the GitHub repo. The models were trained on the PMC dataset of the ICPR 2022 CHART-Infographics competition, which is licensed under CC BY-NC-SA 4.0.

Citation

@inproceedings{yan2023cached,
  title     = {Context-Aware Chart Element Detection},
  author    = {Yan, Pengyu and Ahmed, Saleem and Doermann, David},
  booktitle = {Document Analysis and Recognition - ICDAR 2023},
  pages     = {218--233},
  year      = {2023},
  publisher = {Springer Nature Switzerland},
  doi       = {10.1007/978-3-031-41676-7_13}
}