ChartReformer: Natural Language-Driven Chart Image Editing

This is the official model release of ChartReformer from 2024. The paper was published at ICDAR 2024 (arXiv: March 2024). The weights here are the original checkpoints from the 2024 paper. They were uploaded to Hugging Face in October 2026 only to make them easier to download. Before that, they were available only through Google Drive links on GitHub. This is not a new model, a new version, or new work.

• 📄 Paper: ChartReformer: Natural Language-Driven Chart Image Editing (ICDAR 2024) · arXiv:2403.00209
• 💻 Code: github.com/pengyu965/ChartReformer
• 🤖 Model: Pengyu965/ChartReformer (this repo)
• 📚 Dataset: Pengyu965/ChartCraft

Pengyu Yan, Mahesh Bhosale, Jay Lal, Bikhyat Adhikari, David Doermann. University at Buffalo.

Overview

Most charts we meet are just images: in papers, reports, slides, and on the web. Their data tables and plotting code are usually gone. So changing a chart's colors, turning a line chart into a bar chart, or adding a data series used to mean rebuilding the chart from scratch.

ChartReformer is the first work to systematically study chart image editing. It edits a chart image directly from a natural-language instruction, with no access to the original data or plotting code. The paper:

• 🥇 Defines and standardizes the chart-editing task for the first time, with a taxonomy of edits: style, layout, format, and data-centric.
• 📚 Introduces ChartCraft, a chart-editing dataset of about 100K pairs. Each pair has the original and edited chart images, their data tables, their visual attributes, and natural-language instructions. To our knowledge, no public chart-editing dataset existed before it.
• 📏 Proposes evaluation metrics for chart editing, including the Visual Attribute Edit Score (VAES), and uses them together with RMS on data tables and SSIM on the re-rendered images.
• 🛠️ Presents the ChartReformer model, which edits charts by predicting a structured description of the edited chart instead of plotting code. A replotter then renders that description, which makes the results precise and stable.

How it works

The model takes a chart image and an edit instruction. It outputs a JSON description of the edited chart: the underlying data table (plus title and axis titles) and the full set of visual attributes (chart type, colors, line styles, markers, hatches, fonts, legend, grid, and so on). A deterministic Matplotlib replotter turns this JSON into the edited image.

Earlier approaches generate plotting code, which often fails to compile. ChartReformer predicts parameters instead, so almost all of its outputs render (99.9% in the paper).

Edit category	Aspect	Supported edits
Style	Colors / Line / Pattern / Text	plot color · line style, marker · bar hatch pattern · font family, font size
Layout	Axes / Legend	grid lines · legend position and internal layout
Format	Plot type	line ↔ grouped bar · line ↔ stacked bar · grouped bar ↔ stacked bar
Data-centric	Filtering / Addition	range- and series-based filtering · add or remove data points and data series

Checkpoints in this repository

Both checkpoints use the Pix2Struct architecture (about 282M parameters), initialized from DePlot. They were trained in two stages:

Path	Training stage	Task	Text prompt
/ (root, default)	Stage 2: chart-editing fine-tuning (best checkpoint, step 20K)	Chart editing	the edit instruction, e.g. "Add grid lines to the plot"
pretrain/	Stage 1: chart de-rendering pre-training (best checkpoint, step 48K)	Chart → data + visual attributes	"Generate underlying data and visual attributes of the chart images below:"

Input charts are resized to fit 800×800 with white padding (aspect ratio kept). The model uses up to 512 image patches and generates up to 1024 tokens. The models were trained with transformers==4.37.2. The code below was also tested with transformers==5.6.0.

Quick start

Edit a chart

import json, torch
from PIL import Image
from transformers import AutoProcessor, Pix2StructForConditionalGeneration

def resize_and_pad(image, target_size=(800, 800), background_color=(255, 255, 255)):
    """Keep the aspect ratio and pad to 800x800, as done during training."""
    w, h = image.size
    ratio = min(target_size[0] / w, target_size[1] / h)
    new_size = (int(w * ratio), int(h * ratio))
    canvas = Image.new("RGB", target_size, background_color)
    canvas.paste(image.resize(new_size), ((target_size[0] - new_size[0]) // 2, (target_size[1] - new_size[1]) // 2))
    return canvas

device = "cuda" if torch.cuda.is_available() else "cpu"
processor = AutoProcessor.from_pretrained("Pengyu965/ChartReformer")
model = Pix2StructForConditionalGeneration.from_pretrained("Pengyu965/ChartReformer").to(device).eval()

chart = resize_and_pad(Image.open("chart.png").convert("RGB"))
instruction = "Change the line chart into a grouped vertical bar chart"

inputs = processor(images=chart, text=instruction, return_tensors="pt", max_patches=512).to(device)
with torch.no_grad():
    output_ids = model.generate(**inputs, max_new_tokens=1024)
prediction = processor.batch_decode(output_ids, skip_special_tokens=True)[0]

edited = json.loads(prediction)  # tip: json_repair.loads() is more forgiving
print(edited["global_properties"]["chart_type"])
print(edited["data_table"])

If you don't have a chart at hand, try one from the test split of ChartCraft:

from datasets import load_dataset

sample = next(iter(load_dataset("Pengyu965/ChartCraft", split="test", streaming=True)))
chart = resize_and_pad(sample["original_image"].convert("RGB"))
instruction = sample["editing_prompts"][0]

Render the edited chart

To turn the predicted JSON back into an image, use the replotter Chart_Reploter.py. It's included in this repo and is the same file as in the GitHub repo. It draws all five chart types: line, plus grouped and stacked bar charts in vertical and horizontal orientation.

# pip install pandas matplotlib
import os, sys
from huggingface_hub import hf_hub_download

sys.path.insert(0, os.path.dirname(hf_hub_download("Pengyu965/ChartReformer", "Chart_Reploter.py")))
from Chart_Reploter import Replot

Replot(dict(edited)).fig.savefig("edited_chart.png")  # Replot consumes the dict it is given, so pass a copy

De-render a chart (stage-1 checkpoint)

The pretrain/ checkpoint extracts the data table and visual attributes from a chart image:

processor = AutoProcessor.from_pretrained("Pengyu965/ChartReformer", subfolder="pretrain")
model = Pix2StructForConditionalGeneration.from_pretrained("Pengyu965/ChartReformer", subfolder="pretrain").to(device).eval()

prompt = "Generate underlying data and visual attributes of the chart images below:"
inputs = processor(images=chart, text=prompt, return_tensors="pt", max_patches=512).to(device)
with torch.no_grad():
    output_ids = model.generate(**inputs, max_new_tokens=1024)
derendered = json.loads(processor.batch_decode(output_ids, skip_special_tokens=True)[0])

Output format

Both checkpoints output a single JSON object. The data comes first, followed by the visual attributes. Table rows are separated by <0x0A> and cells by |:

{
  "data_table": "Country Name | 2004 | 1997 <0x0A> OECD members | 23.66 | 21.08 <0x0A> South Asia | 20.33 | 14.15 <0x0A> ",
  "chart_title": "...", "x_axis_title": "Country Name", "y_axis_title": "Values",
  "global_properties": {
    "chart_type": "line",
    "x_label_params": {"fontname": "Serif", "fontsize": "x-large"},
    "legend_params": {"loc": 1, "ncol": 1},
    "grid_params": {"visible": true, "axis": "y", "linestyle": "dashed"},
    "...": "..."
  },
  "line_properties": {"linestyles": ["dashdot", "solid"], "markers": ["*", "s"], "colors": ["k", "r"]}
}

Bar charts use bar_properties (colors, hatches) instead of line_properties.

Results (from the ICDAR 2024 paper)

ChartReformer on the ChartCraft test set (5.5K samples). VAES and RMS are F1 scores for the predicted visual attributes and data table. SSIM compares the re-rendered chart with the ground-truth edited chart. Success rate is the share of predictions that render successfully.

Metric	Style	Layout	Format	Data-centric	Total
VAES (F1)	86.63	89.65	90.53	84.24	86.32
RMS (F1)	90.35	89.23	91.22	88.36	89.33
SSIM	84.50	82.06	85.19	83.07	83.65
Success rate	99.81	100	99.90	99.77	99.90

Comparison with ChartLlama on a 10% test subset, shown as SSIM / success rate:

Method	Style	Layout	Format	Data-centric	Total
ChartLlama	73.09 / 10.63	64.77 / 26.47	64.32 / 8.04	67.68 / 19.33	67.46 / 16.95
ChartReformer	83.30 / 100	82.55 / 100	84.22 / 100	81.43 / 100	82.39 / 100

Intended use and limitations

• Domain: The model was trained on synthetic Matplotlib charts of five types: line charts, plus grouped and stacked bar charts in vertical and horizontal orientation. Charts with very different styles may be out of distribution.
• Data accuracy: Editing quality depends on how accurately the data is extracted. Values read from the image are close but not always exact.
• Instructions: Each instruction should describe one edit from the taxonomy above. For complex or abstract requests (e.g., "make it colorblind-friendly"), split them into a chain of simple edits.
• Output: Generated JSON can occasionally be malformed. json_repair fixes most of these cases. The paper reports results without any repair.

Citation

If you find ChartReformer or ChartCraft useful, please cite:

@inproceedings{yan2024chartreformer,
  title={ChartReformer: Natural Language-Driven Chart Image Editing},
  author={Yan, Pengyu and Bhosale, Mahesh and Lal, Jay and Adhikari, Bikhyat and Doermann, David},
  booktitle={International Conference on Document Analysis and Recognition},
  pages={453--469},
  year={2024},
  organization={Springer}
}

License

Released under the MIT License, the same as the GitHub repository. The base model, DePlot, is released under Apache-2.0.