README.md

deplot_kr

deplot_kr is a Image-to-Data(Text) model based on the google's pix2struct architecture. It was fine-tuned from DePlot, using korean chart image-text pairs.

deplot_kr은 google의 pix2struct 구조를 기반으로 한 한국어 image-to-data(텍스트 형태의 데이터 테이블) 모델입니다. DePlot 모델을 한국어 차트 이미지-텍스트 쌍 데이터세트(30만 개)를 이용하여 fine-tuning 했습니다.

How to use

You can run a prediction by input an image.
Model predict the data table of text form in the image.

이미지를 모델에 입력하면 모델은 이미지로부터 표 형태의 데이터 테이블을 예측합니다.

from transformers import Pix2StructForConditionalGeneration, Pix2StructImageProcessor, AutoTokenizer, Pix2StructProcessor
from PIL import Image

image_processor = Pix2StructImageProcessor()
tokenizer = AutoTokenizer.from_pretrained("brainventures/deplot_kr")
processor = Pix2StructProcessor(image_processor=image_processor, tokenizer=tokenizer)

model = Pix2StructForConditionalGeneration.from_pretrained("brainventures/deplot_kr")

image_path = "IMAGE_PATH"
image = Image.open(image_path)

inputs = processor(images=image, return_tensors="pt")
pred = model.generate(flattened_patches=flattened_patches, attention_mask=attention_mask, max_length=1024)
print(processor.batch_decode(deplot_generated_ids, skip_special_token=True)[0])

Preprocessing

According to Liu et al.(2023)...

• markdown format
• | : seperating cells (열 구분)
• \n : seperating rows (행 구분)

Train

The model was trained in a TPU environment.

• num_warmup_steps : 1,000
• num_training_steps : 40,000

Evaluation Results

This model achieves the following results:

metrics name	%
RNSS (Relative Number Set Similarity)	99.5483
RMS F1 (Relative Mapping Similarity)	16.6401

Contact

For questions and comments, please use the discussion tab or email gloria@brainventur.com