mSOP-765k: A Benchmark For Multi-Modal Structured Output Predictions
Authors: Bianca Lamm
Janis Keuper
Venue: Accepted by TMLR ()

Abstract: This paper introduces mSOP-765k, a large-scale benchmark for the evaluation of multi-modal Structured Output Prediction (mSOP) pipelines. Besides novel evaluation metrics, the benchmark provides combined training and test datasets with over 765,000 images taken from real-world product advertisements. Each of these images contains product visualizations, textual information like product name or brand, and numerical data such as product weight, price, and discount. All images are annotated with the corresponding structured information in form of dictionaries containing key-value pairs.
An initial baseline evaluation, including various LLMs and VLMs, as well as multi-modal RAG approaches, shows that the proposed benchmark provides a challenging problem which can not yet be solved completely by state-of-the-art mSOP methods. The benchmark and dataset are available under a creative-commons license: https://www.msop-765k.org/

Published in Transactions on Machine Learning Research (01/2026)
mSOP-765k: A Benchmark For Multi-Modal Structured Output Predictions
Bianca Lamm
Bionca. maano..Lamm@de.mmrrroooo
Markant Services International GmbH
Janis Keuper
keuper Eampo..@ ai
IMLA, Offenbury University and University of Mannheim
Reviewed on OpenReview: https: // openreview. net/ forum? id- H7eYL4yFZS
Abstract
This paper introduees mSOP-765k, large-scale benchmark for the evaluation of mulli- modal Structured Output Prediction (mSOP) pipelines. Besides novel evaluation metrics, the benchmark provides combined training and test datascts with over 765,000 images taken from real-world product advertisements. Each of these images contains product visualiza- tions, textual information like product name brand, and numerical data such as product weight, price, and discount. All images annotated with the corresponding structured information in form of dictionaries containing key-value pairs.
An initial baseline evaluat ion, including various LLMs and VLMs, as well as multi-modal RAG approaches, shows that the proposed benchmark provides a challenging problem which can not yet be solved completely by state-of-the-art mSOP methods. The benchmark and dataset are available under creative-commons license: https: mww.mmm50k-------- org/
Introduction
Large Language Models (LLMs) (Vaswani et al. ZZhoo et al.) have become indispensable components modern natural language processing, demonstrating remarkable capabilities across a wide range of tasks. More recently, Vision Language Models (VLMs) (Radford al. Zhang et al.p have extended this paradigm by enabling the processing and understanding of multi-modal inputs, effectively bridging the gap between visual and textual information. Although both LLMs and VLMs primarily generate responses in human- readable lext, numerous applications require outpuls in machine-readable formats. Addressing this need, recent advances have endowed LLMs and VLMs with the ability to produce Structured Output Predictions (SOPs) (Liu et al. c), allowing their predictions to be provided as well-defined and easily parseable data rep- resentations. This emerging capability opens new avenues for leveraging these models in complex workflows that require precise and unambiguous interpretation of generated content.
mSOP-765k multi-modal
int key; 1234
float key; : 78.9
string key: mSOP-765k multi-modal
1234
listkstrings key, sun, moon, star
78.9
Multi-Modal Input mSOP-765k dataset
Custom Pipeline including, e.g. LLM. VIT. VLM, RAG.
Structured Output Predictions
Figure 1: Illusrationn of the multi-modal Structured Output Prediction task performed with the mSOP-700 dataset. The proposed benchmark measures the ability of given processing pipeline to predict structured key-value pairs of textual, numerical, and list data.

---

Published in Transactions on Machine Learning Research (01/2026)
A comprehensive benchmark with clearly defined quality metrics for the systematic evaluation of multi-modal SOPs is essential to advance the evolving research and the practical application of multi-modal structured output prediction methods. However, to the best of our knowledge, so far no large-scale benchmark exists that thoroughly assesses these capabilitics across diverse structured prediction tasks on multiple modalitios. In order to fill this gap, we introduce mSOP-765k. Figure Figure[]jllustraan Tillustrates the evaluation of provided multi-modal structured output prediction pipelines, such as LLMs (Vaswani et al. Zhao al.p. VLMs Radford et al. Zhang et al., Vision Transformers (ViTs) (Dosovitskiy et al.p, or Retrieval Augmented Generation (RAG)
(aao et al.) Lamm & Keuper b) systems by the proposed benchmark.
An extensive baseline evaluation of current approaches shows, that while many methods are able to predict single key-value pairs with high accuracy, all tested algorithms essentially fail to predict consistent multi- value data structures in a more complex scenario.
The following key contributions are presented in this paper:
We introduce mSOP-765k. the first large scale benchmark for structured output predictions from multi-modal inputs. mSOP-7638 765k provides over 765 thousand images of product advertisements containing product visualizations, textual product descriptions and promotions, as well as numerical data like prices and discounts. Each image has been manually annotated with detailed key-value pairs provided in machine readable data-structure. The benchmark and dataset are available under creative-commons license.
The benehmark introduces novel evaluation metrics, allowing to measure the prediction quality of complex output structures.
A first baseline evaluation of a wide range of different approaches, including LLMs Vaswann el al.; Padiood et al. Zhang et al, ViTs (Dosovitskly et al.), or RAG (Gao et al.) Lamm b) systems on MSSOP-765K clearly shows the strong limitations of current approaches towards accurate predictions of complex structured output from multi-modal inputs.
Zhao al.), VLMs Radford
& KKuuper
2
Related Work
2.1 Creation of Structured Output Predictions
Most recent LLMs and VLMs are able to create a machine readable output from natural language or visual inputs (Wang et al.). Thoir implementations, as far as publicly known, are based on several different ap- proaches to generate and / or improve SOPs which have been proposed in literature.
Trugalbandara introduces Meaning Typed Prompting (MTP), which integrates the definitions of types, mean- ings, and abstractions in the prompting process. The authors evaluate the performance of their approach on the text based LLM Structured Output Benchmark (Leo). The paper also includes small case study that analyzes the generation of SOPs from single input image using GPT-4o-mini Trugalbandara)
The Generate and Organize (G80) method has been introduced by Li et al.) This approach improves the capability of LLMs to produce SOPs in a two step process which separates the content generation and struc- turing: first, the response is generated in natural language and then a second LLMs call is used to organize and format the received response into the intended structure. Li et al. focus their evaluation on the tasks of Named Entily Recognition and Relation Extraction, which exclusively process textual input. Consequently, the models investigated in this study are limited to LLMs.
Wang et al. present SLOT, abbreviation for Structured LLM Output Transformer. This approach converts the unstructured responses of LLM into structured formats by applying finc-tuned lightweight language model as a post-processing layer. The evaluation of SLOT in (Wang et al.) also exclusively investigates text-to-structure tasks.
Lamm Keuper (b) propose the Visual RAG Pipeline, which generates SOPs given multi-modal inputs comprising both image and text data. Their evaluation includes the analysis of various commercial models, specifically GPT-4o, GPT--o-mini, and Cemini-2.0-flash
2

---

Published in Transactions on Machine Learning Research (01/2026)
2.2 Evaluation of Structured Output Predictions
Irugalbandara and Li et al. evaluate their approaches using aggregaled single variable measures like (micro- averaged) precision, recall, and FI scores. Additionally, [rugalbandara] incorporates the Geometric Mean Score (GMS) and Consistency to provide a further assessment. In the work of Wang et al., Schema Accuracy and Content Similarity serve as evaluation metrics. The previously discussed Visual RAG Pipeline is assessed based on the accuracy of predictions and Ground Truth (GT) values (Tamm & Keuper b)-
Chen et al.) introduce a framework that pools multiple evaluation metrics for SOPs. The responses are reprosented "as objects of certain data types', and the metrics are calculated by 'matching of common substructures, possibly followed by normalization" (Chen et al.pp Lu et al.] focus on the evaluation of LLMs output in JSON format. The analysis refers to the performance of "valid JSON outputs against a given schema" * {Lu et al.p. The following aspects are examined: 'structure underslanding, escaping, and natural language description, to determine how to assess and enable LLMs to generate valid responses" (Lu et al.). The requirements on the evaluation metrics for the proposed mSOP-765k benchmark has to be flexible different ways. First, the occurrence of key items varies depending on the input image, i.e., a value to each key is not always presented in an image. Furthermore, the evaluation metrie has to support the evaluation of different data types. For keys with numerical data types, simple equality evaluation can be suitable. However, evaluation of string keys requires semantic distance metric. Morcover, we are not only interested in the sum of the single errors over the different keys of the SOPs. We are also interested in the correct of the entire structure of the output predictions. Consequently, we introduce combined evaluation metrics in Section [.
2.3 Benchmarks in Related Studies
Existing structured output benchmarks differ with respoct to the prodiction output formats. These include formats such as JSON schemata and related textual structured representations, as well as visual outputs. Lu et al. provide the SchemaBencl benchmark that consists of about 40k different JSON schemata. Further- more, the JSONSchemm.Bench benchmark has been introduced by Geng al.] This benchmark comprises about 10k JSON schemata.
Tang et al.) introduce the STRUC-BENCH benchmark, which aims to evaluate the generation of structured tables from textual input. Specifically, the benchmark focuses on producing raw text tables, HTML tables, and LaTeX tables. The StruciEval benchmark is divided into two subsets, StructEoal-7 T and StructEval- V. designed to generate SOPs for generation and conversion tasks (Yang et al.)). StructEval- T targets the gen- eration of text-based structures. The supported formats within this subset include JSON, XML, YAML, and Markdown. In contrast, StructEval- V is dedicated to producing executable code as SOPs, enabling visual rendering of generated structures. Example output formats in SbrucLFval-V encompass HTML, Malplollib, Canvas, LaTeX, SVG, and more.
The previously established benchmarks have been exclusively based on textual inputs, typically formatted as JSON schemata or other text-based representations. Consequently, these benchmarks are not suitable for evaluating multi-modal inputs that combine both image and text modalities. To address this limitation, we introduce the novel mSOP-765k benchmark. The characterizations of the proposed benchmark are a large number of 765k data samples with manually annotated GT structured prediction targets. The structured output data demonstrates significant variability in multiple dimensions: it encompasses substantial number of variables with strong variability different data types which range from simple numerical and string to complex list data types. The evaluation of mumerical variables can be achieved by means of a straightforward numerical evaluation of the predictions. However, for variables with other data types, a more complex evalustion is required. Furthermore, the mSOP-765k benchmark combines the SOPs task with the fine- grained visual object problem due to its large-scale visual dataset. Consequently, the benchmark underscores the elevated practical relevance that may be applicable to similar world problems.
3

---

Published in Transactions on Machine Learning Research (01/2026)
3 Dataset
The objective of this paper is to establish benchmark for the evaluation of Structured Outpul Predictions from multi-modal input data. Hence, the key components for the task are the multi-modality of input data and the provision of output data in structured for- mat. In addition, the data types of the struc- tured output are expected to vary represent the complexity of real-world problems.
Knorr Suppenliebe diverse Sorten, für 750 ml je Beutel (1) 51 % gespart
Knorr
Supporiaes
TOMATER EN
Knorr
Suppel HÜNNER
In the retail domain, product advertisements in leaflets exhibit the characterizations de- scribed previously. The visual product adver- tisement images contain a wealth of uextual in- formation, whereby the multi-modality is en- sured. Information about products and pro- motions is typically advertised with a variety of descriptive details, including the product name, the brand, and the sales price. This output data is suitable for the provision in a structured format. Furthermore, the data types of the product and promotion data vary, from strings, integers, floating-point numbers, lists, and other data formats. The proposed large-scale mSOP-765K benchmark measures the ability of models to map such advertise- ment images to the according structured prod- uct information.
- -.44
(a) Input sample: product visualizations and textual meta- information are combined single image.
Data
Target
Value
product
brand
Knorr, Suppenliebe
product category CTINs
Suppentopf, Suppen, Suppe [08710908937095, 08710908937590
product weight different types
NaN
yes
promotion
price
0.44
regular price
NaN
relative discount
51
absolute discount
NaN
(b) Associated structured larget data; missing target val- ues are indicated by NaN. Because the inputs displays more than one product, the GTINs target variable holds list of multiple product ids.
Image Dataset: The raw image dataset is based the Retail-786k dataset (Lamm & Keuper] a). This image dataset consists exclusively of image advertisements that have been cropped from leaflet pages. It has been published in context of a fine-grained image classification benchmark in which potentially visually similar products are classified by their Global Trade Ttem Number (GTIN) (ASBBI).. GTINs are internationally standardized 14-digit numbers for unique product identification. We extend the Retail-7860 image dataset by adding structured data annotations (as described in the next paragraph) to each adver- tisement image. Due to data cleaning procedures based on missing structured data, the number of images in our benchmark is lower than the original 786k. In total, our provided image dataset now contains 765,463 images divided into a train split of 728,892 images and a test split of 36,571 images. The partitioning of the dataset into training and test split was adopted from (Lamm Keuper a The training/ter test splits contain minimum number of 10/3 images per label, where labels are defined by the semantic grouping of GTINs. In addition, our image dataset is available in two versions: one with images resized so that the longer edge measures 512 pixels, and another where the longer edge measures 256 pixels. Moreover, the image dataset ex- hibits long-tail tail distribution with regard to the GTIN classes which is illustrated in Figure 7 in the Appendix.
Figure 2:
Representative sample from the mSOP-765k
dataset, consisting of a query (input) image 2a and the corresponding structured product larget (2b)
In our mSOP-765k datasel, structured data refers to product and promotion data shown in the images. The product data comprises properties about the product(s) advertised. This data includes the information of brand, product category, GTINs, product weight, and differrnt types. In contrast, the promotion data comprises the information on the advertised price, regular price, relative discound, and absolute discount. Most variables can be extracted directly from text in the advertisement.
Structured Data:
4

---

Published in Transactions on Machine Learning Research (01/2026)
images, such price, discount, and brand. However, other targets must be classified from the images, GTINs or product calegory. Therefore, the fine-grained problem of the original dataset still exists. An example of the remaining fine-grained task is shown in Figure 3 Two images from the mSOP-765k dataset 3a 3b and the corresponding product and promotion targets per image are depicted. The fine-grained diflerence illustrated by the similar visual packaging of the products and the minimal number of largets that differ in their target values (see GTINs and product weight). Other characteristics of the images the dataset are that not all variables always have appear in the images and that there may be several products depicted single image. If multiple product images displayed, this will result in list of GTINs in the output data structure. For each image in the image dataset, the structured product and promotion have been recorded manually. These data are released in parquet format for the training and test split, respectively. Table [
presents the number of GT values for each product and promotion data in the test split. The values are reported absolute numbers (n) and in percentages (%). Furthermore, the currecy of the promotion data price, regular price, and absolute discount is declared Euro (6). The promotion data relative discounl is specified as percentage.
Develey Das Original" Tormetinkkett tchup 500ml flechh i 2:98
Target
Image
Image
brand
Develey
Develey
product category
Tomato Ketchup
Tomato Ketchup
Original
Develey Tomato- Ketchup 750ml Rosche 16:1 100
GTINs
(04006824003551
[400682440361]
Original
product weight different types
500.0 Milliliter
750.0 Milliliter
Develey TOMATEN KETCHUP
NaN
NaN
price regular price
1.49
1.49
1449
1,49
NaN
NaN
TOMATO KETCHOP
relative discount
NaN
NaN
absolute discount
NaN
NaN
(a)
(b)
Figure 3: Illustration of the product and promoted data from the images differences of the two advertisements are underlined.
3a
and
3b
The fine-grained data
Table 1: Absolute numbers (n) and percentages (%) of GT values for each target in the test split.
product category
different types
product weight
regular price
relative discount
absolute discount
brand
GTINs
price
36,571
36,571
36,571
36,571
36,541
35,365
12,854
11,719
667
%
100.0
100.0
100.0
100.0
99.92
96.70
35.15
32.04
1.82
Representative Sample: Figure illustrates representative sample from the mSOP-765k dataset. The sample consists of an advertisement image [2a] and the corresponding product and promotion data (2b) The data types of the product and promotion data vary, encompassing numerical data types (e.g. integers and Boating-poin numbers) as well as complex data types like lists. In this example, the product data GTINs are annotated as a list because the advertisement image displays more than one product and the hint of various types (ger: diverse Sorten) is legible. From these occurrences, the price validity of the advertisement for multiple products can be deduced.
Data Access: The image dataset and the structured product and promotion data can be accessod at the following URL: htttps//hhuginggfeet cttee.coooaaoao and are licensed under a Creative Commons License (CC BY-NC-ND 4.0) (Creative Commons). As a further addition, the description texts appearing in the advertisement images are also provided in the repository. The description texts have been extracted using the PaddleOCR tool (PaddlePaddle) Du et al.).
4.0 International
5

---

Published in Transactions on Machine Learning Research (01/2026)
4 Evaluation
In this section, we provide a baseline empirical evaluation of current state-of-tho-art SOPs methods on the proposed mSOP-765k benchmark. These experiments include VLM based approaches (Section 4.2 as well
as a visual RAG pipeline (Section 4.3 Moreover, investigations of image and text classification on the
dataset are described in Appendix A.3.
We fix the required system- and http:/hmmn--pommppp Prompts and Structured Output Schemata: for all methods to the same input.
The system prompt is specified as: You are an assistant for question-answering tasks. While the user/human prompt is defined as: Do the user-drovided task on the input image. The answer must be provided in JSDN format. The task is: "Extract the features.", If there is no information of a target, return NaN.
Table in the Appendix shows, for each target, whether a prediction is required and specifies the data type. Data types vary from string, float, list of strings, and enumeration. It is important to note that the product attribute product weight is divided into two separate queries: one for the weight number and one for the weight wnil. The data types for the largets weight unil. and different types are enumeration. For the target weight unit, the valid values are: Gramm, Kilogramm, Milliliter, Liter, Waschladungen, Blatt, and Stück (engl. gram, kilogram, milllittrr liter, wash loads, sheot, piece). For the target variable different types, the possible values are: yes and no.
Table 2: Detailed description of the evaluation metrics for individual target variables within the prediction structure.
Target
Description of Evaluation Metric
brand
1. String preprocessing on prediction and GT values: converting all characters to lowercase; removing accents and other diacritical marks using Unicode normalization; and removing all apostrophes from the strings.
Matching criteria: the Levenshtein (Levenshtcin) distance between the prediction and GT strings is caleulated to derive similarity score. If the similarity exceeds 0.5, the strings are considered share any common words after splitting by non-word characters. If they do, the strings are still considered as a match.
match. If not, the algorithm checks whether the two strings
3. Accuracy colculation: the accuracy is computed as the proportion of matched results.
product weight
Concatenation: the prediction value is formed by combining the weight number and weight wnit predictions.
2. Matching criteria: if the prediction string is exactly equal to the GT string, it is considered a match. If not, equivalent representations with different units (e.g., 1000g = 1kg) are treated as matches by converting between grams/kilograms or milliliters/lice based on their numerical values, followed by string equality checks.
Accuracy calculation: the accuracy is computed as the proportion of matched results.
1. Accuracy calculation: the custom accuracy is measured based on exact string equal- different types
ity between the predictions and GT values which can either yes or no.
price date
1. String preprocessing on prediction: any occurrences of the character '.' are re- moved from the prediction value (if present).
2. Accuracy calculation: the custom accuracy is measured based on the exact string equality between the adapted prediction and the GT value.
Includes the targets: price, regular price, relative discovnt, and absolute discount.

---

Published in Transactions on Machine Learning Research (01/2026)
4.1 Evaluation Metrics
Evaluation metrics for single target variables: For all product and promotion sub-target within the structured output target, a custom accuracy score is determined. The evaluation metric is defined to ensure comparability between the predictions and the process by which the GT was generated. The GT values were specifically defined by human annotators. The metric for promotion data as well as the target product weight must be strict, a subsequent comparison of identical products would otherwise not be guaranteed. For the target brand, an exact match is not necessary because the spellings may vary depending on the retailer's leaflets. Based on these considerations, we introduce a custom accuracy metric based on the Levenshtein distance (Levenshtcin), avoiding the use of an exact match calculation. The process used to calculate the score for each target described in Table [ Since not all targets are advertised in every image, an absence of prediction of the targets can be valid. This applies in particular to the targets regular price, relative discound, and absolute discount but may also occur for the other targets.
Evaluation metrics for the structured output: In addition to the evaluation for the single larget variables, we propose a novel evaluation metric for the structured output. The measurement criteria is based on the union of the ovaluation metric for single target variables. There are two distinct metrics: Usargets and Usest The evaluation metric Utargets only includes samples for which the GT is given and a prediction has been delivered with regard to the considered targets. Consequently, the size of the reference set varies depending on how many target variables are considered. The evaluation Usest also considers samples that receive prediction but no GT is given, where GT is available but no prediction has been generated. In this case, the reference set is the number of the lest. datasel, i.e. 36,571 samples.
Uiargets
In general, the following applies to the evaluations in Sections 4.2 and 4.3 the experiments are executed on the mSOP-766 765k dataset with an image resolution of 512 pixels and the best result in each evaluation table is highlighted in bold, while the second-best result is underlined.
4.2 Evaluation of VLM Based Zero-Shot Approaches
We evaluated the performance of both commercial and open-source models in order to capture model di- versity. All VLMs have been used in a zero-shot setting, iee. neither training nor fine-tuning of the mod- els has been performed on the dataset. We evaluated the following models: GPT-4o-mini in the version dated 07 07/18/2024 (gpt-46-minn-20024-77-1 (OpenAT a). Gemini-2.0-flash (gemini-2.0-flashh) (Mallick & Kil-
patrick a). : Gemini-2.5-flash (gemini-2.5-ftash Mallick & Kilpatrick, bb, GPT-5-mini in the version dated
08/07/200 2025 (gpu-5-mini-2025500--0) (OpenAT dd) LLaVA version 1.6 with 34b parameters (Ilava-44b) (Liu ett al. b), Qwen2.5-VL with 32b parameters (qwen2.5vv-322)
(Bai et al.),, Mistral Small 3.1 with 24b (MistralA]), ; and Llama 3.2 Vision with 11b parameters (Ilama3.2 2-vision- 11b) Meta)). The models Ilava-34b, qwen2.5vl-32b, mistral-suallI:. 1-24b, and Ilama3 2-vision have been deployed via the Ollama framework (Ollama).
parameters (misttrl-ssmall33112111
Table reports the percentage of evaluable predictions for the models on the test split of mSOP- datasct introduced in Section g The numerical values in the table indicate the percentage of predictions that can be evaluated. A prediction is considered evaluable only if the corresponding predictions for all targets except the targets product category and GTINs are defined, i.e. no missing entries are allowed. Note that the prediction values of not advertised product and promotion data are indicated by NaN which is accepted as valid value. The highest rate of evaluable responses has been achieved by the VL. gpt-5-mini- 2025-08-07. The VLM ppt-44-miii-2024-00-11 has returned only cight invalid predictions, corresponding to 99.98% appraisable responses. The VLMs gemini-2.5-llash and gemini-2.0-flasl) have achieved 99.95% and 99.56% reviewable predictions. The VLMs 1lava-34b, qwen2.5yl-32b, and mistral-small331-2244 have provided similar magnitude of evaluable predictions whereas the VLM 2ana3.2-vision-1 has returned 78.95% assessable predictions.
The prediction of each product and promotion target variable is not feasible using VLMs alone. Specifically, information about the targets' product category and GTINs is not presented in the advertisement images. Conscquently, no valid predictions can have been generated for these targets and, therefore, no evaluations have been conducted in this zero-shot setting.
7

---

Published in Transactions on Machine Learning Research (01/2026)
Table Correctness output structures: Numerical values represent the percentage of evaluable predictions, for the test split and for each VLM. Prodictions are considered to be evaluable if the output structures contain all requested variables with the correct type. Note: positive evaluability of output structures does not implicate correctness of the prediction.
gpt-4o-mini- 2024-07-18
gemini- 2.0-flash
qwen2.5 v1-32b
mistral- small3.1-24b
llama3.2-
gemini- 2.5-flash
gpt-5-mini- 2025-08-07
Ilava-34b
vision-11b
99.98%
99.56%
99.12%
99.54%
96.53%
78.95%
99.95%
100.00%
Table 4: VLM-based Scores of the evaluation metrics for each single larget variable and VLM. Percenlages
are calculated based on comparable GT and predicted values.
small3 1-24b
gpt-4o-mini
2024-07-18
gemini-
2.0-flash
llava-34bb
qwen2.5
vI-32b
mistral-
llama3.2 2-
vision-11b
gemini-
2.5-flash
3pt-5-mini
2025-08-07
brand
95.55%
96.61%
70.79%
93.37%
94.12%
90.77%
96.73%
95.82%
product weight
71.26%
80.97%
8.04%
17.12%
16.71%
15.76%
72.44%
80.49%
different types
85.48%
88.97%
60.23%
75.64%
79.62%
56.03%
89.99%
90.70%
price regular price
97.19%
97.37%
52.53%
94.15%
94.28%
91.57%
97.37%
97.53%
85.93%
92.47%
19.00%
65.19%
84.14%
51.79%
94.89%
92.84%
relative discount.
98.75%
99.02%
48.40%
97.90%
96.19%
92.68%
98.93%
98.83%
absolute discount
82.01%
92.63%
11.41%
72.07%
62.19%
40.11%
91.89%
86.51%
Figure 13 in the Appendix illustrates the difference between the number of predictions and the number of GT values for all target variables- except product category and GTINs and VLMSs. For the targets brand, product weight, different types and price, the number predictions has matched the number of GT values for all VLMs closely, except for the Ilama3.2 2-visiovv---11 model. Other models have produced slightly more fewer predictions. A notable disparity among the VLMs evident in the number of predictions for the target clearly have generated fewer predictions
regular price. The VLMs qwen2.55v-32b and Hamma.2-vision. than the other investigated models.
Figure 15 in the Appendix presents the prediction error in É for the target variable price and each VLM. The prediction error is defined as the difference obtained by subtracting the predicted value from the GT value. The VLMs 1lava-34b and qwen2.5vl-32b, have demonstrated notable outliers, with deviations reaching up to -35,0006. Evaluations of the other models reveal a narrower distribution; however, outliers of considerable magnitude, approximately -4,0006, are also present. These findings are illustrated in Figure 14 in the Appendix. Moreover, the median of each VLM evaluation is 0.00E.
Table summarizes the results of the evaluation metrics for single target variables. The percentages refer to the subset of comparable GT values and predictions, i.e. only entries with defined values are considered. and apt-5-mini-2025-085
The VLMs gemini-2.00 2.0-lashh, gemini-2.5 results. Among all product and promotion data, the VLM gemini-2.5-flas has provided best or second best accurate predictions. The VLM gemini-2.0- has delivered the highest custom accuracy score of 99.02% for the (arget variable relative discounl.
consistently have achieved the best
Table 14 in the Appendix shows the results of the evaluation for structured output when the target variables are systematically incorporated based on the set of comparable GT values and predictions, i.c. only entries with defined values are considered. Hence, the evaluation metric Utargets is calculated. Initially, the score for the target variable brand shown; subsequently, the score the union the target variables brand and product weight are considered, and so on. Typically, the score decreases as the union of the variables under consideration expands. This incidence holds for nearly all models. Exceptions have occurred in the evaluations of the VLMs 1lav--34b and qwen2.5vl-32b. For example, for the VLM 1lava-34b, adding the target
8

---

Published in Transactions on Machine Learning Research (01/2026)
Table 5: VLM-based Results the evaluation metric Utest for the structured output. Scores are calculated
based on comparable GT and predictions values relative the entire test dataset.
gpt-4o-mini-
2021-07-11 18
mistral-
small3. 5mall3.1-21b 1-24b
llama3.2 3-2
vision-111 11b
gpt-5-mini-
2025-08-07
gemini--
2.0-flash
llava-34b
qwen2.5
v1-32b
gemini-
2.5-flash
brand
95.53%
96.19%
70.17%
92.94%
90.85%
71.66%
96.68%
95.82%
U product weight
66.08%
75.81%
5.54%
15.51%
15.30%
14.14%
68.02%
74.84%
test
different types
24.33%
30.58%
0.87%
3.04%
4.51%
5.23%
29.53%
32.57%
best
price
23.50%
29.65%
0.46%
2.88%
4.29%
4.93%
28.65%
31.67%
beat
0.04% 1
regular price
6.83%
9.59%
0.04%
0.95%
0.01%
9.67%
10.32%
0.01%
relative discount
3.66%
4.87%
0.01%
0.44%
0%
4.92%
5.28%
absolute discount
0%
0%
0%
0%
0%
0%
0%
0%
test
different types to the previously union of targets, the results have improved due to a substantial reduction in the number of evaluable GT values and predictions.
Table provides a comprehensive evaluation ion the metrie for structured output related to the entire Lest. split have dominated the
of the datasct. Once again, gemini-2. 2.0-llss,, gemini-2.5-faan and gpt-5-mini-2025-08-- top results However, has not been possible predict all product and promotion targets for any single advertisement image in the test dataset. This follows from the score value of 0% for all product and promotion data; refer to the last row of the table.
Table 15 in the Appendix illustrates the average elapsed time per request in seconds, as well as the total elapsed time in hours (optionally in days) for the test split of the dataset and for each VLM. The gemini-2.0. flash model has required the least amount of time, with a total elapsod time of 21.7 hours for all requests, followed by the VLM gwen2.5vl-32b, which accumulated total clapsed time of 46.2 hours. Furthermore, the average total costs per request and the total costs given in USD are presented. The costs for the open- source VLMs have been estimated based the assumption of Linux VM in Microsoft Azure (Microsoft). The minimal total costs for processing the test split requests using the VLM gemini-2.0-fHi has been approximately USD 7.40, followed by the VLM qwen2.55l-32b at around USD 8.70.
4.3 Evaluation of Visual RAG Based Approach
For the evaluation of the SOPs generated by visual RAG based approach, we focused on the Visual RAG Pipeline (Lamm Keuper b). Sigureereerreenn the architecture of the Visual RAG Pipeline. The pipeline consists of five main steps: Preprocessing, Vector Store, Retrieval / Classification Relational Query, Prompt Generation Completion. The pipeline based on the RAG approach (Gao et al.) and is distinguished by extension of the prompt using historical context. The context comprises historical samples that similar to the input image. These historical samples consist of advertisement images along with their corresponding product and promotion data. In the last step, called Completion, VLM request is applied. Therefore, we investigated the following VLMs: spl-4o-minn-2022-07-BB gemini-2.0-flash, gemini-2..-flash qwen2.55l-33b, and mistral-small3.111 1-224b. The commercial VLMs mentioned were selected taking into account both time constraints and cost efficiency. The VLM gwen2.5vl-320 has needed the lowest clapsed time per request, as well the lowest total costs of all requests with regard to the open-source models (Table 15 in the A Appendix). As another open-source model, the VLM mistral-small3.11 1-24b examined.

---

Published in Transactions on Machine Learning Research (01/2026)
bennd: Zuti Zuitardlla
RETRIEVAL,
picc:
CLASSIFICATION,
regular peice:
REI ATIONAL QUERY
relatrve discount:
product cregory Mozzarells
PROCESSING
GTINs 0401 4500017310.
"Extrec the Rearuros"
brand Zoil. Zolterella price 0.79
veesch. Sonten, 125
Zoliandlla Kugel
product weight 125
product category Muzazell,
brand: Zon, Zottarelaa
PROMPT
GTINs: 04014500173100,
Zait Zalizalaell
250-g-Bound
GENERATION
0401450024816
779
Mannel Waffala
Mozzrella
product category
COMPLETION
product weight 125
different sorts yes
GTIN 040145000173100 (4014500254816
VECTOR STORE
Figure 4: Illustration of the architecture of the Visual RAG Pipeline as presented in (Lamm & Keuper The pipeline consists of five main steps and is distinguished by an extension of the prompt using historical context and the RAG approach (Gao al.)
Table ( shows the percentage of evaluable predictions. The VLMs 8pt-4o-minn-20024-7--1 gemini-2.5- flash, and mistral-small3. 1-24b have achioved a maximal ovaluability of 98.14%. The VLM qwen2.5vl- 32b has providedapppraisable predictions for 11 promotion images more than the gemini-2.0-lashh model.
Table 6: Correctness of output structures: Numerical values rep- resent the accuracy of evaluable predictions, expressed as per- centages, for the test split obtained by the Visual RAG Pipeline for each VLM. Invalidity mainly arises from errors or request. timeouts.
The difference between the number of predictions and the number of GT val- ues are presented in Figure 17 in the Ap- pendix. For all product variables and the promotion variable price, there have been minimal fewer predictions than the avail- able GT values. This results from the fact that the number of fewer predictions is the number of requests that have re- turned non-defined values for all targets variables. For the remaining target variables, all models have produced a higher number of predictions, the VLMs spt-4o-miini22024-07--8 and gemini-2.0-llash have gen- erated particularly between 20,000 and 25,000 additional predictions.
gpt-4o-mini- 2024-07-18
gemini- qwen2.5 vl-32b
mistral-
gemini- 2.5-flash
2.0-flash
small3.1-244
98.14%
98.03%
98.06%
98.14%
98.14%
In Figure 16 in the Appendix, the prediction errors for the target price per VLM in the Visual RAG Pipeline is analyzed. The VLM spt-4o-mmin-20024-0711 has exhibited the most pronounced outliers in both positive and negative directions, exceeding 150.00E and dropping below -150.00E. The distribution of the VLMs gemini-2.0 2.0-flshh, qwen2.5vl-32b, and mistral-small3, 1--44 very similar. Consistently, the median value for each VLM evaluation have remained 0.00E.
Table 7: Visual RAG-based Scores of the evaluation metrics for each single target variables and VLM used in the Visual RAG Pipeline. Percentages are calculated based on comparable GT and predicted values.
gpt-4o-mini- 2024-07-18
gemini- 2.0-flash
qwen2.5 vl-32b
mistral-
gemini- 2.5-flash
small3. 1-24b
brand
98.53%
95.21%
90.04%
90.01%
98.81%
product category
83.30%
82.54%
80.26%
78.76%
70.58%
GTINs
70.87%
71.28%
69.26%
71.56%
66.85%
product weight
85.68%
85.96%
80.72%
80.63%
85.75%
different types
91.55%
89.61%
88.41%
79.95%
88.64%
price
96.48%
59.07%
18.57%
18.52%
96.45%
regular price
84.71%
68.90%
26.90%
27.33%
93.76%
relative discount
95.99%
70.39%
17.72%
15.98%
97.98%
absolute discount
77.37%
70.80%
25.24%
20.78%
90.52%
10

---

Published in Transactions on Machine Learning Research (01/2026)
Table shows the results of the evaluation metric for the single target variables. The VLMs gpt-4o-mini- 2024-07-18 and gemini-2.5-flash have delivered the best and second-best scores for all target variables except GTINs and product weight. For the variable GTINss the VLM mistral-small3.111 1-244 has achieved the highest score with 71.56% and for the target product weight, the VLM gemini-2.0- f.a-falh has attained 85.96%.
The evaluation metric for the structured output in relation to the valid GT and prediction values, Lee. Usurgets is calculated, is presented in Table 20 in the Appendix. The VLMs apt-4o-mini-2024--0 flash have dominated the results by always achieving the best and the second-best scores, respectively. Although these models have outperformed the results, the evaluation of the scores in relation to the test split, i.c. Usest is calculated, shows weakness. Table 8 demonstrates that although the two models continue to dominate the results, the Visual RAG Pipeline likewise have failed to produce correct predictions for all product and promotion variables within any single promotion image.
and gemini-2.0-
Table 21 the Appendix shows the evaluation of the clapsed time and the total costs per request and for all requests using the Visual RAG Pipeline. The shortest duration for the total elapsed time for all requests has been achieved by the VLM gemini-2.00 2.0-flash, with duration of 42.04 hours. In relation to the total costs for all requests, the VLMs mistral-small3.I 1-24b and 3v1-225 have been significantly cheaper than the other models. It should be noted that the cost estimates for the open-source VLMs have been based on running a Linux VM in Microsoft Azure (Microsoft). Among these, the previously named models have stood out with estimated costs approximately USD 13.90 and USD 17.10, respectively. Crucially, the costs only represent the step Completion of the Visual RAG Pipeline. In this evaluation, additional costs from the other pipeline steps were not considered.
Table 8: Visual RAG-based Results of the evaluation metric Usest for the structured output. Scores are calculated based on comparable GT and predictions values relative to the entire test datasel.
gpt-4o-mini- 2024-07-18
gemini- 2.0-flash
qwen2.5 vl-32b
mistral-
gemini-
small3. 1-244b
2.5-flash
brand
96.70%
93.33%
88.29%
88.41%
96.98%
U
product weight
81.92%
81.41%
75.59%
75.50%
82.14%
different types
39.00%
39.00%
35.68%
35.98%
39.73%
price
37.55%
25.74%
8.20%
8.30%
38.01%
regular price
11.17%
8.70%
0.68%
0.94%
12.56%
test
relative discount
5.97%
4.40%
0.30%
0.40%
6.58%
test
product category
5.36%
3.92%
0.27%
0.38%
5.03%
GTINs
4.78%
3.49%
0.24%
0.35%
4.49%
absolute discount
0%
0%
0%
0%
0%
test
4.4 Error Analysis of VLM Based and Visual RAG Based Approaches
Validity of Predictions Analysis. The validity of the predictions has boen reduced using the Visual RAG Pipeline (see: Tables 3 and G For instance, the VLM xpt-44-miin-2022-00711 has achieved 99.98% valid predictions by using VLM based approach in contrast to using the Visual RAG Pipeline with 98.14%. The drop in valid predictions, or rather undefined values, from the requests originates from issues in the previous steps in the Visual RAG Pipeline. Hence, the nunber of valid predictions from the pipeline depends on multiple factors in the different steps, and not only on the VLM used in the step Completion.
Analysis of Evaluation Metrics for Single Variables. Tables and 7 show the scores of the evaluation metrics for each single target variable. For the variable price, there has been a significant difference between the evaluation of the two investigated approaches. The VLM based approach using (emini--20-flass) has
11

---

Published in Transactions on Machine Learning Research (01/2026)
achieved score of 97.37% while the visual RAG based approach has only reached 59.07%. This difference is approximately 40 percentage points. The false predictions using Visual RAG Pipeline have been followed by the fact that the predictions have been identical to the price information of the historical data provided in the VLM prompt. The percentage of price predictions identical to historical price information for the VLM gemini-2.0 l used in the Visual RAG Pipeline has been 66.75%. The evaluations of the larget price for the VLM 2.mmini-2.5-fassi do not demonstrate such behavior. In this instance, a discrepancy of about I percentage point has been found between the evaluation of the VLM based and Visual RAG based approach.
Analysis of Evaluation Metrics for Structured Output. The results of the evaluation metric Usargen are illustrated in Tables 14 and 20 in the Appendix. It can be deduced from these tables (when taking into account the target variable brand and the VLMs gemini-2. fl-lashhh and sp1-4o-minn-2022-07-11 that the score using the Visual RAG Pipeline has been about 2 and 3 percentage points higher than using the VLM alone. All other models have returned worse scores using the Visual RAG Pipeline for the same variable. For the union of the target variables brand and product weight, it has been generated higher scores from all VLMs used in the visual RAG based approach. The integration of additional variables has led to a reduction in scores. Moreover, the table comparison shows that, when using the VLMs alone, only the gemini-2.5-flash model has produced better results comparable to the Visual RAG based approach. The evaluations of the right predictions in relation to the entire test dataset are illustrated in Tables 5 and The scores using Visual RAG Pipeline with the best, VLM (gemini--2.5-flash) have been reached in higher percentages than using the best models (gemini-2.5-flash and
in the VLM based approach in terms of the union of variables from brand up to relative discount inclusively. It is also important to mention that product category and GTINs is only predictable by the visual RAG based approach. This characterization stands out for this method. Conscquently, the analysis of the two tables also shows that there has been promotion image for which all the predictions of the target variables are valid.
Qualitative Error Analysis. In addition, an analysis of three example promotion images is presented. Figure 5 displays the promotion images and the corresponding approach combination that has correctly predicted the (arget variables (excluded: absolule discound), whereas the other approach has failed. Moreover, Figure 5c shows a promotion image for which both methods have made incorrect predictions. For the promotion image in Figure 5a the VLM gemini-2.0 2.0-flash has correctly predicted (all) product and promotion data. It is important to mention that using the VLM based approach, the product data GTINs and product category have not been predicted because these information are not printed in the advertisement image. The predictions for the promotion image shown in Figure 5a are listed in Table 22 in the Appendix. The false predictions of the Visual RAG based approach Visual RAG Pipeline/ 1opo--4ommni-2222444 caused by the information used from historical data. For the advertisement image shown Figure 5b the list of predictions provided in Table 23 the Appendix. The prediction using the VLM gemini-2.00flann for the target product weight (consists of requests for weight number and weight unit) has differed to the GT values. In contrast to this, the method Visual RAG Pipeline/gpt--oommine 100--44-mini-222--7---11 has predicted all targets
have been
Mentos Gum Pure Fresh
GREA
verschiedene Sortan 35 Stück le 70- Dose g
PILSNER UROUELL 6 Woaun zmp. 0.48 (11 3.99
L'OREAL MEN 150
ver Sortem
1.18/
MVNCCE SLL
je 11699
69
96H NO STOP
1 79 2.48
mentos PURE
NORDBRAND SAURE KIRSCHE
13
3.9 19 455
guspart
sparon 28%
(a) VLM/gemini-2.00 2.0-flash
(b) Visual RAG Pipeline/gpi Baaiine/gpt-eere wio-ooo------ 202---202--0--1
(c) neither of the two methods
Figure 5: Illustration of promotion images and the approach combination that generates correct SOPs. The predictions and GT values have been returned from the approaches are shown in Tables 22 to 24 in the Appendix.
12

---

Published in Transactions on Machine Learning Research (01/2026)
correctly (exclusively: absolute discount). The prediction listings for the advertisement image, illustrated in Figure 5c are shown in Table 24 in the Appendix. The incorrect predictions have likely been caused by the fact that two products are visible in the advertisement image, along with their product and promotion data.
Error Analysis With Minimal Training Images. As a further quali- tative analysis, we evaluated the evaluation metric for the structured out- put for a label containing only the minimal number of training images (10 images) and test images (3 images). One of the promotion test images and the predictions of the two approaches LLM/gemmni-22 2aadd-2.0-fffa 0---sahhhhhhhe and Visual RAG Pipcline/gpt-ane 2opt-4o-miii-2224-07--2 are shown in Figure G Gand Table 25 in the Appendix. Both approaches have correctly predicted the targets brand, product weight, and price. However, for the other targets, incor- rect predictions have been observed. This discrepancy has arisen because the GT values do not always align with the information displayed in the images. For instance, some images lack any indication of different types, although the GT values specify them. Furthermore, both approaches occasionally have generated predictions for targets that are not visually present in the images, particularly for regular price, relative discount, and absolute discount.
Deo Roll-on 50 ml 100ml
Fa
1-35 0. 88
4% SPAREN
Figure 6: One of three test im- ages from a label with only 10 training images.
5 Discussion, Limitations and Outlook
In this paper, we present the mSOP-765k benchmark that comprises about 765k samples, primarily product advertisement images alongside the corresponding product and promotion data. These data are characterized by their structured format with data types varying depending on the specific variable. Moreover, not all images necessarily feature the same set of variables, as the elements being advertised can differ from one image to another. Our evaluations of SOPs focus on VLM based approaches as well visual RAG based approach. To evaluate the different methods, self-defined metrics for single variables and structured output are developed and investigated.
The evaluation of the Visual RAG Pipeline indicated that for certain variables, such as GTINs and product category- which are not depicted in promotion images - training data like historical data are required. However, for other targets, this approach can be unsuccessful, as it can lead to erroneous predictions resulting from the direct adoption of historical data, espocially for the promotion data. Figure 18a in the Appendix shows an advertisement image for which the Visual RAG based approach incorrectly predicts the target price. The false predictions are enumerated in Figure 18b in the Appendix. Furthermore, the historical data of the target price is presented that were incorporated into the prompt. The majority of the predictions coincide with the values found in the historical data, indicating a tendency for the model to replicate these inputs as its predictions. Similarly, the evaluations of VLMs demonstrate that accurately predicting product and promotion data through a single unified process remains a significant challenge.
The error analysis of the VLM based approach reveals that VLMs exhibit challenges in accurately interpreting images containing multiple price specifications. Detailed price prediction results for each VLM, corresponding to the advertisement image presented in Figure 19a in the Appendix, are provided in Figure 19b in the Appendix. Moreover, product weight specifications may be represented using different units of measurement. For instance, the product advertisement illustrated in Figure 20a in the Appendix includes weight information expressed both in grams and in pieces. The associated product weight predictions are documented in Figure 20b in the Appendix. Section A.8 in the Appendix shows that the fine-tuning based approach is not eligible because of its high cost and lengthy training time. Fine-tuned models are static, requiring retriinig whenever the range of product advertisements change. Moreover, the evaluations show that the fine-tuned models tend to be overfitted. In conclusion, the SOPs task stays unresolved with the mSOP-765k datasct. Based on these findings, the choice of approach may depend on the promotion and product variable in order to achieve better results. Moreover, further investigation into reasoning models (Zhong et al.), which offer additional potential by incorporating a thought process, is warranted. Finally, the proposed mSOP-765K benchmark provides a solid foundation for future research and can be used as a benchmark in a variety of tasks.
13

---

Published in Transactions on Machine Learning Research (01/2026)
References
GSI AISBL. Global trade item number (GTIN). URL httto://www.ggl..or/////////////
Executive Srrt__Smmarryypp
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Penglei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VI technical report. URL http://arxiv org/ abs 25022.
13923
Yunmo Chen, William Gantt, Tongfei Chen, Aaron Steven White, and Benjamin Van Durme. A unified view of evaluation metrics for structured prediction. URL http: /ttp:///rxiv.. http:/arxvvv..orabbaae/aa/ wrgg/ba/////...o.. 13793
Creative Commons. AtrrbbutoonKunCCmmmmerraAAAAooit
4.0 international.
URL https://
creativeconmons.org... http:///iiiccceee///oooo///////0
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finctuning of quantized LLMs. URL
http:/wwwwrrg/bb///000000 14314
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North, pp. 4171-4186. Association for Computational Linguistics. doi: 10.18633vv/NN191144 URL http://aclweb. http://..rrr../mmooooo/////11111
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. URL http://arxiv.out hwtpaaaii...oggbbba.
11929
Yuning Du, Chenxia Li, Ruoyu Guo, Xiaoting Yin, Weiwei Liu, Jun Zhou, Yifan Bai, Zilin Yu, Ychua Yang, Qingqing Dang, and Haoshuang Wang. PP-OCR: practical ultra light weight OCR system. URL
http://arxiv/ http://aaxivvv.r/abbbe
www.:.rggg....... htt//bbs/2000 209999999999111
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Rotrievalaungmented generation for large language models: A survey, URL http:
//arxiv.org/aat httpb/orrr/abbs211..00000 10997
Saibo Geng, Hudson Cooper, Michal Moskal, Samuel Jenkins, Julian Berman, Nathan Ranchin, Robert West, Eric Horvitz, and Harsha Nori. JSONSchemaBenchh A rigorous benchmark of structured outputs for language models. URL http::/aardd..or///b////5000
Google. Fine-tune gemma for vision tasks using hugging face transformers and QLoRA, a. URL https:
//ai.googa /ttp////ww/.cooo/oor////a//aaaaa
_alortttee_g_rr_______ glora
Google. Gemma model overview, b. URL
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778. IEEE,
ISBN 978-1-4673-8851-1. doi: 10.1109/CVPR.2016.000 URI http://1eeexpplore ieee wraeeee.rrrreeee documettt 7780459/
HuggingFaco. Transformers. URL httppp/s//uggggnaaae.../o/////os
Chandra Irugalbandara. Meaning typed prompting: A technique for efficient, reliable structured output generation. URL
18146
Bianca Lamm and Janis Keuper. Retail-886k: a large-scale dataset for visual entity matching, a. URL
http://arxiv.. wrw. wtt.:////g/........///// habbs/2300... 17164
14

---

Published in Transactions on Machine Learning Research (01/2026)
Bianca Lamm and Janis Keuper. A visual RAG pipeline for few-shot fine-grained product classification. In Proceedings of the IEEE/CVE /CFF conference on compuler vision and pallern recognilion, pp. 2089-2000000 2098, b.
VT Levenshtein. Binary coors capable or "correcting deletions, insertions, and reversals. In Soviel physics- doklady, volume 10. Issue:
Marie Stephen Leo. 5281/zenooo 12327267
Release for zenodo.
URL https://
zenodo otg//ol//00.558//zzedd.
Yinghao Li, Rampi Ramprasad, and Chao Zhang. A simple but effective approach to improve structured language model output for information extraction. URL http:://arsi...r//aabb////////// 1333441333 13364
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, b.
Michael Xicyang Liu, Frederick Liu, Alexander Fiannaca, Torry Koo, Lucas Dixon, Michael Terry, and Carrie J. Cai. "we need structured output": Towards user-centered constraints on large language model output. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pp. 1 1-9. ACM, c. ISBN 979-8-4007-0331-7 doi: 10.1145/361390553665000 URL https://dl..... acm. org/doi/10. 6667677 3650756
1145/3613905
Yaxi Lu, Haolun Li, Xin Cong, Zhong Zhang, Yesai Wu, Yankai Lin, Zhiyuan Liu, Fangming Liu, and Maosong Sun. Learning generate structured output with schema reinforcement learning. URL http:
/ 18878
Shrestha Basu Mallick and Logan Kilpatrick. Gemini 2.0: Flash, flash-lite and pro, a. URL https://
Shrestha Basu Mallick and Logan Kilpatrick. Gemini 2.5: Updates to our family of thinking models, b. URL
googleblog..oooooooooooooooooooo cwm/ eww.//nigemmiii-- 2-----2----hhne wewt-tiiinnnnnlelnnl-iieeeee model
Meta. Llama 3.2 vision models (11b/90b).
URL https://www. wttp///wwllaaaaa.mmm.m cww/docs/
model-cards-and-paa anterdd-andd-rooppt-.. pwwppt www.oooroatttaaaaaaaaaaaaaaaaaae 3/#-1lama-3.22 1 (11b/900))
Microsoft. Azure machine learning pricing.
Socaills/nachine-eeamiii
MistralAL. Mistral small 3.1. URL httptppp//aattaaaaaa///////aaa13
MLCommons, Croissant. CRL
Ollama. Get up and rumning with large language models. URL https://ollama.com
OpenAl. GPT-4o mini: advancing cost-efficient intelligence, a. URL https://openai. compti..ooinnnii index/
OpenAI. GPT-4o, b. URL
OpenAl. GPT-5 mini, c. URL htt.tt/er//bbbbooommaaaaacca/r/r
OpenAl. Introducing GPT-5, d. URL
Ap Supervised fine-tuning, e.
URL httee://aaatoomm...naaa..ccoooop
Supervised-finettuile
OpenAI. Vision fine-tuning, f. URL
PaddlePaddle. PaddleOCR, awesome multilingual OCR toolkits based on PaddlePaddle.
URL https:
//github..com/Paddleee htt.thah..cc/aaaaaaaaaaaaaaaaaad
15

---

Published in Transactions on Machine Learning Research (01/2026)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and others. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PmLR.
Xiangru Tang, Yiming Zong, Jason Phang, Yilun Zhao, Wangchunshu Zhou, Arman Cohan, and Mark Ger- stcin. Struc-bench: Are large language models good at generating complex structured tabular data? In Proceedings of the 2024 Conference of the North American Chapler of the Associalion for Compulational Linguistics: Human Language Technologies (Volume 2: Short Papers), pp. 12 34. Association for Com- putational Linguistics. doi: 10.18653 v1 2224..aaccl-shon
URL https://aclantholoog hwrpolloooogggg/ogooooooo org/2024.
naacl-short.2
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. 30.
Darren Yow-Bang Wang, Zhengyuan Shen, Soumya Smruti Mishra, Zhichao Xu, Yifei Teng, and Haibo Ding, SLOT: Structuring the output of large language models. URL
04016
Jialin Yang, Dongfu Jiang, Lipeng He, Sherman Siu, Yuxuan Zhang, Disen Liao, Zhuofeng Li, Huaye Zeng, Yiming Jia, Haozhe Wang, Benjamin Schneider, Chi Ruan, Wentao Ma, Zhiheng Lyu, Yifei Wang, Yi Lu, Quy Duc Do, Ziyan Jiang, Ping Nie, and Wenhu Chen. StructEval: Benchmarking LLMs' capabilities to generate structural outputs. URL http::/ttt//tt wwwporb////...211......3
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. URL
wtw/ 2204../23440000 00685.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ron, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Yun Nic, and Ji-Rong Wen. A survey of large language models. URL httt:://arxa orgooogg/aao. abs//a/a.../// 18223
Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, Chao Cao, Hanqi Jiang, Hanxu Chen, Yiwei Li, Junhao Chen, Huawen Hu, Yiheng Liu, Huaqin Zhao, Shaochen Xu, Haixing Dai, Lin Zhao, Ruidong Zhang, Wei Zhao, Zhenyuan Yang, Jingyuan Chen, Peilong Wang, Wei Ruan, Hui Wang, Huan Zhao, Jing Zhang, Yiming Ren, Shihuan Qin, Tong Chen, Jiaxi Li, Arif Hassan Zidan, Afrar Jahin, Minheng Chen, Sichen Xia, Jason Holmes, Yan Zhuang, Jiaqi Wang, Bochen Xu, Weiran Xia, Jichao Yu, Kaibo Tang, Yaxuan Yang, Bolun Sun, Tao Yang, Guoyu Lu, Xianqiao Wang, Lilong Chai, Hc Li, Jin Lu, Xin Zhang, Bao Ge, Xintao Hu, Lian Zhang, Hua Zhou, Lu Zhang, Shu Zhang, Zhen Xiang, Yudan Ren, Jun Liu, Xi Jiang, Yu Bao, Wei Zhang, Xiang Li, Gang Li, Wei Liu, Dinggang Shen, Andrea Sikora, Xiaoming Zhai, Dajiang Zhu, Tuo Zhang, and Tianming Lin. Evaluation of OpenAI ol: Opportunitics and challenges of AGI. URL http://arxiv. wwwaaii..gg/aaaaaaaaaaaaoo aewaab/......//////../////ggo/ee 2409. 18486
16


Reviews:
[{"review_id": "kAUHiv8a5e", "reviewer": "Reviewer_5YLW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Contributions:\n\n- Introduces mSOP-765k, a large-scale benchmark for multi-modal structured output prediction (mSOP) with 765k product advertisement images annotated with structured data (text, numbers, lists).\n- Provides novel evaluation metrics for complex structured outputs, beyond single-value predictions.\n- Performs a baseline evaluation using LLMs, VLMs, Vision Transformers (ViTs), and multi-modal RAG pipelines, showing current state-of-the-art methods struggle on multi-value predictions.\n- Dataset and benchmark are openly available under a Creative Commons license.\n\nStrengths:\n\n- Scale and diversity: Large dataset with real-world images and multi-modal inputs.\n- Structured output focus: Supports machine-readable outputs (key-value pairs) covering multiple data types (strings, floats, lists).\n- Comprehensive evaluation: Metrics for both individual targets and full structured outputs, enabling robust benchmarking.\n- Baseline analysis: Highlights current model limitations and provides reference points for improvement.\n\nWeaknesses / Limitations:\n\n- Zero-shot limitations: VLMs and LLMs fail to predict all targets accurately, especially multi-value outputs like GTINs and product categories.\n- Complexity: Long-tail distributions and multiple products per image make accurate prediction challenging.\n- Partial coverage: Not all targets are present in every image; some require manual extraction or OCR preprocessing.\n- State-of-the-art gaps: Existing methods perform well on single targets but struggle with consistent multi-value structured outputs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Clear benchmark scenarios.", "audience_yn": "Yes", "audience_explanation": "VLLM are certainly a hot topic these days.", "requested_changes": "- Baseline Models: Include additional baselines, particularly fine-tuned LLMs/VLMs, to allow more meaningful performance comparisons beyond zero-shot evaluation.\n\n- Evaluation Metrics Clarity: Explain in more detail how the structured output metrics handle missing values, lists, and multi-product entries. Include step-by-step examples for clarity.\n\n- Zero-Shot Limitations: Discuss the limitations of VLMs in zero-shot settings more explicitly, highlighting which targets remain unsolved and why.\n\n- Error Analysis: Provide qualitative examples of prediction errors, including extreme outliers in price predictions, to illustrate real-world challenges.\n\n- Handling Missing/NaN Values: Include clear guidelines on how missing or NaN target values are treated during evaluation and how models are expected to respond.\n\n- Long-Tail Distribution: Discuss strategies to handle class imbalance, given the long-tail distribution of GTINs and other product categories, to help interpret results in a realistic retail setting.\n\n- Multi-Product Images: Include analysis of images containing multiple products and how predictions of list-type attributes (e.g., GTINs) are evaluated and scored.", "broader_impact_concerns": "would store names have an impact in your model learning/predictions?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3F1hycMZZw", "reviewer": "Reviewer_6iZv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a dataset to test the multimodal capabilities of VLMs. This dataset is based on product advertisements, which typically display the product, with a canonical brand, and other information like the price, size, weight, or discount. This allows the target predictions to follow a structured verifiable format, making it easier to benchmark. The performance of different open-source and closed source VLMs is evaluated.\n\nThe authors claim that the proposed benchmark is a challenging problem that cannot yet be completely solved. To me, this may have been true at time of creation, when the most powerful VLMs were GPT-4o-2024-07-18 and Gemini 2.0 Flash, but the SOTA has progressed quite far since then, with the paper itself acknowledging that GPT-4o achieves a 99.98% valid prediction rate (see Table 3). Prediction rates only go down when filtered to single target attributes.\n\nAdditionally, the authors mention all images are resized to be at most 512 pixels wide. In my experience this can often make small text close to illegible, even for humans, and it's possible it may not even be possible to achieve 100% accuracy.\n\nTo me, this lowers the usefulness of the dataset, but it does seem like an interesting dataset to have around, and even datasets that are mostly-solved can have value (see a large number of papers built off of MNIST, for example)", "claims_and_evidence_yn": "Yes", "claims_explanation": "Paper provides enough details on dataset contruction, VLMs used, scoring mechanisms, etc.", "audience_yn": "Yes", "audience_explanation": "Could be of interest to people working on VLMs", "requested_changes": "N/A", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WYCwmDhu70", "reviewer": "Reviewer_zGWs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents a large-scale benchmark for the evaluation of multimodal Structured Output Prediction. Concretely, an extension of the Retail-786k dataset, which contains real-world product advertisements, and which in this work is augmented with manual annotations for product name or brand, and numerical data such as product weight, price, and discount. It also introduces ad-hoc metrics for the evaluation of the prediction of labels in an escenario of multiclass multilabel classification.\n\nStrengths:\n- The evaluation of VLM-based models and Visual RAG-based models in itself is already a contribution.\n- The work is of high relevance for the TMLR community, as it introduces the largest corpus of its kind up to now.\n- The document is well written. All ideas flow smoothly, and the contributions are clearly grasped.\n- The constraints for the different tasks that are evaluated are well defined.\n- The analysis of results, in quantitative terms, provides good hints for future applications.\n\nOpportunities for improvement:\n- The limitations of current state-of-the art seems poorly explored or explained.\n- There is a fixed split for training and test sets. Although this is often the way to make datasets public, it lacks a statistical analysis of the distribution of variables which could provide hints on whether data points are separable or not. For instance, color or intensity pixel distribution, or the distribution of outputs of hidden layers of a pre-trained CNN model, or sentence embeddings.\n- It seems that Table 3 refers to recall, and Table 4 to precision. If so, please use this common terminology. Otherwise, more detail about the discrepancy between the numeric results must be provided.\n\nOther minor observations:\n- It is often better if figures and tables appear right after the paragraph where they are mentioned for the first time.\n- The explanation about \"Evaluation metrics for the structured output\" is a bit unclear. It could be improved.\n- The architecture for the visual RAG could be included in the main document.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Claims are support by either, citations to relevant work, of the results presented in the submitted manuscript.", "audience_yn": "Yes", "audience_explanation": "The findings are of high relevance to the machine learning community.", "requested_changes": "It seems that Table 3 refers to recall, and Table 4 to precision. If so, please use this common terminology. Otherwise, more detail about the discrepancy between the numeric results must be provided.", "broader_impact_concerns": "No particular comment on this regard.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)