Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper introduces a new benchmark, FETA - Foundational Models for Expert Task Applications, that targets the evaluation and improvements of foundational models on expert domain tasks common in real-world applications. The authors propose an automatic text-image pairs extraction pipeline, evaluation metrics and baseline using Foundational models such as CLIP and FETA-tuned versions of those FMs. The paper concludes that current out-of-box foundational models suffer significant performance drop once given domain-specific expert tasks, and FETA can further advance real-world practical applications of foundational models in expert domains.

Overall, the paper is well motivated and could bring potential real-world impact through improving expert task performance with foundational models. However, some of the discussions regarding methodology and experiment results are not as clear, which leads to my reservation of this paper.
strengths: This paper has a clear motivation and potentially would be beneficial for many of the academic community and beyond in the industry. Out-of-box foundational models that can perform practical, real-world expert tasks are of high interest to the industry, and benchmarks that advance the progression in this direction would be an important contribution.

Overall, the paper is easy to follow, and creates a novel expert domain dataset that would be interesting for many.
weaknesses: There are a few points in data collection methodology and experiment result conclusions that could be further discussed and strengthened:

1. When creating the dataset from raw PDF files, the procedures of data conversion to images and texts are non-trivial and potentially error-prone:
- 1.1 In the supplement, the authors "merged together spatially close text boxes". How is spatially close defined? Given the density of information for data such as car manuals and IKEA catalogs, this could be potentially error-prone.
- 1.2 From sample data, especially for IKEA catalogs, there are many cases where one single large image (e.g. living room or cockpit) fills one or two pages, with multiple text boxes describing parts of the image (e.g. sofa, shelves, speedometer). Would the accuracy be inflated if any texts relevant to this whole image be counted as a correct matching under MIL?
- 1.3 The automatic annotation method only considers contents within the same one page, while in catalogs and manuals, potentially many contents extend beyond one page. It is a fair assumption and minor point - is there any estimate that such assumption would not affect overall data quality?

2. When conducting the experiment, the data is split into five folds, and the results are being averaged and further averaged across manufacturers. Could you explain the choice of using average (e.g. instead of median)?

3. In the baseline method, there are two methods worth discussing: "concatenation" that concatenates all texts in the vicinity of the image into one long text and set it as positive example; and "choose-one" that randomly select one of the texts from the MIL “bag” as the positive example and train using the original CLIP loss. Neither methods are particularly accurate, and it could worsen the results as those texts could belong to other images on the same page. Is it worth it keeping these two methods?

4. Given the complexity and error-proneness of the data retrieval and automatic annotation process, there should be a direct comparison between manual annotation and automatic annotation. Given the same raw dataset, how accurate is the automatic annotation itself? The paper concludes that since the there is similar trend in model performance improvement with automatic and manual annotated data, the automatic annotation method is validated. I believe that there should be more rigor testing the reliability and accuracy of the automatic annotation process.

5. Therefore, when discussing model accuracy, I would be more comfortable with the same raw dataset experiments tested on ground truth or manually annotated data.

6. In the discussion of experiment result, the paper mentions that locked parameter method consistently over-performs, but it is not consistent with the experiment data. For example, in many-shots testing, CLIP-MIL with unlocked parameters perform consistently better in text-to-image task than the locked parameter one.

7. When choosing expert tasks for this paper, the authors note that they largely look for two criteria - abundant text & images, as well as real-world "unnatural" images that would fall under the long tail distribution of the training set for large FMs. A more diverse dataset in different domains might be interesting outside of manufacturing / catalog, such as healthcare. In addition, within the car manual dataset, the diversity of manufacturers could also impact how well the expert tasks are represented. For example, Nissan and Renault are strategic partners that sometimes share the same design - minor details but would be interesting to point out.

correctness: I have voiced my concerns on the correctness of data collection methodology and experiment result discussion in the previous section above.
clarity: This paper is clear and easy to follow.
relation to prior work: To the best of my knowledge, this work discusses how it differs from previous contributions.
additional feedback: Minor typos: 
- Line 169 "one of" instead of "on of".
- Table 3 and Table 4 description: "results" are misspelled.

In Table 1, one of the metrics is average texts per doc - could you define texts more clearly? Paragraphs?

It is good that authors have considered other FMs since the paper results and benchmark is largely based on CLIP for image-text / text-image retrieval. It might be beneficial to add FLAVA results to the main paper.
review_title: Starting Point for Evaluating Foundational Models on Expert Task Applications

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposed the first of its kind foundational models for expert task applications benchmark and dataset focused on evaluating foundational models on expert data tasks. An automatic text-image pairs extraction pipeline from product documentation data is released. Presented a novel Multiple instance learning + contrastive learning approach. Extensive experiments are conducted, including sufficient ablation studies.
strengths: 1. The authors claim this work is the first of its kind foundational model for expert tasks dataset and benchmark.
2. This work released an automatic text-image pairs extraction pipeline from product documentation data.
3. Extensive experiments are conducted, and multiple baselines and sufficient ablation tests are provided.
4. Overall, this paper is well structured and statements are clearly presented.
weaknesses: 1. It would be good if there are simple figures illustrating the automatic annotation and the differences between CLIP and those MIL variants of CLIP.
2. The resulting recall score on this documentation dataset is significantly lower than those of normal multimodal datasets - which is good, have you conducted any error analysis? For example, is it because the textual descriptions in the dataset are similar(as all come from a narrow scope) or is it the limitation of the model performance?
3. Typo: pyTorch(P.6 #190)
correctness: The paper is constructed in a sound way, and evaluation methods and experiment design for benchmark models are appropriately designed.
clarity: The paper is clearly written.
relation to prior work: The paper clearly discussed how this work differs from previous works.
additional feedback: No additional feedback.
review_title: Review on FETA

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper introduces the FETA benchmark for multi-modal (text-to-image and image-to-text) retrieval tasks for technical documentation derived from car manuals and Ikea catalogues and provides a pipeline for the extraction of image-text pairs from other PDF data sources. The authors explore a broad range of baseline modeling approaches, evaluation configurations, and ablations on this task to benchmark the performance on this novel task.
strengths: - The application of large scale pretrained models to technical diagrams is understudied and this work contributes to filling an important gap.
- The authors explore extensive baselines and experimental configurations and demonstrate the significant impact of modeling choices on the resulting performance.
- The paper provides a novel multiple instance learning (MIL) extension to the CLIP model
- The authors release the code pipeline for PDF processing that could enable the generation a broader range of diverse paired image-text datasets

weaknesses: - While the paper makes claims about tackling the general challenge of benchmark data for expert data domains, the current datasets cover a very narrow set of domains and lack sufficient diversity to make generalizable conclusions about the ability of models to tackle expert data tasks.
- The current framing of the paper tends to overstate the novelty of the application area. While this area is certainly under-studied, there is existing work on non-natural images in technical and expert domains – scientific diagrams, patents, biomedical data, etc. See the “Relation to Prior Work” section below.
- Given the automated nature of the generated labels, further analysis is needed to understand the potential errors and limitations of the dataset. While the comparison of the relative model results between the automated labels and the manual labels provides some confidence in the automated labeling process, manual validation of a sample of annotations to identify common error types and their prevalence would be informative.
- The paper emphasizes the connection to real world practical applications, but the applicability of evaluation tasks to real world use cases is not clear to me. For example, language accompanying illustrations in car manual is unlikely to match the types of queries a real user would use to search for diagrams. A better connection to the potential downstream tasks that such a pretrained model could be used for is needed.
- The observed model generalization performance between manuals from different car manufacturers seems poor, with a substantial amount of training data from the given manufacturer (Many-Shot) required to achieve good performance. This seems to point to a lack of sufficient diversity to enable generalization to new manufacturers which gives me doubts that the results can provide insights that generalize more broadly to other technical diagram applications.

correctness: The claims and approaches seem correct.
clarity: The paper is generally structurally well-written although there are some issues with typos and proofreading.  
relation to prior work: The paper is missing significant prior work in the space of tackling expert tasks in general and those specifically related to sketch-like and synthetic images outside the style of typical foundation model training sets. The authors should include further references in the fields of patent analysis, scientific figures, and biomedical applications. Some potentially relevant citations:
- Kucer, Michal, et al. "DeepPatent: Large scale patent drawing recognition and retrieval." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.
- Gryaditskaya, Yulia, et al. "OpenSketch: a richly-annotated dataset of product design sketches." ACM Trans. Graph. 38.6 (2019): 232-1.
- Lee, Jieh-Sheng, and Jieh Hsiang. "Patent claim generation by fine-tuning OpenAI GPT-2." World Patent Information 62 (2020): 101983.
- Pustu-Iren, Kader, Gerrit Bruns, and Ralph Ewerth. "A multimodal approach for semantic patent image retrieval." Proceedings of the 2nd Workshop on Patent Text Mining and Semantic Technologies (PatentSemTech) 2021 co-located with the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021). Aachen, Germany: RWTH Aachen, 2021.
- Jobin, K. V., Ajoy Mondal, and C. V. Jawahar. "DocFigure: A dataset for scientific document figure classification." 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW). Vol. 1. IEEE, 2019.
- Chaudhry, Ritwick, et al. "Leaf-qa: Locate, encode & attend for figure question answering." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2020.
- Mathew, Minesh, et al. "InfographicVQA." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.
- Kembhavi, Aniruddha, et al. "Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension." Proceedings of the IEEE Conference on Computer Vision and Pattern recognition. 2017.

additional feedback: - The paper argues that the challenge of the FETA benchmark data is due to the bias towards natural image photos in common training sets rather than sketch-like and synthetic images. If this is the case, what is the intuition for the fact that freezing the image encoder to pretrained CLIP-weights leads to the best model performance? This seems to point to the text representations needing the most adjustment compared with the pretrained models to address this task.
- The paper has a fair amount of typos and proofreading issues (e.g. inconsistent “json” and “JSON”, “pdf” and “PDF”, “resutls" in all of the table captions, etc.)
- There is an inconsistency between main text and supplementary material in terms of the manually annotated dataset: are there 15 or 18 manually annotated documents?
- I understand that space limitations make it difficult to include details on both the car manual and Ikea datasets, but additional comparison of these data sets to understand their commonalities and differences and how they both relate to “expert tasks” would be informative. For example, Ikea catalogues seem more like natural images than the car manuals, which appears to lead to less improvement from the baseline CLIP models relative the improvement seen for the car manuals.
review_title: Dataset addresses important gap but lacks sufficient diversity

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes FETA, a retrieval benchmark with a dataset to evaluate Vision and Langauge Pretrained models (e.g. CLIP) on the downstream tasks including car service manuals and IKEA catalogs. Since prior works focus on images of common objects, the authors claim that FETA is the first benchmark aiming at technical documentation with illustrations. They indicate that there is still a large domain gap between previous pre-trained datasets(e.g. ImageNet) and their presented dataset. Also, they extract PDF data based on the DeepSearch tool and propose an automatic annotation of text-image pairs. They evaluate the pre-trained CLIP model on FETA and analyze the experiments by four kinds of settings: zero-shot, one-shot, few-shot, and many-shot. The result shows the effectiveness of Multiple Instance Learning, while there is still room for improvement.
strengths: 1. This paper proposes a new retrieval dataset for the understanding of expert documents.
2. They propose an automatic text-image pairs extraction pipeline for any kind of PDF file.
3. With experiments in different settings, they show the effectiveness of Multiple Instance Learning on retrieval tasks.
weaknesses: 1. This paper provides the statistics of the FETA including the number of documents for each car manufacturer and the average text/image numbers in these documents. Nevertheless, each car manufacturer may provide different content in their car manuals. The paper should give a more detailed analysis of the text and images in the documents.   
2. Although the authors provide the IKEA-catalogues dataset and provide an experimental evaluation of it, they do not provide any analysis of it.
3. After downloading the dataset, the compressed file includes several directories of car manuals without any files in them. The compressed file also includes IKEA images but no texts. The supplementary material should provide a more clear guideline to use FETA.
correctness: The authors indicate that there is a large natural domain gap between their expert domain data and common object data. However, based on their ablation study of parameter locking, the result shows that locking the image encoder can achieve the best performance. If the domain of images in the expert data is different than the domain of images in the previous dataset, locking the image encoder seems like a contradiction.

Further, from Table 1, the statistics show the average number of documents and the average number of text/images in documents. But the number of pages and the number of text are too disparate. Take Chevrolet as an example. The car manual has the most pages but less text and images. I am curious about the content in Chevrolet's car manuals.
clarity: The paper is fairly well written and contains several grammar and spelling errors. For example:

Line 97: comprising of -> comprising

Line 123: ... within the same page these cases...-> within the same page, these cases...

Line 153: stets -> sets

Line 232: evaluated the performance or ->  evaluated the performance on

Also, using questions in academic papers may not be common, and asking an open question is not appropriate.

eg. Line 45, Line 151

Further, when we mention the supplementary material in the main paper, it is better to provide the section number and the hyperlink.
relation to prior work: The authors introduce the Vision and Language tasks and Multiple Instance Learning. They also explain and cite several methods of image-text retrieval. However, this paper should cite more datasets about image-text retrieval (e.g. COCO, Conceptual Captions) and compare their difference.
additional feedback: Nothing to add here. The sections above contain the main points I want to discuss.
review_title: A text-to-image and image-to-text retrieval benchmark to evaluate foundational models' performance on expert tasks such as car manuals.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: **UPDATE**: After reading all reviews and discussions, I remain positive about acceptance. I agree with other reviewers that the automated labeling procedure is a source of potential label noise in the dataset, but I believe the authors have taken reasonable steps to verify the annotations.

----------

The submission introduces a benchmark called "Foundational models for Expert Task Applications" (FETA) which evaluates the performance of foundational models (FMs) for vision and language on expert tasks whose data distributions are very different from the data distribution seen during training. More specifically, the benchmark focuses on matching illustrations to their textual description in technical documentation for car service manuals and IKEA yearly catalogues.

The PDF source documents are parsed using IBM's Deep Search tool to extract images and text, and images are automatically paired with up to five text snippets from the same page, resulting in images being associated with multiple pieces of text (and vice versa). A small subset of the data is also manually annotated to validate the correctness of the automatic annotation procedure.

The submission uses the pre-trained open-source CLIP model for baseline evaluation along with several extensions which adapt the model to the FETA data distribution in varying data regimes. To address the many-to-many nature of the benchmark's annotations, the paper proposes to consider multiple instance learning (MIL)-inspired loss formulations. Parameter-locking approaches are also investigated in an ablation study. Performance is measured in terms of recall@{1,5,10} for both image-to-text and text-to-image retrieval accuracy.

Results on the manually-annotated subset are shown to be consistent with results on the automatically-annotated data. CLIP without any adaptation is shown to underperform FETA's expert domain data, and conversely the adaptation strategies investigated improve model performance, especially in the case of MIL-inspired strategies.
strengths: - The submission is clear and well-written.
- The benchmark highlights failure modes in vision and language foundational models.
- Many adaptation strategies are evaluated.
- The submission contains a high level of implementation details and provides reference code.
weaknesses: - Baseline results are very CLIP-centric, although results for FLAVA are presented in the supplementary material.
- Not a whole lot is said about the choice of evaluation metric in the main text. The submission would benefit from a paragraph justifying this choice.
correctness: I don't see any significant correctness issue with the submission.
clarity: The submission is well-written and clear.
relation to prior work: The submission's discussion of the foundational models, vision and language, multiple instance learning, and image-text retrieval literature is thorough.

The inability of foundational models to perform well on tasks way outside of the training distribution being one of the submission's core observations, it would be relevant to discuss how this echoes findings in other subfields such as cross-domain few-shot classification, domain adaptation and generalization, and transfer learning.
additional feedback: Overall this is a strong submission from my perspective. The two key aspects that would strengthen it would be to elaborate on the choice of evaluation metric and to expand the discussion of related work to draw parallels to other subfields in terms of generalizing to very different downstream tasks.
review_title: Review

Meta-review: metareview: Looking at the rebuttal, the authors have addressed the main concerns of the reviewers. Overall work is novel and beanchmarks in experimental section add a lot of value.
confidence: 5: The area chair is absolutely certain
Decision: Accept