Synchronous Scene Text Spotting and Translating

Authors: Qiuyue Ma, Yaping Zhang, Yu Zhou
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: multimodal machine translation

Abstract:
Text image machine translation aims to translate the content of textual regions in images from a source language to a target language. Compared with traditional document, images captured in natural scenes have more diverse text and more complex layout, posing challenges in recognizing text content and predicting reading order within each text region. Current methods mainly adopt pipeline pattern, in which models for text spotting and translating are trained separately. In this pattern, translation performance is affected by propagation of mispredicted reading order and text recognition errors. In this paper, we propose a scene text image machine translation approach by implementation of synchronous text spotting and translating. A bridge and fusion module is introduced to make better use of multi-modal feature. Besides, we create datasets for both Chinese-to-English and English-to-Chinese image translation. Experimental results substantiate that our method achieves state-of-the-art translation performance in scene text field, proving the effectiveness of joint learning and multi-modal feature fusion.

Decision: reject
Reviewer scores: 3, 3, 5, 5

Reviews and discussion:
<thread>
creation date:
1729994113091

thread_type:
official_review

<thread_object>
creator: reviewer_cjtp

<content>
summary:
This paper introduces a unified framework for scene text image spotting and translation. A BAF Module is proposed to connect and integrate visual and textual features. Additionally, the authors have created the Scene TIMT (Scene Text Image Multilingual Translation) datasets for both Chinese-to-English and English-to-Chinese translations. Experimental results demonstrate the effectiveness of the proposed method.

strengths:
1. This paper proposes a unified framework designed to enhance the performance of text image machine translation.
2. This paper introduces a new Scene TIMT (Scene Text Image Multilingual Translation) dataset for this field.

weaknesses:
1. The proposed method involves a larger set of parameters compared to other methods.
2. When comparing with multimodal large models, the analysis lacks a comparison against the most recent large multimodal models, such as mplug-owl (CVPR 24), Monkey (CVPR 24), and InternVL (CVPR 24). It would be beneficial to have the performance results of the large multimodal models after they have been fine-tuned using the corresponding dataset.
3. The authors claim that “In this pattern, translation performance is affected by propagation of mispredicted reading order and text recognition errors.” However, there is a lack of experiments or evidence to verify this claim.
4. Lack of experiment to verify the effectiveness of proposed BAF Module using different text spotter and Translation module.
5. There is a lack of experimental validation to show how a unified framework improves different modules. Does it provide a greater improvement for end-to-end text spotting or for translation?

questions:
Why is it necessary to unify end-to-end spotting and translation? The author did not emphasize the necessity of unification in the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730188214853

thread_type:
official_review

<thread_object>
creator: reviewer_wqqr

<content>
summary:
This paper proposes a synchronous text spotting and translating approach for Scene TIMT by jointly training the model with three sub-tasks, detecting text regions in the image, recognizing the source language text content, and translating it into the target language.
In order to solve the problem of mis-ordered text recognition and translation that occurs in existing pipelined methods, the paper introduces a “Bridging and Fusion (BAF)” module to more effectively utilize visual and textual features to achieve efficient scene text translation. The method is evaluated on the self-constructed STST800K dataset and compared with existing methods.

strengths:
1) The paper proposes that the synchronized training approach can improve the sequential reading accuracy of translation compared to the traditional pipeline model. 
2) The proposed BAF module is creative and helpful in fusing visual and textual features.
3) The paper creates a large-scale dataset containing 800,000 samples of Chinese-English translations with real and synthetic data annotations, providing a new benchmark for scenario-based text translation tasks.

weaknesses:
1) For text recognition and translation of complex scenes, especially fine-grained and layout complexity is not reflected in the paper. 
2) The way of synchronized training is not clearly described. The model's training data is large, and the resources consumed are not mentioned. Large datasets lead to more resources consumed for training models.

questions:
1) How does the BAF module affect the reliability of the translated output in cases where errors occur in detection or recognition? Could more error analysis be provided to help understand the effects of the fusion of visual and textual features?
2) How about the recognition and translation performance of the model in more complex scenarios?
3) Can the model be applied to translations in languages other than Chinese and English?
4) Can there be a more detailed explanation of the construction of the dataset？
</content>
</thread_object>

</thread>
<thread>
creation date:
1730428276905

thread_type:
official_review

<thread_object>
creator: reviewer_ibpm

<content>
summary:
This paper proposes a method for synchronous scene text recognition and translation, which introduces a Bridge & Fusion module to integrate text and image modalities. Additionally, the paper collects multiple datasets to construct a scene text image machine translation (TIMT) dataset. The real data includes manually annotated test sets from ReCTS and HierText, while other datasets are annotated using the Qianwen API, followed by manual correction. The synthetic datasets are created based on existing tools. Experimental results demonstrate that the method in this paper outperforms existing text image translation models.

strengths:
1.	A method is proposed that can perform text spotting and text image machine translation, achieving better performance than previous methods.
2.	A Chinese-to-English and English-to-Chinese TIMT dataset is constructed based on multiple existing datasets, which includes annotating real data and synthesizing data using existing tools. This dataset is beneficial to the development of the TIMT field.

weaknesses:
1.	The related work section needs adjustment. This paper mainly focuses on TIMT, yet the authors provide very little introduction to this field in related work, instead offering a large amount of introduction on text spotting.
2.	The paper is somewhat difficult to read and does not provide some key experimental settings. For example, the setting of $\alpha$ in Equation 2 and the size of the Qwen-VL model are not specified. Additionally, all models labeled as 'ours' in the experimental results are bolded, yet they are not the best, which is quite confusing. This result does not align with what the paper claims as SOTA.
3.	The novelty is limited. The Bridge & Fusion module essentially extracts visual features based on the predicted text region's coordinates and then obtains multimodal features through cross-attention. This approach is very common in multimodal machine translation.
4.	The paper claims that its model is end-to-end, however, actually the model still requires autoregressively generating coordinates and recognition output first, and then combines them with the image to autoregressively generate the translation. Therefore, it is not fully end-to-end.
5.	The paper illustrates the issue of incorrect reading order in the pipeline method shown in Figure 1. However, the proposed method does not address this problem but merely provides such training data. Given this type of data, can the text spotting model in the pipeline also solve this issue?

[1] PEIT: Bridging the Modality Gap with Pre-trained Models for End-to-End Image Translation

questions:
1.	What is the size of the Qwen-VL model being compared, and what are the differences in settings with AnyTrans [1]? It would be better if compared with AnyTrans.
2.	What is the specific setting of alpha in Equation 2? Is the model training sensitive to this parameter?
3.	Will the data and code be open source?

[1] AnyTrans: Translate AnyText in the Image with Large Scale Models.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730820119179

thread_type:
official_review

<thread_object>
creator: reviewer_nmxn

<content>
summary:
In this work, the authors propose a scene text image machine translation method, which can detect, recognize and translate text in natural scene images. They also create image datasets for both Chinese-to-English and English-to-Chinese image translation tasks.

strengths:
The curated dataset STST800K can be beneficial to the community.

weaknesses:
1. Considering the architecture and pipeline of the proposed model, the novelty of it is limited.
2. The details of casting existing baselines (such as ABCNetv2, SPTSv2 and UNITS) as text spotter and translator are very crucial, but they are absent in the paper.
3. The comparison in Tab. 5 is unfair. The proposed model is pre-trained with data from various sources (such as STST800K and WMT22) and fine-tuned on down-stream datasets, while the Qwen-VL model is not.

questions:
Please address the questions and concerns in the Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733111762327

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 1 - Novelty Limitation

comment:
Reviewers nmXN and iBpM raised concerns about novelty limitation.

>nmXN *“Considering the architecture and pipeline of the proposed model, the novelty of it is limited.”*

>iBpM *“The novelty is limited. The Bridge & Fusion module essentially extracts visual features based on the predicted text region's coordinates and then obtains multimodal features through cross-attention. This approach is very common in multimodal machine translation.”*

***

While we admit that there is a limitation in terms of architecture novelty, we would like to highlight the innovation of our work in the following three aspects.

**1) Data:** All reviewers have acknowledged the potential future benefits that our dataset will bring to the community. Unlike existing datasets in the scene text field, our dataset places a strong emphasis on the significance of reading order in translation tasks. As shown in the table below, our dataset stands out with its greater text richness and layout complexity compared with the latest dataset, AnyTrans[1], which also incorporates reading order. Additionally, it exhibits a remarkable magnitude in the number of images. It could serve as a benchmark for in-depth validation of model structure and training methods in the field of scene text image machine translation. 
| Dataset         | Paras per image | Words per para(min/mean/max) | Non-scanning ratio |
|---------------|---------------|----------------------------|------------------|
| AnyTrans: zh2en | 1.38            | 1/1.08/3                     | 0.38%              |
| Ours: ReCTS     | 2.41            | 1/2.25/39                    | 18.03%             |

Paras per image: paragraphs number per image; Words per para: words number per paragraph; Non-scanning ratio: number of paragraphs with non-scanning reading order / total paragraphs number, representing layout complexity.

[1] AnyTrans: Translate AnyText in the Image with Large Scale Models.

**2) Archetecture:** The synchronous architecture aligns with the human thought process of first understanding image content and then translating. Our method, based on existing architectures, specially addresses the layout issue in translation. Besides, it enhances the query of attention in BAF by using both position and recognition embedding to search visual feature. With moderate model parameters, our translation performance outperforms both large multi-modal models (VLM) trained on document tasks and small models trained on specific image translation tasks (pipeline and E2E) as shown in the table below.

| Description | Method               | Parameters | OCRMT30K |         | ReCTS   |         |
|-----------|--------------------|----------|--------|-------|-------|-------|
|             |                      |            | B        | C       | B       | C       |
|             |                      |            |          |         |         |         |
| VLM         | TextMonkey           | 9.7B       | 2.60     | 65.46   | 1.61    | 57.80   |
| VLM         | mPLUG-OWL2           | 8.2B       | 0.90     | 61.06   | 0.59    | 53.61   |
| VLM         | InternVL2-8B         | 8B         | 12.62    | 75.84   | 7.46    | 67.48   |
| VLM         | Qwen2-VL-8B-Instruct | 8B         | 20.75    | 80.73   | 7.80    | 69.61   |
|             |                      |            |          |         |         |         |
| Synchronous | Ours                 | 2.88B      |**35.60** |**84.46**|**28.23**|**75.55**|
|             |                      |            |          |         |         |         |
| Pipeline    | VALHALLA             | 2.6B       | 28.12    | 70.02   | 25.68   | 69.11   |
| Pipeline    | MCTIT                | 1.58B      | 31.07    | 80.93   | 24.42   | 69.79   |
| Pipeline    | E2E-TIT              | 1.37B      | 16.30    | 42.88   | 10.16   | 39.10   |
| Pipeline    | Gated Fusion         | 0.96B      | 29.87    | 75.60   | 24.31   | 66.09   |
| Pipeline    | Selective Attn       | 0.96B      | 26.80    | 70.69   | 21.24   | 66.83   |
|             |                      |            |          |         |         |         |
| E2E         | SWINTS               | 1.77B      | 16.17    | 39.60   | 18.84   | 45.60   |
| E2E         | UNITS                | 1.33B      | 24.06    | 66.34   | 22.45   | 58.31   |
| E2E         | ABCNetv2             | 0.51B      | 2.17     | 33.69   | 4.34    | 16.14   |
| E2E         | SPTSv2               | 0.36B      | 1.90     | 38.32   | 8.50    | 36.80   |

**3) Longterm Impact:** The exploration of synchronous methodologies offers a promising approach in the future for seamlessly integrating multimodal large models with highly precise and specialized preprocessing tasks, which are often challenging for large models. Specifically, the process involves first training a text spotter and then conducting joint fine-tuning with a multimodal large model on tasks related to translation, understanding, and reasoning.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733114361414

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 2 - Compare with Large Model

comment:
According to reviewers iBpM and CjTp's suggestions:
>iBpM *“What is the size of the Qwen-VL model being compared, and what are the differences in settings with AnyTrans [1]? It would be better if compared with AnyTrans.”*

>CjTp *“When comparing with multimodal large models, the analysis lacks a comparison against the most recent large multimodal models, such as mplug-owl (CVPR 24), Monkey (CVPR 24), and InternVL (CVPR 24). It would be beneficial to have the performance results of the large multimodal models after they have been fine-tuned using the corresponding dataset.”*

We supplemented comparisons with more large multi-modal models(VLM), including AnyTrans which is trained on our scene-TIMT tasks, and also the most recent VLMs trained on general document tasks as CjTp recommended. Our model achieves SOTA in a majority of the evaluated scenarios.
- **B**LEU represents the consistency between translations and labels.
- **C**OMET represents the naturalness and intelligibility of translation results.

[1] AnyTrans: Translate AnyText in the Image with Large Scale Models.

***

**(1) AnyTrans**

Our model achieves comparable BLEU score and higher COMET score compared with AnyTrans. To keep dataset fairness, we tested our model directly on AnyTrans's dataset without any fine-tuning and compared the result with the highest score in AnyTrans's paper.
| Methods               | AnyTrans: zh2en |         |
|-----------------------|-----------------|---------|
|                       | B               | C       |
| Ours                  | 47.50           |**79.72**|
| AnyTrans: Qwen-vl-max |**48.70**        | 78.00   |

***

**(2)Most recent large multi-modal models**

- GT Coord + Ours recognition: Input ground truth coordinate to our model and use its recognition as reference input to ask VLM to do visual-textual translation, only testing the translation performance. Our model got apparent higher BLEU score than all VLMs, and only slightly lower than Qwen2-VL in  OCRMT30K - COMET.

- GT Coord + VLM recognition: Do not provide any recognition reference, just draw ground truth coordinate on the image and ask VLM to translate the text within the box, testing the ability to translate directly from image. We achieves SOTA.

- Translate Whole Image: Do not given any information of text regions, just ask VLM to translate the whole image, testing with considerations of both translation and implicit text detection ability. We achieves SOTA. 

| GT Coord + Ours recognition |            |          |         |         |         |
|-----------------------------|------------|----------|---------|---------|---------|
| Method                      | Parameters | OCRMT30K |         | ReCTS   |         |
|                             |            | B        | C       | B       | C       |
| Ours                        | 2.88B      |**35.60** | 84.46   |**28.23**|**75.55**|
| Qwen-vl-max                 | -          | 21.92    | 83.74   | -       | -       |
| mPLUG-OWL2                  | 8.2B       | 12.61    | 78.75   | 7.96    | 68.60   |
| InternVL2-8B                | 8B         | 23.06    | 84.22   | 14.47   | 75.16   |
| TextMonkey                  | 9.7B       | 9.69     | 80.85   | 5.78    | 71.00   |
| Qwen2-VL-8B-Instruct        | 8B         | 23.28    |**84.65**| 12.38   | 73.98   |

| GT Coord + VLM recognition |            |          |         |         |         |
|----------------------------|------------|----------|---------|---------|---------|
| Method                     | Parameters | OCRMT30K |         | ReCTS   |         |
|                            |            | B        | C       | B       | C       |
| Ours                       | 2.88B      |**35.60** |**84.46**|**28.23**|**75.55**|
| Qwen-vl-max                | -          | -        | -       | -       | -       |
| mPLUG-OWL2                 | 8.2B       | 0.90     | 61.06   | 0.59    | 53.61   |
| InternVL2-8B               | 8B         | 12.62    | 75.84   | 7.46    | 67.48   |
| TextMonkey                 | 9.7B       | 2.60     | 65.46   | 1.61    | 57.80   |
| Qwen2-VL-8B-Instruct       | 8B         | 20.75    | 80.73   | 7.80    | 69.61   |

| Translate Whole Image |            |          |         |         |         |
|-----------------------|------------|----------|---------|---------|---------|
| Method                | Parameters | OCRMT30K |         | ReCTS   |         |
|                       |            | B        | C       | B       | C       |
| Ours                  | 2.88B      |**23.44** |**55.73**|**24.55**|**58.65**|
| Qwen-vl-max           | -          | -        | -       | -       | -       |
| mPLUG-OWL2            | 8.2B       | 0.36     | 28.84   | 1.01    | 32.71   |
| InternVL2-8B          | 8B         | 3.52     | 49.64   | 2.21    | 50.56   |
| TextMonkey            | 9.7B       | 1.13     | 40.36   | 1.28    |  40.75  |
| Qwen2-VL-8B-Instruct  | 8B         | 2.79     | 50.19   | 1.60    | 51.98   |
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper describes an novel approach to scene text machine translation, addressing the complexities of translating textual content in natural images which possess diverse text and intricate layouts. Unlike traditional methods that separate text spotting and translation tasks, this work proposes a synchronous model that integrates both processes. The authors propose a bridge and fusion module to optimize multi-modal feature utilization and create datasets for Chinese-to-English and English-to-Chinese translation tasks.

The consensus on this paper is clear, with several significant weaknesses identified by reviewers. First, the novelty of the proposed model is minimal, with limited innovation beyond established architectures. Key details regarding the adaptation of existing baselines and the experimental settings are missing, leading to lack of clarity and questions about fairness of experimental comparisons (in particular for the pre-training and fine-tuning procedures). Moreover, while reviewers appreciate the contribution of a new dataset, the proposed approach lacks originality compared to existing multimodal approaches, and gaps in experimental validation raise concerns about the claimed advantages of the proposed framework.

In rebuttal, the authors addressed some of the questions raised by reviewers. However, there remain significant concerns about the experimental validation of the proposed approach and lingering questions about novelty beyond the proposal of a new scene text translation dataset.

Collectively, these shortcomings raised by reviewers outweigh the strengths and indicate that the work is in need of significant revision in form and substance before it can be accepted for publication, and the decision is to Reject.