ConText: Driving In-context Learning for Text Removal and Segmentation
Authors: Fei Zhang
Pei Zhang
Baosong Yang
Fei Huang
Yanfeng Wang
Ya Zhang
Venue: ICML 2025 poster (2025)
Keywords: In-context learning
text removal
text segmentation
Primary area: applications->computer_vision
TL;DR: This paper for the first time explores the visual in-context learning for some OCR tasks.

Abstract: This paper presents the first study on adapting the visual in-context learning (V-ICL) paradigm to optical character recognition tasks, specifically focusing on text removal and segmentation. Most existing V-ICL generalists employ a reasoning-as-reconstruction approach: they turn to using a straightforward image-label compositor as the prompt and query input, and then masking the query label to generate the desired output. This direct prompt confines the model to a challenging single-step reasoning process. To address this, we propose a task-chaining compositor in the form of image-removal-segmentation, providing an enhanced prompt that elicits reasoning with enriched intermediates. Additionally, we introduce context-aware aggregation, integrating the chained prompt pattern into the latent query representation, thereby strengthening the model's in-context reasoning. We also consider the issue of visual heterogeneity, which complicates the selection of homogeneous demonstrations in text recognition. Accordingly, this is effectively addressed through a simple self-prompting strategy, preventing the model's in-context learnability from devolving into specialist-like, context-free inference. Collectively, these insights culminate in our ConText model, which achieves new state-of-the-art across both in- and out-of-domain benchmarks. The code is available at https://github.com/Ferenas/ConText.

ConText:
Driving In-context Learning for Text Removal and Segmentation
Fei Zhang 1
Pei Zhang Baosong Yang"
Fei Huang
Yanfeng Wang'
Ya Zhang
Abstract
Previouss...
This paper presents the first study adapting the visual in-context learning (V-ICL) paradigm to optical character recognition tasks, specifically focusing on text removal and segmentation. Most existing V-ICL generalists employ reasoning- as-reconstruction approach: they turn to using straightforward image-labed compositor the prompt and query input, and then masking the query label to generate the desired output. This direct prompt confines the model to challenging single-step reasoning process. To address this, we propose task-chaining compositor in the ion,
Demonstration
Label
V-ICL
In-painting
Output
Query
Regiond Simple Tumina?
Curreist...
Chaining Demonstration
Image Pair
Label
form of image- Lmage-remevall reweval-segnenne providing enhanced prompt that elicits reason- ing with enriched intermediates. Additionally, introduce context-aware aggregation, integrating the chained prompt pattern into the latent query representation, thereby strengthening the model's s in-context reasoning. We also consider the is- sue of visual heterogeneity, which complicates the selection of homogeneous demonstrations in text recognition. Accordingly, this is effectively addressed through simple self-prompting strat- egy, preventing the model's s in-context learnabil- ity from devolving into specialist-like, context- free inference. Collectively, these insights culmi- nate in our ConText model, which achieves new state-of-the-arl across both in- and out-of-domain benchmarks. The code is available at https: /
?
In-painting
Chaining
Figure 1. Comparison with previous V-ICI. paradigm and pro- posed OCR OCR-adapted in-context system. Instead relying simply task-speeific tuning, our Con'Tex focuses chaining together related visual tasks to leverage their mutual benefits, thereby em- bracing powerful in-context understanding and reasoning.
powerful reasoning capabilities. Notably, in-context learn- ing (ICL) (Rubin al., 2021; Dong et al., 2022; Wies et al., 2023), method deserving of particular attention, empowers models with training-free free learning ability by using merely few input-output examples (demonstrations) as the context. This efficient -Li-inspired paradigm has sparked interest among computer vision researchers, leading the explo- ration of visual in-context learning (V-ICL) paradigm (Bar et al., 2022; Wang et al., 2023b;d; Bai et al., 2024), and paving the way for developing vision-centric contexts.
1. Introduction
Recent years have witnessed significant advances due the emergence of large language models (LLMs) (Brown et al., 2020; Touvron et al., 2023), enabling models with
Similar to ICL defined in natural language processing, current works have regulated V-ICL as a maasoning-from demonstration process, where an in-context prompt is de- fined an image-label pair, guiding any query image input generate its target output. To instantiate this, main- stream approaches (Bar et al., 2022; Wang al., 2023b;d; Fang et al., 2023; Wang et al., 2024) proposed to composite two image - -label pairs into single visually grid-like in- put, with one pair serving as the query by masking the label regions, and then perform mask- reconstruction based on MAE (He et al., 2022). This in in-painting based baseline
*Work was done during the internship at Tongyi Lab <ferenas @ edd.u.edducc... 'Shanghai Jiao Tong University (Yan- feng Wang and Ya Zhang with School of Artificial Intelligence) "Shanghai Innovation Institute Tongyy Lab, Alibaba Group. Cor- respondence to: Ya Zhang <ya_zhang @ @sjtu.edu.cn.
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Con'Tex:: Driving In-context Learning for Text Removal and Segmentation
nurtures an effective in-context model by implicitly learning the image-laa Image-label mapping from the given context.
mediates. In this way, this enriched demonstration is encouraged to comprehensively improve the model's ICL. capabilities with exploiting multi-task logic.
In contrast to the above studies focused on nabural-object- oriented generalists, this paper introduces the first V-ICL framework tailored for optical character recognition (OCR) tasks, including text segmentation and removal. Intu- itively, the straightforward upproach goes to task-specific fine-tuning of existing MAE-based pipelines, as seen in Wang et al. (2023d); Fang et al. (2023); Wang et al. (2024). While effective, this method often leads to single- task-centered paradigm, limiting the model to straightfor- ward input-out put mappings and single-step reasoning. This raises an intriguing question: Is immediate reasoning the most suitable approach for visual tasks? To enhance the reasoning abilities of LLMs, Wei al. (2022); Wang & Zhou (2024) have proposed to exploit the task-wise cor- relation that chaining relevant task as one holistic prompt, yielding a comprehensive reasoning with enriched multi- task information. Inspired by this, our paper explores link- ing multiple visual tasks to create a task-chaining in-context prompt. This approach explicitly enhances generalized rea- soning capabilities through the integration of multi-task ra- tionales, thereby facilitating more powerful ICL inference.
We propose ConText, the first OCR -CR-uoddeed V-ICL gener- alist that enhances features through explicit context-aware aggregation, and ensures text-level in-context learnability via self-query recognition, leading to significant advance- ments in generalized in-context reasoning.
Extensive results on several benchmarks demonstrate the general superiority and effectiveness of our method compared to all baseline V-ICL generalists and special- ists, yielding new state-of-the (SOTA) performance on both text removal (+4.50 PNSR) and segmentation (+3.34% fgloU). Surprisingly, Con'eet also emerges amazing training-free reasoning prompted from human- oriented visual instructions, sufficiently exhibiting its comprehensive in-context inference abilities.
2. Related Work
2.1. In-context Generalists Mechanistic Exploration. To uncover the mystery of ICL, many studies have extensively explored the mechanistic interpretability. Theoretical approaches formalized ICL ei- ther as an implicit functional learner using standard algo- rithms (Xie et al., 2021b; Garg et al., 2022; Akyürek et al., 2022; Li et al., 2023c), or as a meta-learner performing in- ternal gradient descent based on demonstrations (Dai et al., 2022; Von Oswald et al., 2023). Empirical studies (Min et al., 2022; Su et al., 2022; Mavromatis et al., 2023; Li & Qiu, 2023: Pan, 2023; Liu al., 2023) have examined latent feature changes with demonstration-level operations like replacement, reformatting, and ordering. Building these insights, Wang et al. (2023a); Yu & Ananiadou (2024) have concluded that label words play crucial role in extracting and synthesizing the input information within the demon- stration during ICL inference. Inspired by this input-label interplay, this paper introduces an innovative context aggre- gation module explicitly enhance the visual representation of labels, thereby augmenting the reasoning abilities of ICL.
To this end, we propose ConText, an enhanced V-ICL frame- work specifically designed for text removal and segmenta- tion tasks. Considering the implicit logic shared between these tasks, we propose restructuring the original single- reasoning prompt (input- into a mask-hhaining format. input centtt-eemsssee seg, where the masking reconstruc- tion process is applied to both task labels. This restructuring transforms our in-context generation into an end-to-end multi-task generalist. Building on the insights of Wang et al. (2023f); Yu & Ananiadou (2024), who revealed that the query label plays a crucial role in ICL reasoning by con- solidating all demonstration--eve information, we aim to enhance the model's reasoning-by-deen demoonstttttiii capabil- ities. To this end, we design the context-aware aggrega- tion (CAA) module that explicitly integrates prompt knowl- edge patterns into the query feature, thereby improving the model's 's contextual understanding. Additionally, given the inherent heterogeneity in text recognition, finding an appro- priate "same-class" in-context prompt for the query input poses a challenge. To address this, we propose a simple yet effective training technique named the self-prompting strat- egy, which periodically uses the same visual demonstration as the input query. Experimentally, this strategy signifi- cantly aids in maintaining generalized in-context reasoning, preventing the model from devolving into a specialist that reasons without demonstration. In summary, our overall contributions are as follows:
V-ICL. The advancement of visual in-context learning (V- ICL) in computer vision has been slow due to diverse and complex task types. Early attempts focused on effective V-ICL representation, with Bar et al. (2022) pioneering a composited- prommpttdgpppppiigggggg pattern using MAE (He et al., 2022) to perform mask-targeted in in-painting by concatenating one task-specific input-output pair with query-mask image pair. To enhance its in-domain per- formance, Zhang et al. (2023b); Sun et al. (2023); Zhang et al. (2024b) have focused on visual retrieval for optimal demonstration selection. Building on Bar et al. (2022), some studies have used additional curated data to train/fine-tune MAE-like models, improving ICL across tasks (Wang et al.,
images created
We propose the task-chaining prompting that enables vi- sual in-context reasoning with explicit task-wise inter-
2

---

Con'Text: Driving In-context Learning for Text Removal and Segmentation
2023b) or specific domains like segmentation (Wang al., 2023d), skeleton recognition (Wang et al., 2024), and 3D point cloud analysis (Fang et al., 2023). Beyond this MAB- based implementations, Wang et al. (2023f) enhanced the stable diffusion (Rombach et al., 2022) with ICI. via condi- tional fine-tuning, while Bai et al. (2024) explored sequen- tial modeling for visual auto-regressive generation. This paper proposes an OCR-targeted ICL minimalist based on MAE-like architecture, which enables concurrent multi-task inference by leveraging task-wise correlations.
Training Flow
Masking Tolaen
contbett himegionding
Backbore
Coottach frst Pastess
4
V1
F,
Shallow Layers
V
Spat
Fr
Compesitee 12xx
x
1)
12 11
Single tack Coutrased
Lissar Menge
2.2. OCR-targeted Specialists
4 + P
Scene Text Segmentation. Scene text segmentation fo- cuses on pixel-level character recognition, a derivative of foreground-background segmentation tasks. Initially, tra- ditional methods like thresholding (Otsu et al., 1975) and low-level features (Vo et al., 2018) struggled with complex colors and textures. Recent advances have seen deep learn- ing methods like SMANet (Bonechi et al., 2019), ARM- Net (Ren et al., 2022), and TextFormer (Wang al., 2023c), which incorporated multi-scale attention, high-level seman- tics, and enhanced text detail perception, respectively. Addi- tionally, character/line-leell discriminators (Xu et al., 2021; 2022) have been utilized. The advent of vision transformers (ViT) (Dosovitskiy et al., 2020) has led to efficient fine-
Iansking-ttte wiis
Deep Loyara
V
Par
Final Output
Puaeed Festurre
Figure 2. The training pipeline of previous V-ICL generalists (ob- ject segmentation as the illustrative task). This MAE-based frame- work formalize ICL. as an composited-image (image-label) recon- struction process. During training, the two image-label pairs serve as the mutual in-context demonstration for each other, while during inference, only one pairs' label is masked to generate the query output. This baseline possesses key characteristics integral to the foundation of establishing an OCR-targete ICI. paradigm.
grained text segmentation approaches (Yu et al., 2023b; 2024; Ye al., 2024), with Hi-SAM (Ye et al., 2024) em- ploying SAM (Kirillov al., 2023) for generalized frame- work. Unlike these discriminative models, this paper pro- poses a universally generative-based framework for the task.
3. Composited-Promptiin V-ICL Generalists As previously discussed above, most V-ICL paradigms art- fully model the in-context inference as a composited im- age in-painting process based on MAE (He et al., 2022). Formally, given an image-label pair representing the input- output (I;, Ys) E R3xxxx ; {1,..., n). where h x w denotes the image size, and is the number of data sam- ples. The final in-context visual input is generated by concatenating two image-label pairs at h-dimension, i.e.. F [Fi, Fx]
Text Removal. Text removal seeks to seamlessly re- place text with coherent backgrounds. Early one-stage ap- proaches combined text localization and in in-painting within single network using image-to-image translation tech- niques (Mirza, 2014; Phillip et al., 2017), but often left noticeable text remnants due to limitations in text percep- tion. To improve precision, two-stage methods have gained traction by incorporating explicit text segmentation mod- ules (Bian et al., 2022; Hou et al., 2022: Du et al., 2023a;b; Lyu et al., 2023) or using external text detectors (Zdenek & Nakayama, 2020; Tursun et al.. 2020; Tang et al.. 2021; Con- rad & Chen, 2021; Liu et al., 2022b) to enhance text local- ization. Additionally, strategies such as coarse-to-fine (Liu et al., 2020; Tursun et al., 2020; Jiang et al., 2022) and multi- step progressive refinements (Lyu & Zhu, 2022; Wang et al., 2023e) have been explored for more comprehensive text removal. Despite the complexity of two-stage methods, ViT- Eraser (Peng et al., 2024a) showed that a streamlined one- stage framework using VIT can outperform these methods, offering promising alternative. In this paper, we employ this method to erase images for segmentation benchmarks lacking human- annotttdd removal labels.
1k Y
E 3x322xx2 During the train-
I, Y,
ing stage, certain label areas in F are erased, yielding F F1, Fy] = [F,, Fy]. Here My E {0, 133x2kxw refers the randomly generated mask, where 0 indicates the masked areas equipped with learnable masking tokens. With this global erasing operation, these two input-output pairs, without special distinction in demonstration- mutually serve as the in-context information to support each other in reconstructing the label targets. Specifically, after forwarding through an encoder-decoder backbone, the final output is used to predict the pixel values of the origi- nally erased labels using the mean squared error (MSE) loss function. During the inference, masking tokens are applied merely to the label position of one of the input- pairs (Y; or Yj), Furthermore, to take advantage of the input image pair Fi. Wang et al. (2023b;d) proposed a feature fusion operation that integrates the features of the input im- age and output label at a shallow layer. This intuitive fusion not only brings twofold decrease regarding the memory
3

---

Con'Text: Driving In-context Learning for Text Removal and Segmentation
costs, but also enhances the in-context representation of the label by using the input-to-output correspondence. Figure? 2 presents the pipeline of these ceppposstee-rrmmptin V-ICL frameworks, which shares 3 essential characteristics:
here denotes the segmentation mask. The observed improve- ments in pilot experiment (please refer to Appendix A. for more details) verify the benefit of this task-level prompt- ing. To further exploit this advantage, we implement the mask-thhn-reconnsruuc process on Fo during the training, yielding F = [F1. Fo, Fy] = [F1. MoFo. MyFy]. With maintaining the logical task-level connection, we set the mask as Mo = My by preserving their spatial correlations. Correspondingly, we turn to a weight-shared decoder for reconstructing each task using their corresponding labels, ac- companied by different weight regularization. Reasonably, we also set the masking token to the removed query label position (O; or O,) during the inference, thereby generating all task outputs in an end-to-end manner.
Single-task--enttred. This pipeline supports in-context inference for only one task at a time. As a result, evaluat- ing multiple tasks necessitates multiple rounds of inference because of the straightforward image-label composition for constructing demonstration. Consequently, this single-task- centered mechanism is unable to leverage inter-task corre- spondence to enhance the generalized ICL capability.
Context-free fusion. The input-output fusion operation, which involves a linear addition for each image-label pair, solely integrates information within each input-output pair itself. As a result, the combined labels primarily focus on extracting the visual patterns of individual input-output pairs, lacking the learnability from the other given context.
4.2. Context-aware Aggregation: Fusing- Fusing-with- with-promps The findings (Wang et al., 2023a; Yu & Ananiadou, 2024) achieve a critical hypothetical consensus concerning the working mechanism of ICL:: the label position acts as the core for progressively extracting prior demonstration infor- mation in the shallow layers, with the final label absorbing all information (refer to Appendix A.2 for a detailed ex- planation). Clearly, this label-anchor perspective partially supports the feasibility of the baseline early-fusion in V-ICL generalists, where the final output represents the task label by semantically integrating inner-demonstration knowledge. However, simply merging the individual input-output pair features shall weaken the label representation because of the absence of outesr-emontttation fusion, yielding an insuffi- cient understanding of the contextual prompt. To address this, we propose the context-awar -wwere aggregation (CAA) to strengthen the label in-context representation. Specifically, our fusion process 2 steps to form the final labels, and I) goes to similar context-free fusion that yielding F2
Context-homogeneig The two composited pairs shall remain the same objectness. For instance, segmenting an airplane must be instructed with another airplane image- mask demonstration. This visual object homogeneity, as it is easily defined and enriched, explicitly provides diverse contextual environment, thereby leading to demonstration- sensitive V-ICL learnability (Zhang et al., 2023b; 2024a).
This paper aims to leverage the above pipeline to develop the first V-ICL model tackling two representative OCR tasks: text segmentation and removal. Beyond the intuitive task-specific fine-tuning as seen in Pan (2023); Wang al. (2024), are driven to explore targeted improvements these inherent traits to enhance in-context performance.
4. Method
Figure 3 presents the overall framework of our proposed ConText, which integrates three specifically designed mod- ules to improve the MAE-based baseline. The following part will provide detailed explanation of each module.
1 + O, + +ay Y; 1, a. O, + Y, 1, 0, +a,Y, a, +
(Fo,, Fri) =
E
R3x2hx2 where a (ax) is a learnable weight that reg- ulates the chaining prompt from the removal (segmenta- tion) counterpart (note that here we use the same nota- tion represent the latent feature for convenience). In- tuitively, F, centers on the inter-demonstration fusion. To empower demonstration aware context fusion, we propose 2) an additional cross- based module CAA as F2 = ( F₂) o ( F.)
4.1. Task Chaining: Beyond tangle-task-rrompttin As intuitively observed, there is an implicit inter-task log- ical connection between text segmentation and text re- moval: theoretically, the segmentation mask should cor- respond to the visual difference between the original im- age and its erased counterpart. Therefore, exploiting the task-level correlation shall bring expected advancement compared to mangle-task-rromptie end, we propose to recast the task demonstration by form- ing an explicit chain rather than simple input-output pair. Specifically, we denote O R3xxx as the erased image, and we define a new prompt demonstration as
o F2) E R3322x22
[ F2 FYJ]
(F
mechanism. To this
where d(query, key/value) : R3xxx + 3x denotes chared-cross-attentee
mapping to the query fea- ture. Based on a further combination as F. + Fz, each context-free label could be explicitly enhanced to extract the information from other demonstrations, resulting in a more comprehensive context understanding.
I O Y,
F= [F, Fo, Fy] =
E R3x2hx3u, where Y
1 O Y,
4

---

Con'Text: Driving In-context Learning for Text Removal and Segmentation
to Find Pesasraatiion
Howt
Quyry
CAAA
Botbbre
ETF
02
P2
R
F2
IF Prisee fock envif
4
f
Sasiknole Teck
Ligis
Dee
PMas
Fackmatffll 127.35
Figure 3. The training pipeline of ConText, V-ICL framework for text removal and segmentation, enhances the baseline by leveraging inherent characteristics. We create an end-to-ent multi-task generation through task-chaining. Additionally. our context-awart aggregation (CAA) enhances label representation towards better in-context understanding. We also employ a self relf-prompting strategy to ensure in- context learnability for text recognition. During inference, masking tokens are merely used for query removal and segmentation labels.
4.3. Self-prompting: Servicing In-context Learnability In scene text recognition, the inherent heterogeneity of text more complex compared to natural scene object recogni- tion. This complexity arises from the diversity in fonts, styles, languages, and contexts, posing significant chal- lenges for achieving homogeneous in-context demonstra- tions (Karaoglu et al., 2012). Consequently, employing a baseline training strategy with randomly selected exam- ples may cause a generalist V-ICL model to degrade into task- sse--speifiic one. This was experimentally validated in Section 5.3, as only minor performance differences were observed between ground-truth-basee and random demon- stration. To address this, Souibgui et al. (2021); Sahay & Coustaty (2023) have introduced the "few-shot" learning concept in OCR tasks, suggesting the use of fragments of the query image itself as effective demonstrations. Inspired by this, we propose constructing model inputs by using two identical input-output pairs (F. = =F)) with a certain proba- bility. This approach is expected to enable the ICL model to maintain both task-specific reasoning and generalized in-context learnability. While this self-prompting strategy may seem simple, we emphasize that it is a crucial training technique for preserving the text-targeted ICL ability.
we follow the prevailing pipelines (Du et al., 2023b; Peng et al., 2024a) and adopt two datasets: SCUT-EnsText (Liu et al., 2020), and SCUT-Syn (Zhang et al., 2019), where the latter one is group of artificially synthesized data. Additionally, we incorporate HierText as another bench- mark for this task by using the annotation from (Zhu et al., 2024). To evaluate the performance of removal, we use seven commonly-used image generation metrics: PSNR, MSSIM. MSE, AGE, pEPs, pCEPs, and FID. Note that fgloU, MSSIM and MSE are presented in (%) in this paper.
Implementation Details. To train generalist, we have two different pipelines based on the training data volume: i) ConText with HierText train set & ii) ConTextV with (Hi- erText + TextSeg + TotalText + SCUT-EnsText) train set. Here i) works for conducting the ablations of the designed modules, and ii) serves as task-specific comparison with the prevailing specialists. We use Adam optimizer (Kingma & Ba, 2015) and cosine learning rate scheduler, accompa- nied with base learning rate of 0.0001, and weight decay of 1. The training epoch is set to 150, and the batch size is set to 2 with a two-step gradient accumulation. We adopt 16 A A10 (80GB memory) to implement the training proce- dure, leading to total batch size of 64. As the choice of visual demonstration shall have a considerable impact dur- ing the in-context inference (Rubin et al., 2021; Zhang et al., 2023b), we report the model's 's performance averaged among 3-times trial. More details could refer to Appendix B.
5. Experiments
5.1. Experimental Settings
Tasks & Benchmarks & Evaluation Metrics. Our work centers on two representative pixel-level OCR tasks, text seg- mentation and text removal. For text segmentation, we, fol- lowing the majority of the pipelines (Yu et al., 2023a; Wang et al., 2023c; Yu et al., 2024; Ye et al., 2024), adopt four datasets with high-quality lixel-level labels: HierText (Long et al., 2022), TotalText (Ch' ng & Chan, 2017). 13 FST (Karatzas et al., 2013), and TextSeg (Xu et al., 2021). We use the foreground Intersection-over-UUnee F-score for evaluating the segmentation. For text removal,
5.2. Global Comparison Comparison with ICL Generalists. Our first experiment involves a general comparison with prevailing V-ICI frame- works, which serve as reasonable baselines. Table 1 pro- vides a detailed comparison for text removal and segmen- tation. Note that all fine-tuning-bar solely on HierText and directly evaluated on other down- stream datasets, which also demonstrates model's 's out-
methods are trained
(fgloU) and
5

---

Con'Text: Driving In-context Learning for Text Removal and Segmentation
Table Comparison with different V-ICI. frameworks against several text removal (Rem.) and text segmentation (Seg.) benchmarks.
Text Removal (PSNR* FFDJ)
Text Segmentation (fgloU 1)
Mothod
A
Hie/Text
SCUTTEEssTat
"SCUT-Sya
Hie/Text
TTotalText
"FST
TTexiSeg (vol)
No Fine-tuning Baselines
MAE-VQGAN (Bar et al., 2022)
2022)
28.52/ 32.71
29.12/44.58
27.25/45.81
28.30/41.70
1,93
5.83
6.57
13.54
6.97
Painter (Wang al., 2023h)
17.11
26.29 26.29/52.08 52.08
24.07/ 55.60
24.35 51.28
4.08
6.01
4.88
9.70
6.17
SegGPT (Wang et al., 2023d)
3.12
9.58
9.45
25.36
11.88
Task-specific Fine-tuming (-+) on Hiertest
Painter Rem.
26.14/31.09
36.15/21.37
33.85/29.30
32.05 27.92
SegGPT - Seg.
60.60
65.10
59.12
75.75
65.14
Painter + Rem. + Seg.
28.17//444 24.76
36,48/ 36,48//1.00 /21005
34.38 / 28.38
32.34/ 24.73
64.72
67.81
61.09
77.02
67.16
SegGPT Seg.
25.51
/1.19
34,42/ 28.32
33.05
65.23
77.40
68.34
ConText
39.48 6.35
37,67 12.887
37.93/ 13.91
38.36 11.04
74.86
78.02
71.02
82.31
76.77
Prompe
1100g
Painter
KiNUT
SegGPT (om..
Qurty
ConText
KoOUT
Ground Truth
iT
Figure 4. Visualized TotalText samples generated from several in-context generalists. Each set of samples consists of the (original image, removal result, segmentation map). The prompt refers the given visual demonstration. The red circles indicate the differences segmentation and removal results between our method and the fine-tuned SegGPT. Zoom in for better view.
of-domain generalization capability. For spsc-ip fine- tning methods, we strictly adhere to the corresponding fine-tuning settings, Clearly, our proposed method outper- forms other models, achieving a PSNR of 38.36 and an FID of 11.04 across three text removal benchmarks, and an average fgloU of 76.77% for four text segmentation datasets. These results are significantly higher than those of non/task-specific fine-tuning baselines. The visualized results in Figure 4 further illustrate the superior in- in-context ability of our model, yielding better removal effects and segmentation masks compared to others.
5.3. In-context Specificity
In-context Learnability. One unique attribute of in-context learning is its infer-by-prompt capability, yielding different levels of reasoning ability. In other words, an in -n-context model should be sensitive to the demonstration in terms of downstream task. Therefore, to verify this prompting flexi- bility, Figure 5 reports all models' performance given both the randomly-selected and ground-truth--un tion samples, yielding the upper and the normal in-context inference abilities. Particularly, all models are unable to implement flawless reconstruction even when being prompted by the ground truth. Therefore, it is emphasized that the improvement of both the upper and lower performance important to ICL models since current vision models are far from performing promising in-context learnability compared with those powerful LLMs. Based these results, direct fine-tuning, as yielding significant improvement, could lead to invalid in-context learnability due to the minimal performance change regardless of the demonstration. However, our method exhibits strong per- formance range, with an averaged difference of +1.01 PSNR in text removal and +5.39 fgloU in segmentation. With the scaling of training benchmarks in our ConTextV, such gap is further accentuated the collective improvements of upper- and lower- pwwer-pefformanc. This indicates the model's capacity to adapt effectively even when demonstration sam-
demonstra-
shown in this figure,
Comparison with Task-Specific Specialists. Here, we present stronger version ConTextV conduct compre- hensive comparison against task-specific specialists. As shown in Table 2 & 3, ConTextV demonstrates an over- whelmingly superior performance in both text segmentation and removal tasks. In Table 2. with the support of just one randomly-selected demonstration, our method yields an av- erage improvement of +2.53% % in fgloU, compared to other state-ofthhe-ee (SOTA) methods. Notably, our approach achieves significant improvements on the previously unseen FST dataset, outperforming data-specific specialists. For text removal, as shown in Table 3, our method also achieves superior crasing effects compared to removal specialists in the SCUT-Ens Text dataset. The related experimental results for SCUT-Syn can be found Appendix B.3.
6

---

Con'Text: Driving In-context Learning for Text Removal and Segmentation
Table Comparison with the text segmentation specialists among four benchmarks. *FST dataset not used for training in our model.
HierText
TotalText
FST
TextSog
Method
fgloUt
F-scoref
Fg
F-score
fgloUt
F-scoret
fgloU?
F-score
SegFormer (Xic et al., 2021a)
73.31
0.846
60.44
0.753
84.59
0.916
D+epLabV3+ (Chen et al., 2018)
74.44
0.824
69.27
0.802
84.07
0.914
HRNe1V2-W48 (Wang ct al., 2020)
75.29
0.825
70.98
0.822
85,98
0.918
HRRNe
W4&+OCR IIRNeIV--WA++OCR
(Wang et al., 2020)
76.23
0.832
72.45
0.830
85,98
0.918
TexRNet + DeeplabV3+ (Xu et al., 2021)
76.53
0.844
72.16
0.835
86,06
0.921
TexRNet+ + HRNetV2-W45 (Xu et al., 2021)
55.50
0.656
78.47
0.848
73.38
0.850
86.84
0.924
TFT (Yu et al.. 2023a)
82,10
0.902
72.71
0.845
87.11
0,931
F.Former (Yu et al., 2024)
82.73
0.906
72.63
0.840
88.06
0.939
UPOCR (Peng et al.. 2024b)
88,76
0,940
Hi-SAM (Ye et al., 2024)
77.76
0.848
84.59
0.887
88.77
0.938
ConTextV
81.21
0.896
85.19
0.919
75.90
0.873
89.74
0.946
Table 3. Comparison with the removal specialists.
Text Segmentation (fgloU/t)
wRs Prorspt
eGT Pronse
Method
SCUT=EnsTest
Contectv
PSNR1
MSSIM-
MSE]
AGEL
pEIPs
PCBPs]
Contest
Pix2Pix (Phillip et 2017)
26.70
98.56
0.37
6.09
0.0480
0.0227
46.88
SagaPT Ron Sag
STE (Nakamarn et al. 2017)
25.47
90.14
0.47
5.033
0.0331
0.0296
43.59
EmNeeT (Zhung
2019)
0.0007
0.0136
32.71
Painter Rom Sog
13.3
MTRNN (Tursun et al., 20100
20.63
93.71
0.23
3.51
0.0305
0.0168
35.50
BegOPT
ErasNeT (Liu et al. 2020)
32.30
95.42
0.15
3.02
0.0160
0.0090
19.27
SSTE (lang et al. 2021)
35.34
96.24
SegOPT
PSSTRNet (1 & 7hu, 2022)
34.65
96.75
0.14
1.72
0.0135
0.0074
CTRNet (Liu el al.. 2022c)
35,20
97,36
0.09
2.20
0.0106
0,0068
13.99
GaRNet
Chot. 2022)
97.14
1.90
0.0105
0.0062
MAE-VOGAN
+12.9
MBE (Hou et al. 2022)
35.03
97.31
2.06
0.0128
0.0088
PEN (Du 2023c1
35.72
96.68
1.95
0.0011
0.0020
PERT (Wang at. 2013e)
33.62
97.00
0.13
2.19
0.1355
0.8088
SAEN (Du et al., 2025a)
34.75
96.53
0.07
1.98
0.0125
0,0073
Text Removal (PSMR())
Prompt
FETNet (Lys al., 2023)
34.53
97.01
0,13
1,75
0.0137
0.0080
ConTectv
ViTBriser (Peng
20244)
36.87
97.51
0.05
1.22
0.0066
0.0035
10:15
UPOCR (Peng et al., 20046)
37.14
97.62
0.04
1.72
0.0064
0,0034
10.47
ConTect
ConTextV
40.83
98.76
0,03
0.76
0.0053
0.0029
11,03
SayOPT Ren
ples differ from the ground-truth. Overall, the proposed model exhibits both strong upper and lower performance bounds compared to other methods, highlighting its seal- ability and versatility. Appendix B.2 presents the specific numerical results for each benchmark, which intuitively demonstrate our model's powerful in-context learnability.
Figure 5. Comparison of several visual generalists on text segmen- tation and removal tasks when given the randomly-selected (RS) and souundtruthh-basee (GT) prompts. Here the segmentation (re- moval) task is evaluated against four (SCUT-Ens Text) henchmarks. This two-case performance range denotes a model's substantial in contenext lcarnability towards these tasks, where a sounding upper and lower bounds indicates strong scalability potential.
In-Context Understanding, As claimed above, one of the amazing advantages of ICL is providing a flexible user- oriented interaction with models. To further evaluate the generalized inference capability of our model towards the given demonstration, we specifically construct a dataset with explicit visual markers, namely PromptText, including randomlyycolooo collored circle, stroke, and box, to mimic the user behavior on demonstration to segment and crase as required (please refer to Appendix B.4 for more details about the con- struction of this dataset). Particularly, such a dataset, merely serving as an evaluation benchmark, not saiiiing-invoved.. Table 4 shows the results of this prompting dataset among several methods, and clearly, our method has achieved an overall promising performance when compared to all other methods. For those specialists, with reflexively performing text segmentaaionn/removal, their low performance is reason- able due to the lack of understanding towards the explicit prompting. The visual generalists, despite having limited
prompt comprehension, also exhibit subpar text recognition capabilities. In contrast, our methods surprisingly demon- strate thorough understanding of these explicit prompts, resulting in superior segmentation and removal performance (as shown in Figure 6). This experiment also highlights the value of exploring visual in-context inference for text recognition, driving more adaptable form of user-model interaction. Besides, we argue that this experiment also reveals the visual-cues prompting ability of our model. As stated in Shtedritski et al. (2023); Yang et al. (2023), an emerging ability of recognizing explicit visual hints has been explored for current foundation models, enhancing the fine-grained and localized recognition capability through
7

---

Con'Text: Driving In-context Learning for Text Removal and Segmentation
DER POMM Revamp Bathrooms
DER FO Revamp Bathrooms
RoOr PMr Revamp Bathrooms
DBR Foe Revamp Bathrooms
RERLUER Revamp Bathrooms
DEP-AYRCO ns Revamp Bathrooms
Table 5. Fffectiveness of designed items on our method. The seg- mentation (removal) is evaluated via TotalText (SCUT-ENS) based on fgloU (PSNR). RS (GT) denotes the model's performance with randomly-selected (ground-truth) demonstration.
DER ROOM FOR Revamp Bathrooms
Reva B
0
p
B th
mp
B th
mp
mp th
Bascline
Lincar Fusion
(F))
(F. + F2)
CAA
SP-0.2 SP-0.6
RS/OT
Sep.
Rem.
GT
ROOMFO m athr
DEP 1000
DER Jn FOOM FOR
DISPL ui REOW Fea Revamp Bathrooms
DEP veco 2
(8.55/+1.57
34,42/+0,17
Reva a rooms
Revamp Bathrooms
Reva a rooms
72.14/+1.088
35.75/ 40.41
79.14/+0.65
38.59/40.33
78.02 +3.98
17.67 41.42
47.14/45.83
36.12/42.13
KENDMELLS Ayannger
KINIONESS Channyy
KINIONEER
KINONESS
KONGOEESS Chynengr
KINDNESS Araaaan
Grarya
Table 6. Effectivennes the masking ratio value (%) on our pro-
posed method. RS (GT) denotes the model's performance with randomly-selected (ground-truth) demonstration.
ON
KINDDEERA
DN
ch
Y
TotalText Seg.
SCUT-Ens Rem.
Masking Ratio (%)
R$ / GT
RS GT
KINDNESE
KINGNEES
75.80/ / +2.34
36.15 +1.39
chiangr
a
35
76.23 / +3.02
36.67 / +1.72
55
77.45 4
36.89 / +1.53
Demonstration
Input
Specialists
Painter
(Rem. + Seg.)
SegGPT
ConTextV
75
85
78.02 / 43,98
77.74 /2284
37.67 / /1.68
37.21 / +1.62
Figure 6. Visualized samples of several methods on the prompting datasets, The results of the specialists are obtained through Hi- SAM and ViT-eraser, respectively. Other generalists are prompted by the given demonstration. Zoom in for a better view.
95
78.04/ +3.04
36.83/ +1.29
(F1) yields a significant improvement (+3.61 %) fgloU for segmentation and +1.33 PSNR for removal) on both down- stream tasks, demonstrating the benefits of context fusion. Based on this, our enhanced context fusion could drasti- cally improve the model's performance by achieving an elation of +10.61 % fgloU and +4.17 PSNR, which strongly verifies the superiority of our proposed method. However, similar improvements have not been observed in our model when given the ground-truth as the demonstration. Different from the natural object-level recognition, the text recogni- tion, though comprised of merely binary units, is difficult to define its visually homogeneous counterpart. Therefore, without the guidance from proper demonstration, these de- signed modules shall drive the original in-context model into a pure task-specific specialists. As shown in this table, merely a marginal fluctuation is observed between the ran- dom and ground -tuth demonstration. By introducing the self- promptingg (SP) manner, the model tends to maintain both the task-specific capacity and in-context generalization, while the over-usage of such a mechanism would degrade the model's task-specific performance due to the reduced demonstration diversity. This finding also highlights the im- portance of balancing demonstration diversity to optimize model outcomes in in-context learning scenarios.
Table 4. Comparison with several frameworks on our designed eser-prompted datasets. Note that all these xppicitly-promptiiee datasets are not used during the training for all methods, which evaluates model's in-context generalized understanding.
Method
ProptfeetRRs Res.
Promprlest Seg.
PSNRT MSSIM
FID.
fgldl!
F
7ank spec-se Steriliste
Hi-SAM (Yo al., 2014)
43.51
0.621
Wi-Enser (Peag
20144)
20.23
90:25
11281
Mins Je-ontten Generaliass
Painter (Wang al. 20236))
1963
88.91
S1.11
7,50
0.140
Painter (Wing al. 20236) - Reas
24.11
90:32
68.02
ScyGPT (Wang et. al. 2013d) + Sey.
43,58
0.610
SerGPT (Wing at, 201330-
Rom
23.73
90:31
68.7
44.00
0.68
ConTeet
33.16
08
41.24
0.708
ConTear
38.14
99.06
33.59
0.741
simple but explicit visual marker on the query object. In conclusion, our model has demonstrated an exceptional training-free free generalized and recognition ability.
5.4. Ablation Studies
In this section, we will make relevant ablations about our method, such as the effectiveness of our designed mod- ule, performance with dealiiidemotttttinnn, and double in- context inference. Unless otherwise specified, the overall ablations are conducted by using ConText (trained with HierText).
Effectiveness of Masking Ratio. We have conducted abla- tions regarding the masking ratio at wide range (25%- -95%). As shown in Table 6, training with lower mask ratio would lead to a certain decrease on both the removal and segmenta- tion tasks, especially under the 25% case. With the growing number of crased parts, the model tends to show consis- tent improvements on both downstream tasks, reaching the
Effectiveness of Individual Module. Table 5 presents an ablation study assessing the effectiveness of various design components of our model. Here the baseline refers to the multi-task fine-tuning version of SegGPT (SegGPT - Rem. + Seg.). As shown in Table 5, the intuitive linear fusion
8

---

Con'Text: Driving In-context Learning for Text Removal and Segmentation
Table Performance of our proposed Con'ext under different mondomly-sllectie demonstration number against 3 benchmarks.
Table 9. Performance comparison on CLWD (Liu et al., 2021).
Method
Rem. (PSNR) Seg. (fgloU)
Demonstration
Tei
TextSeg (vel) Seg. SCUT-Ens Rem.
Number
fgloU (t)
(t)
())
SegGPT
30.11
74.42
PFMNet (Niu et al., 2023)
39,45
79.09
Molii- colli-demonssration Inference
ConText
40.73
82.16
(Baveline)
78.02
82.31
37.67
78.12 (+0.10)
82,47 (+0.16)
37.98 (+0.31)
78.64 (+0.62)
82.83 (+0.54)
38:45 (+0.78)
Double Inference
incurs an additional training burden of +0.4 minutes per epoch. However, this cost is deemed acceptable due to the moderate engagement of SP (SP-0.2) during training. Moreover, SP is not utilized during inference, yielding no additional computational burden for inference. Furthermore, CAA introduces extra computational costs during both train- ing and inference. However, as a lightweight coss-attention module, it only increases model complexity by a manage- able 2%. Consequently, it leads to a mere increase of +0.03 seconds (+0.88 minutes/epoch) during inference (training). Based on this, we can safely conclude these modules indi- cate reasonable level of computational efficiency.
78.26 (+0.25)
82,86 (+0.55)
38.11 (+0.44)
Table 8, Computational efficiency of the designed items on our method. The results are evaluated on HierText, and FL.OPs and inference time are calculated by forwarding one 512 x 512 image on one A100, with the inference time reported in seconds (sec) and training time reported in minutes (min) per epoch.
Method
Training Time
Inference Time|
Model FLOPs
Buseline Buscline SP
3.8 min
0.09 sec
666.76G
4.2 min
0.09 sec
666.76G (0%)
Buseline CAA
4.6 min
0.12 sec
683,96G (+2%)
Buseline CAA+ SP
4.8 min
0.12 sec
683,96G (+2%%
Beyond OCR. To verify the generalization of our task-in- chain concept, we (following similar training strategy) have additionally explored our Con'Tex on one prevailing water- mark removal benchmark, CLWD (Liu et al., 2021). Table 9 reports the results of SegGPT and a leading specialist (Niu et al., 2023). Clearly, our approach demonstrates superior performance, with achieving advanced performance among both removal and segmentation tasks, which further validat- ing the lask-wise generalizability of ConText.
peak point with 85% masking ratio. However, beyond that masking proportion, the model showcases an evident per- formance decline. These results align with the conclusions of Fang et al. (2023), confirming the effectiveness of the proper application of the masking strategy.
Multi-demonstratioo/Dbbbb Inference. Table 7 presents the effectiveness of nulti-demonstration. and double- inference on our proposed framework. To achieve the former one, we follow the feature ensemble operation in Wang et al. (2023d) that first feeds the different demonstration- -uuroy pairs as a batch-wise forward process, and then averagely fuse the query features the specific each layer of the model (specific implementation could refer to Wang et al. (2023d)). As shown in this table, fusing the multi-demonsttrttoon could yield an overall improved performance when compared to normal --shot inference. However, it is observed that this use of multi-shot does not yield as much improvement as reported by Wang et al. (2023d), likely due to the heteroge- neous visual attributes involved in text recognition. Besides, the introduce of multi-demonstrattoo would increase the labeling efforts for both segmentation and removal tasks. Therefore, there exists trade-off' between model accuracy and the cost of data labeling. Another interesting trial of our inference manner is to use the first-time generated re- sults as the new demonstration to perform a second-time in in-context inference, and such a double inference is also similar kind of self-training. Table 7 showcases the ef- fectiveness of this inference mechanism, which also brings certain degree of improvement to the model's 's performance. However, considering the computational costs, we do not adopt this approach for relatively minor improvements.
6. Conclusion
In this paper, we presented, to the best of our knowledge, the first exploration of establishing a visual in-context learning (V-ICL.) paradigm for fine-grained text recognition tasks, including text segmentation and removal. To achieve this, we sought a tiale--as-ttrrettte baseline solution based on the prevailing V-ICL. frameworks, which typically reg- ulates in-context inference as a soory--lbel-rcconsttuucte process. Beyond simple task-specific fine-tuning, we pro- posed an end- end-to-end in-context generalist elicited from a lask-chaining prompt that explicitly chaining up tasks as one enriched demonstration, leveraging inter-task correlations to improve the in-context reasoning capabilities. Additionally, we introduced the context-aware aggregation (CAA) mod- ule and self- promptino (SP) training techniques to further strengthen the model's understanding of in in-context represen- tations, resulting in a significant enhancement of reasoning on heterogeneous visual patterns. Through quantitative and qualitative experiments, we demonstrated the grounding effectiveness and superiority of our framework across var- ious in-domain and out-of-domaile text recognition tasks, outperforming both current generalists and specialists. Over- all, we hope this pioncering work will encourage further development of V-ICI. in text recognition.
Computational Efficiency. Table 8 reports the additional costs of our designed modules. Clearly, we find that SP

---

Con'Text: Driving In-context Learning for Text Removal and Segmentation
Impact Statement
Textdiffuser: Diffusson models as text painters. Advances in Neural Information Processing Systems, 36, 2023.
Note that all our training datasets, and the data for training our framework are sourced from the Internet. Consequently, the collection of these datasets raises concerns regarding pri- vacy if not appropriately regulated. Additionally, the stroke and removal labels heavily rely on human annotators or the off-the-shel tools, which can introduce potential noises and biases, intentional or unintentional, if the annotators are not impartial. It is key to address these issues through proper data regulation, privacy protection measures, and meticulous selection of the annotated information to ensure fairness and relieve potential biases.
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), PP. 801-818, 2018
Ch' 'gng C. K. and Chan, C. Total-text: A comprehensive dataset for scene text detection and recognition. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 1. pp. 935 942. IEEE, 2017.
Acknowledgements
Conrad, B. and Chen, P.-I. Two-stage seamless text erasing real-world scene images. In 2021 IEFE International Conference on Image Processing (ICIP), pp. 1309-1313. IEEE, 2021.
This work is supported by the National Key R&D Program of China (No. 2022/D0110003), National Natural Science Foundation of China (No. 62306178), and STCSM (No. 22DZ2229005), 111 plan (No. BPO719010) This work is also supported by Alibaba Research Intern Program.
Dai, D., Sun, Y., Dong. L.,, Hao, Y., Ma, S., Sui, Z., and Wei, F. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. arXiv preprint arXiv:2212. 2022.
References
Akyürek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. What learning algorithm is in-context learn- ing? investigations with linear models. arXiv preprint
Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Liu, T., et al. A survey on in-context learning. arXiv preprint arXiv:2301 00034, 2022.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. /929, 2020.
Bai, Y., Geng, X., Mangalam, K., Bar, A., Yuille, A. L., Darrell, T., Malik, J., and Efros, A. A. Sequential model- ing enables scalable learning for large vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22861-22877 2024.
Du, X., Zhou, Z., Zheng, Y., Ma, T., Wu, X., and Jin, C. Modeling stroke mask for end-to-end text erasing. In Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision pp. 6151-6159, 2023a.
Bar, A., Gandelsman, Y., Darrell, T., Globerson, A., and Efros, A. Visual prompting via image inpainting, Ad- vances in Neural Information Processing Systems, 35: 25005-25017, 2022.
Du, X., Zhou, Z., Zheng, Y., Wu, X., Ma, T., and Jin, C. Progressive scene text erasing with supervvinni Computer Vision and Image Understanding, 233:103712, 2023b.
Bian, X., Wang, C., Quan, W., Ye, J., Zhang, X., and Yan, D.- M. Scene text removal via cascaded text stroke detection and crasing. Computational Visual Media, 8:273-287, 2022.
Du, X., Zhou, 7.,, Zheng, Y., Wu, X., Ma, T. and Jin, C. Progressive scene text erasing with self-supervision. Computer Vision and Image Understanding, 233:103712, 2023c.
Bonechi, S., Andreini, P., Bianchini, M., and Scarselli, F. Coco_ts is dataset: pixel-level annotations based on weak supervision for scene text segmentation. In International Conference on Artificial Neural Networks, pp. 238 238-250- 250. Springer, 2019.
Fang, Z., Li, X., Li, X.,, Buhmann, J. M., Loy, C. C., and Liu, M. Explore in-context learning for 3d point cloud un- derstanding. Advances in Neural Information Processing Systems, 36, 2023.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Garg, S., Tsipras, D.,, Liang, P. S., and Valiant, G. What can transformers learn in-context? a case study of sim- ple function classes. Advances in Neural Information Processing Systems, 35:30583-30593 2022.
Chen, J., Huang, Y., Lv, T., Cui, L., Chen, Q., and Wei,
10

---

Con'Tex:: Driving In-context Learning for Text Removal and Segmentation
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Li, S., Song, Z., Xia, Y., Yu, T., and Zhou, T. The closeness of in-context learning and weight shifting for softmax regression. arXiv preprint 23X1::2304.12276 2023c.
Li, X. and Qiu, X. Finding supporting examples for in- context learning. CoRR, 2023.
Hou, Y., Chen, J. J., and Wang, Z. Aulti-branch network with ensemble learning for text removal in the wild. In Proceedings of the Asian Conference on Computer Vision, pp. 1333-1349, 2022.
Liu, C., Liu, Y., Jin, L.,, Zhang, S., Luo, C., and Wang, Y. Erasenet: End-to-end text removal in the wild. JEEE Transactions on Image Processing, 29:8760-8777 2020.
Jiang, G., Wang, S., Ge, I., Jiang, Y., Wei, Y., and Lian, D. Self-supervised text erasing with controllable image synthesis. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 1973-1983, 2022.
Liu, C., Jin, L.,, Liu, Y., Luo, C., Chen, B., Guo, F., and Ding, K. Don't 't forget me: accurate background recovery for text removal via modeling local-global context. In European Conference on Computer Vision, pp. 409-426. Springer, 2022a.
Karaoglu, S., Van Gemert, J. C., and Gevers, T. Object read- ing: text recognition for object recognition. In Computer 2012 Workshops and Demonstrations: Flo- rence, Italy, October 7-13, 2012, Proceedings. Part III 12, pp. 456-465 Springer, 2012.
Vision-ECCV 2012.
Liu, C.,, Jin, L., Liu, Y., Luo, C., Chen, B., Guo, F., and Ding, K. Don't 't forget me: accurate background recovery for text removal via modeling local-global context. In European Conference on Computer Vision, pp. 409-426. Springer, 2022b.
Karatzas, D., Shafait, F., Uchida. S., Iwamura, M.. Bigorda, L. G., Mestre, S. R., Mas, J., Mota, D. F., Almazan, A., and De Las Heras, L. Icdar 2013 robust reading competition. In 2013 12th international conference on document analysis and recognition, pp. 1484 1493. IEEE, 2013.
Liu, J., Liu, Y., Zhang, F., Ju, C., Zhang, Y., and Wang, Y. Audio-visual segmentation via unlabeled frame ex- ploitation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26328- 26339, 2024.
Kingma, and Ba, J. Adam: A method for stochas- tic optimization. International Conference on Learning Representations (ICLR), 2015.
Liu, S., Ye, H., Xing, L.., and Zou, In-context vec- tors: Making in context learning more effective and con- trollable through latent space steering. arXiv preprint arXiv:231 1.111.0600 1.06668, 2023.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson. L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., al. Segment anything. arXiv preprint arXiv:2304.02643 2023.
Liu, Y., Zhu, Z., and Bai, X. Wdnet: Watermark- decomposition network for visible watermark removal. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 3685-3693, 2021.
Langley, P. Crafting papers on machine learning. Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (000),000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Long, S., Qin, S., Panteleev, D., Bissacco, A., Fujii, Y., and Raptis, M. Towards end-to-end unified scene text detection and layout analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1049-1059, 2022.
Lee, H. and Choi, C. The surprisingly straightforward scene text removal method with gated attention and region of interest generation: A comprehensive prominent model analysis. In European Conference on Computer Vision, pp. 457-472. Springer, 2022.
Lyu, G. and Zhu, A. Psstmet: progressive segmentation- guided scene text removal network. In 2022 IEEE Inter- national Conference on Multimedia and Expo (ICME), pp. 1-6. IEEE, 2022.
Li, B., Wang, Y., Wang, L., Zhang, F., Liu, T.,, Lin, Z., An, W., and Guo, Y. Monte carlo linear clustering with single- point supervision is enough for infrared small target de- tection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1009-1019, 2023a.
Lyu, G., Liu, K., Zhu, A., Uchida, S., and Iwana, B. K. Fetnet: Feature erasing and transferring network for scene text removal. Pattern Recognition, 140:109531, 2023.
Li, B., Zhang, F,, Wang, L., Wang, Y., Liu, T., Lin, Z., An, W., and Guo, Y. Ddaug: Differentiable data augmenta- tion for weakly supervised semantic segmentation. IEEE Transactions on Multimedia, 26:4764 4775, 2023b.
Ma, C., Yang, Y., Ju, C., Zhang, F., Liu, J., Wang, Y., Zhang, Y., and Wang, Y. Diffusionseg: Adapting diffusion to- wards unsupervised object discovery. arXiv preprint arXiv:2303.08 098//3, 2023.

---

Con'Tex:: Driving In-context Learning for Text Removal and Segmentation
Mavromatis, C., Srinivasan, B., Shen, Z., Zhang, J., Rang- wala, H., Faloutsos, C.,, and Karypis, G. Which examples to annotate for in-context learning? towards effective and efficient selection. arXiv preprint arXiv:2310.2004// 2023.
Rubin, O., Herzig, J., and Berant, Learning re- trieve prompts for in-context learning. arXiv preprint arXiv:2112.08633, 2021.
Sahay, R. and Coustaty, M. An enhanced prototypical net- work architecture for few-shot handwritten urdu character recognition. IEEE Access, 1 11:33682-33663 2023.
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M.,, Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprini arXiv:2202. /2837, 2022.
Shtedritski, A., Rupprecht, C., and Vedaldi, A. What does clip know about red circle? visual prompt engineering for vlms. arXiv preprint wrXiv:2304.06712, 2023.
Mirza, M. Conditional generative adversarial nets, arXiv preprint arXiv:1411.110 2014.
Souibgui, M. A., Fornés, A., Kessentini, Y., and Megyesi, B. Few shots are all you need: A progressive few shot learn- ing approach for low resource handwritten text recogni- tion. arXiv preprint arXiv:2 10064 2021.
Nakamura, T., Zhu, A., Yanai, K., and Uchida, S. Scene text eraser. 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume pp. 832 837. IEEE, 2017.
Su, H., Kasai, J., Wu, C. H., Shi, W., Wang. T., Xin, J., Zhang, R., Ostendorf, M., Zettlemoyer, L.., Smith, N. A., et al. Selective annotation makes language models bet- ter few-shot learners. arXiv preprint arXiv:2209.01975 2022.
Niu, L., Zhao, X., Zhang. B., and Zhang, L. Fine- grained visible watermark removal. Proceedings of the IEEE/CVF international conference on computer vision, pp. 197709 2023.
Sun, Y., Chen, Q., Wang. J., Wang. J., and Li, Z. Exploring effective factors for improving visual in-context learning. arXiv preprint arXiv:2304. 04748, 2023.
Otsu, N. et al. A threshold selection method from gray-level histograms. Automatica, 1 11(28--296:23-2222 1975.
Pan, J. What in-context learning "learns" in-context: Dis- entangling task recognition and task learning. Master's thesis, Princeton University, 2023.
Tang, Z.., Miyazaki, T., Sugaya, Y., and Omachi, S. Stroke- based scene text erasing using synthetic data for training, IEEE Transactions on Image Processing, 30:9306 93200, 2021.
Peng, D.. Liu, C.. Liu, Y., and Jin, L. Viteraser: Har- nessing the power of vision transformers for scene text removal with segmim pretraining. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 1468-4477, 2024a.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.
2023.
Peng, D., Yang, Z., Zhang, J., Liu, C., Shi, Y., Ding, K., Guo, F., and Jin, L. Upoer: Towards unified pixel-level ocr interface. In Forty-first International Conference on Machine Learning, 2024b.
Tursun, O., Denman, S., Zeng, R., Sivapalan, S., Sridharan, S., and Fookes, C. Mtmet++: One-stage mask based scene text eraser. Computer Vision and Image Under- standing. 201:103066,, 2020.
Phillip, I., Yan, Z., Tinghui, Z., Alexei, A., et al. Image-to- to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, volume 3, 2017.
Vo, Q. N., Kim, S. H., Yang, H. J., and Lee, G. Binarization of degraded document images based on hierarchical deep supervised network. Pattern Recognition, 74:568-586, 2018.
Ren, Y., Zhang, J., Chen, B.. Zhang, X., and Jin, L. Looking from higher-level perspective: Attention and recogni- tion enhanced multi-scale scene text segmentation. In Proceedings of the Asian Conference on Computer Vision, pp. -133845,,, 2022.
Von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M. Transformers learn in-context by gradient descent. International Conference on Machine Learning, pp. 35151-35174. PMLR, 2023.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffussin models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C.,, Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., et al. Deep high-resolution representation learning for visual recogni- tion. IEEE transactions on pattern analysis and machine intelligence, 43(10):3349-3333 3364, 2020.
pattern
12

---

Con'Tex:: Driving In-context Learning for Text Removal and Segmentation
Wang, L., Li, L.,, Dai, D., Chen, D., Zhou, H., Meng, F,, Zhou, J., and Sun, X. Label words are anchors: An information flow perspective for understanding in-context learning, arXiv preprint arXiv:2305. 14160, 2023a.
Xu, X., Zhang, Z.., Wang. Z., Price, B., Wang. Z., and Shi, H. Rethinking text segmentation: A novel dataset and text-specific refinement approach. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12045-12055, 2021.
Wang, X. and Zhou, D. Chain-of thaouutf reasoning without prompting. arXiv preprint 2.0.iv::200.1000000 2024.
Xu, X., Qi, Z., Ma, J., Zhang, H., Shan, Y., and Qie, X. Bts: a bi-lingual benchmark for text segmentation in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11152-9162, 2022.
Wang, X., Wang, W. Cao, Y., Shen, C., and Huang, T. Im- ages speak images: A generalist painter for in-context visual learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 0830-6839, 2023b.
Yang, J., Zhang, H., Li, F. Zou, X., Li, C., and Gao, J. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint arXiv:2310.114.: 2023.
Wang, X., Wu, C., Yu, H., Li, B., and Xue, X. Textformer: component-awand text segmentation with transformer. In 2023 IEEE International Conference on Multimedia and Expo (ICME), pp. 1877-1882. IEEE, 2023c.
Yang, Y., Ma, C.,, Ju, C.,, Zhang, F., Yao, J., Zhang, Y., and Wang, Y. Multi-modal prototypes for open- open-world se- mantic segmentation. International Journal of Computer Vision, 152(12):6004-60220 2024.
Wang, X., Zhang, X., Cao, Y., Wang. W., Shen, C., and Huang, T. Seggpt: Segmenting everything in context. arXiv preprint ar XrXiv:2304.03284 2023d.
Ye, M., Zhang, J., Liu, J., Liu, C., Yin, B., Liu, C., Du, B., and Tao, D. Hi-sam: Marrying segment anything model for hicrarchical text segmentation. arXiv preprint Stt1:2401.179000 2024.
Wang, X., Fang, Z., Li, X., Li, X., Chen, C., and Liu, M. Skeleton-in-comtext Unified skeleton sequence modeling with in-context learning. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1436-2446, 2024.
Yu, H., Wang, X., Niu, K., Li, B., and Xue, X. Scene text segmentation with text-focused transformers. In Pro- ceedings of the 31 ACM International Conference on Multimedia, pp. 2898-2901) 2023a.
Wang, Y., Xie, H., Wang, Z., Qu, Y., and Zhang, Y. What is the real need for scene text removal? exploring the background integrity and erasure exhaustivity properties. IEEE Transactions on Image Processing, 2023e.
Yu, H., Wang. X., Niu, K., Li, B., and Xue, X. Scene text segmentation with text-focused transformers. In Pro- ceedings of the st ACM International Conference on Multimedia, pp. 2898 2907, 2023b.
Wang, Z., Jiang, Y., Lu, Y., He, P., Chen, W., Wang, 7.,, Zhou, M., et al. In-context learning unlocked for diffusion models. Advances in Neural Information Processing Systems, 36:8542-8562. 2023f.
Yu, H., Fu, T., Li, B., and Xue, X. Eaformer: Scene text segmentation with edge-aware transformers. In European Conference on Computer Vision, pp. 410-427. Springer, 2024.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E,, Le, Q. V., Zhou, D., et al. Chain-of Chain-of-thoupe th-thuughtt prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2488 2022.
Yu, Z. and Ananiadou, S. How do large language models learn in-context? query and key matrices of in-context heads are two towers for metric learning. arXiv preprint arXiv:2402.02872, 2024.
Wies, N., Levine, Y., and Shashua, A. The Icarnability of in-context learning. Advances Neural Information Processing Systems, 36:36637-3665 2023.
Zdenek, J. and Nakayama, H. Erasing scene text with weak supervision. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision pp. 2238- 2246, 2020.
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., and Luo, P. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34:12077-12090, 2021a.
Zhang, B., Xie, H., Gao, 7., and Wang, Y. Choose what you need: Disentangled representation learning for scene text recognition removal and editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. -23688,,,8, 2024a.
Xie, S. M., Raghunathan, A., Liang, P., and Ma, An explanation of in-context learning as implicit bayesian inference. arXiv preprint 0rXiv:2111.02088 2021b.
Zhang, F., Gu, C., Zhang, C., and Dai. Y. Complementary patch for weakly supervised semantic segmentation. In
13

---

Con'Tex:: Driving In-context Learning for Text Removal and Segmentation
Proceedings of the IEEE/CVF international conference computer vision, pp. 7242-7251, 2021.
Zhang. F., Zhou, T., Li, B., He, H., Ma, C., Zhang, T., Yao, J.. Zhang, Y., and Wang. Y. Uncovering prototypical knowledge for weakly open-vocabulary semantic seg- mentation. Advances in Neural Information Processing Systems, (6673652--7666 2023a.
Zhang, J., Wang, B., Li, L.., Nakashima, Y., and Nagahara, H. Instruct me more! random prompting for visual in- context learning. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pp. 2597- 2606, 2024b.
Zhang, S., Liu, Y., Jin, I.., Huang, Y., and Lai, S. Ensnet: Ensconce text in the wild. In Proceedings of the AAAI conference artificial intelligence, volume 33, pp. 801- 808, 2019.
Zhang, T., Zhang, F. Yao, J., Zhang, Y., and Wang, Y. G4seg: Generation for inexact segmentation refinement with diffusion models, 2025, URL https://arxiv. org/ abs/ 2506. 01539.
Zhang, Y., Zhou, K., and Liu, Z. What makes good examples for visual in-context learning? Advances in Neural Infor- mation Processing Systems, 16.17773-17794, 2023b.
Zhou, T., Xia, W., Zhang, F., Chang, B., Wang, W., Yuan, Y., Konukoglu, E., and Cremers, D. Image segmenta- tion in foundation model era: survey. arXiv preprint arXiv:2408. 12957, 2024.
Zhu, Y., Liu, J., Gao, F,, Liu, W., Wang, X., Wang, P., Huang, F,, Yao, C., and Yang, Z. Visual text generation in the wild. arXiv preprint arXiv:2407.14136 2024.
14

---

Con'Tex:: Driving In-context Learning for Text Removal and Segmentation
A. Motivations
A.1. Pilot Experiments
Similar to natural image segmentation (Zhang et al., 2021; Li et al., 2023a;b; Zhang et al., 2023a; Ma et al., 2023; Yang et al., 2024; Zhou et al., 2024; Liu et al., 2024; Zhang et al., 2025), we believe that text segmentation follows similar inherent pattern learning. Recall that in Section we conduct simple pilot experiment validate the feasibility of our proposed task-chaining demonstration. Table 10 reports the results of using this simple demonstration recasting. Specifically, fine-tune the baseline pipeline Painter (Wang al., 2023b) and SegGPT (Wang al., 2023d) by forwarding this new designed demonstration, and then only reconstruct merely one type of task during the training-ing -nfeiinneee (All features are still directly fused to one label representation). Similarly, merely the training target task would be evaluated during the inference, where the other task prompt is served with ground-truth label. For instance, Rem-based Seg. refers to reconstruct the segmentation task by introducing the removal label unchanged as the ground-truth for both query and demonstration pair, which also provided during the inference. As shown in this table, the observed improvement experimentally validate the potential superiority of utilizing task-level connection improve model's ICL ability.
Table 10. Pilot experiment regarding the motivation of task-chaining.
The training dataset is HierText.
TotalText Seg.
SCUT-Ens Rem.
Method
fgloU (t)
PSNR (t)
Task- Task-specific Fine-tuning
Painter - Rem.
36.15
SegGPT - Seg-
60.60
Tusk-Chaining
Seg-based Rem. (Painter)
37.02 (+0.87)
Rem-based Seg, (SegGPT) 63.22 (+2.62)
A.2. Label role in ICL
We design a simple cross-attention-base architecture to reinforce the label representation in Section 4. This design is motivated by Wang et al. (2023f); Yu Ananiadou (2024), where they validated that label position could serve as an anchor to absorb the pattern of the provided demonstration. Figure 7 shows an illustrative explanation for this hypothesis, and clearly seen that the label position plays a vital role in understanding the demonstration. Besides, the final input text (last position) should have the most comprehensive pattern for all the demonstrations. Therefore, we could find that the linear fusion in Wang et al. (2023b;d) is quite reasonable for the purpose of merging the demonstration information for the final label. we conduct simple pilot experiment to validate the feasibility of our proposed task-chaining demonstration.
bar
foo Z36 France
Y% Cat
Shallow Layers
(a)
(a)
Dog Input text
demonstration
label
demonstration
label
Figure Illustration of label role proposed Yu Ananiadou (2024). Intuitively, shallow layers merge features into label positions and last position within ICL. This mechanism also inspires the design of our context-aware module.
B. Experiments
B.1. More Training Details
Our model adopts vision transformers (ViT) (Dosovitskiy et al., 2020) as the backbone. We use the pre-trained checkpoint from Wang et al. (2023d) as the initialization. Specifically, our model is built on ViT-L + decoder architecture, which is the same as Wang et al. (2023b;d). Notably, our proposed method requires two kinds of labels for the training datasets, i.e., stroke
15

---

Con'Text: Driving In-context Learning for Text Removal and Segmentation
masks and text-removed images. For HierText and TotalText, we directly adopt the off-the-shelf human-evaluated removal labels from (Zhu et al., 2024). For TextSeg, turn promising text-eraser method (Peng et al., 2024a) to generate the corresponding removal labels. Note that most removal-targeted works (Tursun et al., 2020; Peng et al., 2024a) would train and evaluate SCUT-Syn, which is synthetic benchmark. To align with them, we also make a similar comparison, and relevant discussion is presented at Appendix B.3. Compared to ConText, we use the SCUT-EnnsTex as one of the additional training benchmarks during the fine-tuning stage. To make their corresponding segmentation labels, we refine and expand upon the labeling techniques described in Peng et al. (2024a) by computing the distance between two images in gray-scale mode, thereby generating better fine-grainn grained mask. Such method is also applied on the following SCUT-Syn benchmarks. However, such self-generated labels, accompanied by label noises, accurately inferior to those human-annotates masks in segmentation benchmarks. Therefore, to guarantee the training stability of ConTextV, we firstly train the model by using those three segmentation benchmarks (Hier'Text+ TextSeg + TotallText), and then fine-tune the model with all of the training datasets (HierText + TextSeg + TotalText + SCUT-EnsText with merely 2 epochs. The whole training time takes 8 (12) hours for Context (ConTextVV). To guarantee promising fine- mee-grained word-level recognition in the generative paradigm, we follow (Chen et al., 2023) additionally introduce the cross-entropy-baase pixel-level supervision Lpix: accompanied with an decoder. The weight for the removal reconstruction loss set to 0.3, and for the pixel-level supervision loss Cpiz- The removal reconstruction loss is adopted as smooth-ll for both reconstructing the segmentation mask and removal image. The probability of self- prompting is set to 0.2. The simple light-weight decoder for pixel-level supervision, comprised of two convolution layers, is not used during the inference stage. All datasets used in our paper are described as follows:
extra traaliy--tiiinnaaaiiiii
1. HeerText: A fine-grast ane-ggraied real-world segmentation benchmark, including 8,281 training samples, 1,724 validation samples, and 1, 1,634 test samples. We use all the training samples during the training stage and evaluate the model with the validation set.
2. TextSeg: A large-scale fine- annotantt text segmentation dataset with 4,024 images of scene text and design text. The training, validating, and testing sets contain 2,646, 340, and 1,038 samples, respectively.
3. TotalText: A prevailing small-scale text segmentation dataset. The training and validating sets contain 1,255, and 300, respectively.
4. FST: respectively. Besides, the annotation of FST coarse equipped with part patch-like foreground regions.
prevailing small-scale English text segmentation dataset. The training and validating sets contain 229, and 233,
5. SCUT-EnsText:
is real-world scene text removal dataset, comprising 2.749 samples for training and 813 samples for testing.
6. SCUT-Syn: is a purely synthetic scene text removal dataset, comprising 8,000 samples for training and 800 samples for testing.
B.2. Detailed Results of In-context Learnability
Here we present the detailed numeric results of our performance against each benchmark under both the randomly-sclected and the ground-truth demonstration. As shown in Table 1 11,12 and 13, compared to other methods, our model could demonstrate promising in-context learnability with a notable upper-and-lower performance gap. Besides, the numerical comparison with other specialists shows huge room for further improvement of our method.
B.3. SCUT-Syn Evaluation
Here we present our performance against SCUT-Syn benchmark. Compared those data-specific methods, as shown in Table 13, our method could achieve comparable performance on this synthetic dataset. Note that in Table 3, our proposed method is unable to reach the best performance against those specialists, and we speculate such suboptimal performance may attribute to the synthetic-natural training domain gap. To verify this, Table 14 highlights the impact of incorporating the SCUT-Syn synthetic dataset on the model's performance in segmentation and removal tasks. Notably, using only the synthetic data allows the model to achieve strong in-domain performance, equally the state-of-the-art results with 0.01 MSE, However, this comes at the cost of reduced generalization to other datasets. Conversely, without any synthetic data, the model performs well on natural datasets. As more synthetic data is integrated, the model's performance shifts, balancing
16

---

Con'Tex:: Driving In-context Learning for Text Removal and Segmentation
Table Comparison with different V-ICI frameworks against several text removal (Rem.) and text segmentation (Seg.) benchmarks.
The upper in context inference performance is marked (the demonstration is the ground-truthy The performance range denotes a model's substantial co-oenext learnability towards these tasks, where a sounding upper lower bounds indicates strong scalability potential.
Removal (PSNRT (ID()
Text Segmentation (fgloU f)
Method
HierText
TSeeet
*SCUT-Syn
HierText
TTotaTText
"FST
*TextSog (val)
No Fine-tuning Baselines
28.52/32.71
29.12/44.58
27.25
28.30/41.70
1.93
13.54
MAE-VQGAN (Bar et al., 2022)
30.28 /27.69 .9
32.12/36 38.01
30.49/ 36.31
+2.66/-7.07 --7.00
8.82
17.86
20.51
28.30
+11.90
22.68 /47.
26.29 52.08
24.07 54.60
24.35/51.28
4.08
6.01
4.83
9.70
6.17
Painter (Wang al., 2023b)
+0.22/ -0.41
4.18
3.12
9.58
9.45
25,36
11.88
SegGPT (Wang al,, 2023d)
41.28
62.06
60,92
70.12
+46.22
spok-spe Fine-runing (3) HierText
26.14/31.09
36.15/21.37
33.85/ 29.30
32.05
27.992
Painter Rem.
26.29 30.90
36.32/20:47
35.62/ 27.09
+0.69/
/1.776
60.60
75.75
SegGPT Seg.
62.91
66.13
63.56
77.48
+1.88
28. 24.76
36.48/22 /1.05
34.38 28.38
32.34/24.73
64.72
67.81
61.09
77.02
67.16
Painter Rem, Seg.
29. 20.71
36.59
34,53
+1.09
/-1.49
66.97
69.99
78.99
+3.10
28. 25.51
36.56/21.19
34.42 28.32
33.05 24.34
65,23
68.53
62.20
77.40
68.34
SegGPT + Rem. + Seg.
28. 21.49
36.66 20.94
34.59 28.14
+0.09
/0.82
67.97
70.15
66.71
80.65
+3.53
39.48/ / 6.35
37.67 12.87
37.93/ 13.91
38.36/ 11.04
11.04
74.86
78.02
71.02
82.31
76.77
ConText
/4.43
Table 12. Comparison with the text segmentation specialists among four benchmarks. *FST dataset is not used for training in our model.
HierText
TotalText
*FST
TextSeg
Method
fgloUt
F-scoret
fgloUt
F-scoref
fgloUt
F-scoref
fgloUt
F-score
SegFormer (Xie et al., 2021a)
73.31
0.846
60.44
0.753
84.59
0.916
DeepLabV3+ (Chen et al., 2018)
74.44
0,824
69.27
0.802
84.07
0.914
HRNeIV2- W48 (Wang et al., 2020)
75.29
0.825
70.98
0.822
85.98
0.918
HRNeIV2-W48+OCK (Wang et al., 2020)
76.23
0.832
72.45
0.830
85.98
0.918
TexRNet D3+plabV3+ (Xu al., 2021)
76.53
0.844
72.16
0.835
86.06
0.921
TexRNet + HRNetV2-W48 (Xu et al., 2021)
55.50
0.656
78.47
0.848
73.38
0.850
86.84
0.924
TFT (Yu al., 2023a)
82.10
0.902
72.71
0.845
87.11
0,931
EAFormer (Yu et al., 2024)
82.73
0.906
72.63
0.840
88.06
0.939
UPOCR (Peng et al., 2024b)
88.76
0.940
Hi-SAM (Ye al., 2024)
77.76
0.848
84.59
0.887
88.77
0.938
81.21
0.896
85.19
0.919
75.90
0.873
89.74
0.946
ConTextV
83.67
0.911
88.13
0.937
82.98
0.907
93.95
0.969
between in-domain excellence and generalization. The optimal configuration was found by using 25% of the training samples from SCUT-Syn, achieving comprehensive performance balance. This underscores the domain gap issue between synthetic and natural data, emphasizing the importance an appropriate data mix for optimal results.
B.4. PromptText
Recall that in Section 5.3 we introduce a self-designed dataset to mimic the numan-based instructed prompts on the textual images. To this end, we select validation set from TextSeg, and adopts original annotation to make the corresponding explicit prompts, which roughly contains:
1. Select the erasing probability from (0.3, 0.5, 0.7), and such probability is used for deciding whether the annotation is erased.
2. Select the annotation type, which contains stroke-level, box-level, and circle-level. The circle-level annotation could be generated from depiction a circumscribed circle highlighted by a box annotation.
3. Select the color for this prompt from Red, Green, and Blue.
17

---

Con'Text: Driving In-context Learning for Text Removal and Segmentation
Table 13. Comparison with specialists tailored for text removal among two benchmarks. Note that compared to other methods, our framework is not trained with Syn datasets. Particularly, SCUT-Syn is an artificially synthesized dataset.
SCUT-EnsText
"SCUT-Syo
Method
PSNRT
MSSIMI
MSE
AGEL
pEPs
PCHPs]
FID
PSNR1
MSSIM1
MSEL
AGEL
pEPS)
PCHPS]
Pix2Pix (Phillip al., 2017)
26.70
88.56
0.37
6.09
0.0480
0.0227
46.88
26.76
91.08
0.27
5.47
0.0473
0.0244
STE (Nakomura 2017)
25.47
90.14
0.47
5.033
0.0533
0.0296
43.39
25,40
90.12
0.65
9.49
0.0553
0.0347
EnsNeT (Zhang al., 2019)
29.54
92.74
0.24
4.16
0.0307
0.0136
32.71
37.36
96.44
0.21
1.73
0,0069
0.0020
MTRNet++ (Tursun et al., 2020)
29.63
93.71
0.23
3,51
0.0305
0.0168
35.50
34.55
98.45
0.04
EraseNeT (Liu ol., 2020)
32.30
95.42
0.15
3.02
0.0160
0.0090
19:27
38.
97.67
0.02
0.0048
0.0004
(Tang et al., 2021)
35.34
96.24
0.09
38.60
97.55
0.02
PSSTRNet (1yy & Zhu, 2022)
34,65
96.75
0.14
1.72
0.0135
0.0074
39.25
98. 15
0.02
1.20
0.0043
0.0008
CTRNet (Liu ot al., 2022a)
35,20
97.36
0.09
2.20
0.0106
0.0068
13.99
41.28
98.52
0.02
1.33
0.0030
0.0007
GaRNet
Chai, 2022)
97.14
0.08
0.0105
0.0062
15.50
MBE (Hou ol,, 2022)
35.03
97.31
0.0128
0.0088
43.85
98.66 64.64
0.94
0.0013
0.00004
PEN (Du et al., 2023c)
35.72
96.68
0.05
1.95
0.0071
0.0020
38.87
97.83
0.03
1.38
0,0041
0.0004
PERT (Wang et al., 2023e)
33.62
97.00
0.13
2.19
0.0135
0,0088
39.40
97.87
0.02
1.41
0.0046
0.0007
SAEN (Du
2023a)
96.53
0.07
0.0043
0.0004
FE:NNet (Lyy et al., 2023)
34,53
97.01
0.13
1.75
0.0137
0.0080
39.14
97.97
0.02
1.26
0.0046
0.0008
ViTEraser (Peng et al., 2024a)
36.87
97.51
0.05
1.72
0.0066
0.0035
10.15
42.97
98.55
0.01
1,11
0.0015
0.000011 I
UPOCR (Peng el al., 2024b)
37.14
97.62
0.04
1.72
0.0064
0.0034
10.47
ConTextV
40.83
98.76
0.03
0,76
0.0053
0.0029
11.63
38.30
98,30
0.07
0.99
0.0049
0.0032
41.26
98.86
0.02
0,0047
0.0025
10.82
38.65
98.37
0.06
0.94
0.0044
0.0029
Table 14. Performance with varying proportions of training samples from SCUT-Syn. The '+' symbol indicates the number of SCUT-Syn training samples added to the baseline training dataset used Con TextV, RS (GT) denotes the model's performance with randomly- selected (ground-truth) demonstration.
SCUT-Syn Training
SCUTT-Syn Rem. (RS)
TotalText Seg.
SCUTT-Ens Rem.
Data Volume
MSE /PNR f
RS GT GT
RS GT
ONLY SCUT-Syn (100%)
0.01 44.114
62.15 +3.33
36.78 +0.76
NO SCUT-Syn (0%)
0.07 / 38.30
85.19 / +2.94
40.83 / +0.43
2,000 (25%)
0.04 / 39.53
85.02 / +2.68
40.33 / +0.56
+ 4,000 (50%)
0.04 / 39.82
84.73 / +2.37
40.06 +0.43
8,000 (100%)
0.03 / 40.07
83.19 +2.45
39.47 +0.62
4. Mark each image with the selected color and annotation type for each no-erased label, and generate the corresponding removal and segmentation image.
This dataset contains 429 samples, and each image has 3-level annotation based on the erasing probability. Some visualized samples are shown in Figure 8. Note that this dataset is merely used for evaluation.
B.5. More Visualized Results
Figure 9 presents a visual comparison between our method and other SOTA specialists. The enhanced and fine- ane-ggained segmentation and erasing detail highlights the superiority and effectiveness of our proposed Con in addressing these tasks. Particularly, our model could even achieve better results than the given ground-trui -ruth label.
18

---

Con'Tex:: Driving In-context Learning for Text Removal and Segmentation
ARMSTRONG CREEK EAST
JGIS the cake
ROBY flepi WORRY.
GO WID
Louggall 6|
COMMUNITY HUB
RMS RONG
J
BY RY
GO WIL
Lout
CREEK EAS
CO MUNI Y
Le
Ee
UB
A
T
LL
RU R
Frlk &
T
t
W
M
T
CA
Stroke-Level
Circle-Level
Box-Level
Figure Visualized samples of designed PromptText. Zoom in for better view.
Removal
Segmentation
L'SSPLNADDE
WENE-WERRR RER BESTEE CASTION
Bendigo feath 01K3MI (807 R1AM
Bandga Frall care
Bendigo Healtharee 01/DMM 0BUTT
5
Berdga Hoaltcure
08:77
Al
MONY
NGNY CIT
MONY
NONY US
NONY OUS
NONY OUS
26 AOOKS
Ground-Truth Figure 9. Qualitative comparison of method and existing specialists CexTeE and HierText. Our method is prompted by random demonstration. Clearly, our framework demonstrates promising performance across these tasks. Zoom in for a better view.
Input
ViT-eraser
ConTextV
Ground-Truth
Input
Hi-SAM
ConTextV
19


Reviews:
[{"review_id": "NALxmzWHEU", "reviewer": "Reviewer_qfVE", "summary": "This paper proposes ConText, an adaptation of the visual in-context learning (V-ICL) paradigm specifically tailored for optical character recognition tasks, focusing on text removal and segmentation. To address the single-step reasoning bottleneck in existing V-ICL methods, ConText introduces a task-chaining compositor, sequentially linking text removal and segmentation tasks. Additionally, the authors propose a context-aware aggregation module to enhance latent query representation and introduce self-prompting to maintain robust in-context reasoning and prevent overly specialized, context-free inference. Extensive experiments on multiple benchmarks demonstrate that ConText significantly outperforms existing V-ICL generalist and specialist methods, achieving new state-of-the-art results in both text removal and segmentation tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### W1 (Additional Task Exploration)\n>While the task-chaining compositor is shown to be effective...This may be due to dataset limitations, but extending the approach to additional tasks, such as watermark removal or text registration, would be valuable for further exploration.\n\nThanks for this valuable advice. We also appreciate your understanding regarding the challenging scarcity of relevant OCR benchmarks. Considering data availability, we have additionally explored our ConText on 1 prevailing *watermark removal* benchmark, CLWD [1]. The table below reports the results of SegGPT and a leading specialist [2]. Clearly, our approach demonstrates superior performance, further validating the **task-wise generalizability of ConText** (We also kindly invite you to refer to our response for **W1-Reviewer L63m**, where we provide additional generalizations to more tasks).  We will include this analysis. \n\n| Method | PSNR↑ (Rem.) | fgIoU↑ (Seg.) |\n| ------- | -- | ----|\n| SegGPT (generalist) |   30.11     |  74.42     |\n| PFMNet [2] (specialist) |   39.45       | 79.09  |\n| ConText |   **40.73**      |  **82.16**     |\n\n**CLWD includes both pixel-level binary watermark masks and the removal images. We utilize its training set to train SegGPT and ConText with similar learning strategy. Rem. (Seg.) refers to the removal (segmentation) task.*\n\n---\n>### Reference\n>[1] Wdnet: Watermarkdecomposition network for visible watermark removal, CVPR22.\n\n>[2] Fine-grained Visible Watermark Removal, ICCV23.", "claims_and_evidence": "The paper makes three primary claims:\n\nFirst, the authors claim that existing methods limit models' in-context learning (ICL) capability by restricting prompts to image-label pairs, and propose a task-chaining compositor to enhance in-context reasoning. Second, they argue that their proposed context aggregation module effectively improves contextual understanding. Third, the authors assert that the inherent heterogeneity of text is more complex than object-level scenes, motivating their simple yet effective self-prompting strategy.\n\nCollectively, these claims underpin the development of ConText, which achieves state-of-the-art performance on several benchmarks for text removal and segmentation, verifying the combined effectiveness of the proposed methods. Individually, the authors provide supporting evidence for each claim; for instance, Fig. 5 demonstrates that the task-chaining compositor significantly enhances out-of-domain generalization capabilities.", "methods_and_evaluation_criteria": "The proposed methodology is clearly written and logically aligns with the ICL framework for text removal and segmentation. The benchmarks, baselines, and evaluations are comprehensive.", "theoretical_claims": "No theoretical claims were evaluated.", "experimental_designs_or_analyses": "The experimental design is sound, and the analysis is comprehensive.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper introduces fresh ideas in visual in-context learning, particularly through the task-chaining compositor and context aggregation module, which enhance task chaining. The methodology is easy to understand yet effective, offering potential insights for future research in visual in-context learning.\n\nWeaknesses:\n\nWhile the task-chaining compositor is shown to be effective, the paper only explores chaining two tasks, i.e. text removal and text segmentation. This may be due to dataset limitations, but extending the approach to additional tasks, such as watermark removal or text registration, would be valuable for further exploration.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XECRAmG3jt", "reviewer": "Reviewer_L63m", "summary": "This paper explores the application of visual in-context learning to fine-grained text recognition tasks, including text segmentation and removal. Rather than employing single-task solutions, the authors propose a task-chain prompt framework that connects multiple tasks. Through extensive experimentation, they demonstrate state-of-the-art results on both text removal and text segmentation benchmarks while showing the effectiveness of the task-chaining approach compared to single-task methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### W1 (Beyond two tasks)\n>...is it possible to generalize to other tasks?...how would the framework perform with three or more chained tasks...\n\nWe'd like to argue that **our method can intuitively be integrated with other tasks (even image-level tasks) alongside chaining extension**. Except for adapting ConText to watermark removal (please refer to **W1-Reviewer qfVE**), we have conducted two tasks: image *edge detection* and *denoising*. As shown in the table below (HierText), our method also shows promising performance on these new tasks, and the progressive task-chaining could lead to a global improvement. Overall, these results help demonstrate the **task-wise scalability of ConText**. We will add this analysis.\n|Method|MAE↓ (Edge.)|PSNR↑ (Den.)| PSNR↑ (Rem.) | fgIoU↑ (Seg.)|\n|-|-|-|-|-|\n| Painter -> Edge.|29.78|-|-|-|\n| ConText -> Edge.|**22.74**|-|-|-|\n| Painter -> Den.|-|30.45|-|-|\n| ConText -> Den.|-|**34.57**|-|-|\n| Painter -> Edge. + Den.  |27.17|32.68|-|-| \n| ConText -> Edge. + Den. |**19.79**|**36.83**|-|-|\n| Painter -> Edge. + Rem. + Seg.  |26.48|-|30.12|66.43|\n| ConText -> Edge. + Rem. + Seg.  |**17.76**|-|**40.23**|**76.61**|\n| Painter -> Edge. + Den. + Rem. + Seg.|25.22|34.73|31.19|68.65|\n| ConText -> Edge. + Den. + Rem. + Seg.|**16.14**|**38.19**|**41.26**|**77.98**|  \n\n*For edge detection (Edge.), we use the Canny operator for the whole image regions. For denoising (Den.), we follow [1] and add the image with noising. To chain these tasks, we keep the noised counterpart as the 1st input, set the original image as the 2nd one, and the edge-detected image, along with the removal and segmentation masks, as the following parts.*\n\n---\n### W2 (Discussion on Self-prompting Strategy)\n>...self-prompting has limited effect in randomly selected settings...(e.g., SP = [0, 0.1, 0.2, ... 1.0]) would better demonstrate how to optimize this parameter.\n\nThe observed weakened performance could essentially be attributed to the **in-context empowerment mechanism** of the self-prompting (SP). SP enables the model to maintain in-context generalization by guiding it to perform shortcut-like reasoning based on the self-demonstrations. Compared to the no-SP process, where no valid counterparts are provided during the training, **the model with SP may reduce the complexity of task reasoning by learning valid prompted patterns from the given self-demonstrations**. Consequently, this simplification shall reasonably yield a performance drop in demonstration-free (randomly selected) cases.\n\nThe table below provides a more detailed assessment of SP. **As the involvement of SP increases, the model generally exhibits decreased demonstrate-free reasoning ability and improved in-context learnability,** highlighting the significance of proper utilization of SP. We will add this discussion.\n| Method   | Seg. (RS / GT) fgIoU   | Rem. (RS / GT) PSNR  |\n| -------- | -------------- | -------------- |\n| Baseline | 79.14 / +0.65  | 38.59 / +0.37  |\n| SP-0.1   | 78.45 / +2.12  | 38.02 / +0.77   |\n| SP-0.2   | 78.02 / +3.98  | 37.67 / +0.92  |\n| SP-0.3   | 77.94 / +4.05  | 37.12 / +1.24  |\n| SP-0.4   | 77.63 / +4.17  | 36.82 / +1.45  |\n| SP-0.5   | 77.39 / +5.04  | 36.52 / +1.94      |\n| SP-0.6   | 77.14 / +5.83  | 36.12 / +2.13      |\n| SP-0.7   | 76.64 / +6.44  | 35.82 / +2.75      |\n| SP-0.8   | 76.33 / +6.73      | 35.37 / +3.47      |\n| SP-0.9   | 76.12 / +6.95      | 35.03 / +3.96      |\n| SP-1.0   | 75.94 / +7.04      | 34.81 / +4.37      |\n\n**The table follows same setting of Table 5. RS (GT) refers to the randomly selected (ground-truth) demonstration case. Here Baseline refers to the baseline method + CAA.*\n\n---\n### W3 & 4 (Ablation Studies on ConText)\n>The ablation study is conducted only on SegGPT rather than on the proposed ConText model.\n\nDue to the page limit, we have presented several ablations on ConText at **Appendix B.3 & B.6**, including the **influence of synthetic training datasets**, **masking ratio**, and **different inference mechanisms**. We will adjust the formatting for better clarity. \n\n>...it lacks direct comparisons between individual task paths...\n\nThanks for this advice. Below we present the suggested single-task-tuned experiments on HierText. Clearly, our single-task-tuned ConText outperforms multi-task baselines, and our task-chaining ConText further shows a significant improvement over its single-task-oriented counterparts, validating the advantage of the task-in-chain concept (our response to your *W1* could also help validate this). We will add this analysis.\n|Method|PSNR (Rem.)| fgIoU (Seg.)|\n|-|-|-|\n| Painter -> Rem. + Seg.|28.17|60.60|\n| SegGPT -> Rem. + Seg.  |28.16|65.23|\n| ConText -> Rem.|**37.35**|-|\n| ConText -> Seg.|-|**70.88**|\n| ConText -> Rem. + Seg.|**39.48**|**74.86**|\n       \n**We implement single-task-tuned ConText through implementing the designed modules by only fusing the single task feature.*\n\n---\n>### References\n>[1] Visual Prompting via Image Inpainting, NeurIPS'22.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "The evaluations make sense to me.", "theoretical_claims": "correct", "experimental_designs_or_analyses": "Please refer to the weaknesses", "relation_to_broader_scientific_literature": "The paper explores how to extend single-task in-context learning to multi-task in-context learning, using text removal and text segmentation as a case study.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "- **Strengths**\n  - The paper is well-written, with good presentation and extensive experimental evaluation.\n  - The work successfully extends visual in-context learning from single-task scenarios to task-chaining prompts, enhancing the model's ICL capabilities through multi-task reasoning.\n  - Extensive benchmarking demonstrates that the approach achieves state-of-the-art performance on both text removal and segmentation tasks, with relatively huge improvements over other ICL methods.\n\n- **Weaknesses**\n  - The idea of extending single-task in-context learning to multi-task is interesting, is it possible to generalize to other tasks? Also, the task of text segmentation and removal only has two tasks, how would the framework perform with three or more chained tasks? A discussion of the generalizability of the framework would be helpful.\n  - Table 5 shows that self-prompting has limited effect in randomly selected settings. Is there any interpretation of this? Additionally, the threshold selection for self-prompting lacks comprehensive evaluation - including results across a range of thresholds (e.g., SP = [0, 0.1, 0.2, ... 1.0]) would better demonstrate how to optimize this parameter.\n  - The ablation study is conducted only on SegGPT rather than on the proposed ConText model. \n  - While the paper demonstrates significant improvements from ConText, it lacks direct comparisons between individual task paths (ConText → Rem and ConText → Seg) against both other methods and the combined approach (ConText → Rem + Seg). While there are comparisons shown on the SegGPT and Painter model, a direct comparison of the proposed model would give more intuitive insights of the advantage of task-in-chain over a single task.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Please see the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "va1SRfAtDg", "reviewer": "Reviewer_AZf8", "summary": "The authors present a visual in-context learning (V-ICL) approach, ConText, for fine-grained text recognition tasks (segmentation and removal). In order to accomplish this they focus on three novelties:\n\n1. Task-chaining: The two tasks are chained together instead of being done independently to leverage inter-task correspondence and enhance the generalised capability. The tasks would have to be related however for this benefit to manifest\n2. Context aware aggregation (through attention mechanisms): helps leverage learnability from other image-label pairs in the given context, instead of simple addition\n3. Self-prompting: helps select the right demonstrations; during training the most relevant demonstrations are selected dynamically which makes the more adaptive and helps deal with the context-homogeneity issue inherent in older models due to random or fixed demonstrations.\n\nDuring training the standard masked-based strategy is used.\n\nOverall, the authors show that their approach is superior to single-task methods across three datasets when considering both other generalists and specialists. Ablations are also presented to provide compelling qualitative evidence.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### W1 (Designed Modules Efficiency)\n>...extra cost of self-prompting and context aggregation...\n\nBelow we report the additional costs of these modules, and find that\n\n1. Self-prompting (SP) incurs an additional training burden of **+0.4 minutes/epoch**. However, this cost is deemed acceptable due to the moderate engagement of SP (SP-0.2) during training. Moreover, SP is not utilized during inference, yielding **no additional computational burden for inference.**\n2. Context-aware aggregation (CAA) introduces extra computational costs during both training and inference. However, as a lightweight cross-attention module, it only **increases model complexity by a manageable 2%**. Consequently, it leads to a mere **increase of +0.03 seconds (+0.8 min/epoch) during inference (training).**\n\nBased on this, we can more safely conclude these modules indicate **a reasonable level of computational efficiency.** We will add this analysis. \n| Method|Training Time (per epoch) | Inference Time| Model FLOPs | \n|-|-|-|-|\n|Baseline|3.8 min|0.09 sec |666.76G|\n|Baseline + SP|+0.4 min|+0 sec|+0G (0%) |\n|Baseline + CAA | +0.8 min | +0.03 sec | +17.20G (2%) | \n|Baseline + CAA + SP| +1.0 min | +0.03 sec | +17.20G (2%) | \n\n**The results are evaluated on HierText, and FLOPs and inference time are calculated by forwarding one 512x512 image on one A100, with the inference time reported in seconds.*\n\n---\n### Q1 (Performance on Document Case)\n>How does ConText perform on document-type datasets (i.e. dense text) such as DocLayNet.\n\nWe believe that **ConText could address this case due to its superior performance on HierText [1], which collects a substantial number of high-fidelity, real-world document-based samples** (please refer to [1] for some visualized examples and its data curation in *\"Section 3\"*). We will add more these visualized samples in the updated version. Since **DocLayNet lacks pixel-level stroke and removal annotations**, we may not be able to provide valid numerical evaluations during this rebuttal period. We hope for your understanding and plan to explore this further in the future.\n\n---\n### Q2 (SP *v.s.* Nearest Neighbor Retrieval)\n>...ablation on retrieval (nearest neighbor) vs self-prompting...\n\nCompared to SP, we agree with you that **the retrieval strategy should exhibit lower training costs due to its direct prompt-specific assignment.** To verify this, we adopt DINOv2 [2], a leading visual retriever [3-4], to assign the nearest counterpart for each sample as the in-context prompt during training. As shown in the table below, this once-for-all solution demonstrates slightly better training efficiency (**-0.35 minutes/epoch**) than SP. However, it is observed that **this retrieval-based selection yields inferior ICL ability**, as it has a narrower prompt selection scope compared to the adaptive SP. We will add this study.\n\n| Method| Training Time (per epoch) | Seg. (RS / GT) fgIoU↑|\n|-|-|-|\n|ConText + Retrieval|  **4.45 min** | 72.11 / 73.25  |\n|ConText + SP | 4.8 min | **74.86** / **78.12**    |\n\n**The above results were obtained using HierText. The training time is reported in minutes per epoch, and pre-processing time of retrieval is not calculated. RS (GT) refers to the randomly selected (ground-truth) demonstration case.*\n\n---\n### Q3 (Task-chaining *v.s.* Joint-training)\n>...what particular scenarios (i.e. types of images) chaining is better than joint training (and in which it is not)\n\nThanks for raising this. Here we provide a corresponding case-by-case study as follows: \n\n1. For those high-performance samples, as shown in Figure 4, we find that most of them contain **hard-to-recognize visual patterns**, e.g., texts with similar background colors (the last sample), small textual fonts (the 5th sample), and character-like objects (the 2nd and 3rd samples). In these cases, task-chaining explicitly highlights the prompted patterns across tasks, thereby yielding significant improvements.\n2. Meanwhile, we also identify a small number of samples that exhibit limited or even worse performance compared to joint-training. Most of them have relatively **low resolution and noisy annotations in text regions**. In this way, simply relying on task-chaining may exacerbate the accumulation of these noises, leading to performance results that fall below expectations compared to joint-training.\n\nIn conclusion, **task-chaining shows overall superiority over joint-training.** We will add this discussion.\n\n---\n### S1 (Crammed formatting)\n>...it would be better to be much more selective w.r.t the core information.\n\nThanks for this advice and we will polish our formatting for better clarity. \n\n---\n>### References\n>[1] Towards End-to-End Unified Scene Text Detection and Layout Analysis, ICDAR'23.\n\n>[2] DINOv2: Learning Robust Visual Features without Supervision, TMLR'24.\n\n>[3] Retrieval-augmented embodied agents, CVPR'24.\n\n>[4] FORB: a flat object retrieval benchmark for universal image embedding, NeurIPS'23.", "claims_and_evidence": "Yes, the authors show results supporting claims", "methods_and_evaluation_criteria": "The datasets and metrics make sense for this task", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, the experiment description in the ablation makes sense", "relation_to_broader_scientific_literature": "Text Removal/Segmentation is often a component of OCR pipelines, however these require explicit fine-tuning instead of in-prompt learning\n\n(Visual) In-context Learning has leveraged prompt context to provide examples and avoid fine-tuning. However, currently tasks are done independently and not chained together (and also don't have the self-prompting and context-aggregation present in ConText)\n\nChain-of-though reasoning, popular with models such as R1, however hasn't been explored much for the visual modality", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths\n- Overall I think this is a novel paper that extends the current V-ICL space to leverage chain-of-thought and help produce good results for text-tasks without need for fine-tuning or human effort in curating labelling samples (through self-prompting)\n\n\nWeaknesses\n- May need to consider extra cost of self-prompting and context aggregation when evaluating this compared to competing methods", "other_comments_or_suggestions": "Both the text and most diagrams feel very crammed with information, instead of fitting everything into the given page limit it would be better to be much more selective w.r.t the core information.", "questions_for_authors": "- How does ConText perform on document-type datasets (i.e. dense text) such as DocLayNet\n- I couldn't find an ablation on retrieval (nearest neighbour) vs self-prompting and wondering if former can be a lower cost way\n- Would be interesting to understand in what particular scenarios (i.e. types of images) chaining is better than joint training (and in which it is not)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EBTWcoq69c", "reviewer": "Reviewer_qCFq", "summary": "This paper introduces ConText, a visual in-context learning framework tailored for OCR tasks such as text removal and text segmentation.\nIt employs task-chaining, context-aware aggregation, and a self-prompting strategy to leverage multi-task logic and enhance in-context reasoning. Extensive experiments demonstrate state-of-the-art performance on multiple benchmarks, confirming its effectiveness and adaptability.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### W1 (Limited Contribution & Novelty)\n>...incremental improvement in visual in-context learning...lack significant technical novelty... \n\nWe respectfully disagree with this and argue that our work, **rather than being merely incremental, offers substantial technical novelty and valuable contributions**:\n\n1. *Novelty*: Beyond CAA and SP, we are **the first to explore visual task-chaining to enhance multi-task performance**, which is a novelty consistently acknowledged by other reviewers. Furthermore, as suggested by Reviewers qfVE & L63m, its **promising generalized scalability** has been experimentally validated across various tasks. Thus, we believe this simple yet effective design warrants recognition for its novelty and represents a significant contribution to visual-ICL.\n2. *Contributions*: We'd like to emphasize that we propose **the first visual-ICL paradigm for OCR**, achieving SOTA performance against both generalist and specialist models. Notably, we find that our ConText **emerges with in-context understanding of visual-specific instructions in a training-free manner** (see **Figure 6**), e.g., removing and segmenting the text of the specified color, thereby facilitating flexible user-model interaction. We believe these valuable insights will encourage further developments in visual-ICL.\n\n---\n### W2 & S3 (Theoretical Support & Inappropriate Submission)\n>...lack robust theoretical justification...paper be targeted to a computer vision conference rather than a machine learning venue... \n\nOur method may lack theoretical analysis, but we argue it is grounded by **robust experimental findings and the underlying mechanisms of ICL** (please refer to Appendix A). We contend that this does not render it unsuitable for ICML, as **no stringent theoretical support is required for submissions.** This openness has fostered a series of impressive theoretical-free works in computer vision presented at ICML, such as the notable CLIP [1], the multi-modal generalist mPLUG-2 [2], and the OCR framework UPOCR [3]. Meanwhile, this embrace of diversity is also reflected in *Reviewer Instruction*: \"*reviewers are encouraged to be open-minded in terms of potential strengths...particularly for application-driven ML papers.*\" Therefore, while we understand your concerns, we respectfully hope for your inclusive understanding. We believe that our paper, which presents **a powerful and practical visual-ICL OCR paradigm with valuable insights and contributions** (as agreed upon by other reviewers), deserves consideration at this conference.\n\n---\n### W3 (Task-chaining Efficiency)\n>...it fails to discuss the impact of this design on training efficiency.\n\nThanks for raising this. It is claimed that **a single task chain can address multiple tasks simultaneously, while the input-output baseline requires a separate training sample for each task**, e.g., one for segmentation and another for removal. As such, one sample of task-chaining effectively equals two samples from the baseline, which should yield **no significant training efficiency gap** between these approaches. As shown in the following table (HierText dataset), task-chaining adds merely **+0.6 minutes/epoch** compared to the baseline, indicating a quite acceptable level of training overhead. We will include this analysis.\n|Method|Training Time (per epoch)|\n|-|-|\n|SegGPT|3.2 min| \n|SegGPT+task-chaining|+0.6 min| \n\n---\n### W4 (Scalability)\n>...confined to OCR...limiting its extensibility to other domains.\n\nWe agree that task-chaining utilizes inter-task logic for collective benefits, but we contend that **it is not merely limited to OCR domains**, and can serve as a flexible pipeline to **bridge different tasks with promising scalability:** \n\n1. Intuitively, the segmentation-removal concept could be directly applied to the **natural image domain.** To verify this, we utilize one natural image removal benchmark, PIPE [4]. The results in the table below demonstrate ConText's superiority in the natural domain.\n\n|Method (PIPE dataset)|PSNR (Rem.)|fgIoU (Seg.)|\n|-|-|-|\n|SegGPT|31.72|58.92|\n|ConText|**35.33**|**62.76**|\n\n2. Additionally, beyond the removal-segmentation logic, we argue that there are **inherent logical connections among various vision tasks that can be explored implicitly.** By scaling this task-in-chaining concept, we can enhance other image-level tasks, e.g., *watermark removal, edge detection, and denoising* (please refer to our responses to **W1-Reviewer L63m & qfVE**). \n\n---\n### S1 & 2 (Writing Issue) \n>...weakly written...Figure are not detailed...\n\nThanks for this and we will make the suggested improvements.\n\n---\n>### References\n>[1] Learning Transferable Visual Models From Natural Language Supervision, ICML'21\n\n>[2] mPLUG-2: A Modularized Multi-modal Foundation Model\nAcross Text, Image and Video, ICML'23\n\n>[3] UPOCR: Towards Unified Pixel-Level OCR Interface, ICML'24\n\n>[4] Paint by Inpaint: Learning to Add Image Objects by Removing Them First, CVPR'25.", "claims_and_evidence": "The paper's claims are generally supported by extensive experimental evidence across several benchmarks.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are logically appropriate for addressing OCR tasks within a visual in-context learning framework.", "theoretical_claims": "The paper does not include any formal proofs, as its theoretical claims are supported primarily by intuitive reasoning and empirical evidence.", "experimental_designs_or_analyses": "The experimental designs and analyses, including ablation studies and cross-benchmark comparisons, are generally sound.", "relation_to_broader_scientific_literature": "The paper's contributions can be seen as an incremental improvement over existing V-ICL approaches like SegGPT and Painter, integrating task chaining, context-aware aggregation, and self-prompting strategies to better address OCR challenges.", "essential_references_not_discussed": "The paper appears to have sufficiently discussed all key related works, with no significant essential references omitted.", "other_strengths_and_weaknesses": "Strengths:\n\n1.The paper presents a comprehensive framework that effectively adapts visual in-context learning to OCR tasks, offering a practical solution for both text removal and segmentation.\n\n2.It demonstrates thorough experimental validation across multiple benchmarks, which supports the claimed improvements of the proposed methods.\n\nWeaknesses:\n\n1. Overall, the work represents an incremental improvement in visual in-context learning for OCR tasks; both the context-aware aggregation and self-prompting strategies lack significant technical novelty and robust theoretical justification.\n\n2. The paper proposes to recast the task demonstration by forming an explicit chain rather than a simple input-output pair, but it fails to discuss the impact of this design on training efficiency.\n\n3. The method is tightly confined to OCR tasks, as it leverages the inherent logical relationship between text removal and segmentation—where the removed text is exactly the text to be segmented—limiting its extensibility to other domains.", "other_comments_or_suggestions": "1.The related work section is weakly written; for example, the description \"Unlike these discriminative models, this paper proposes a universally generative-based framework for the task.\" does not clearly outline the shortcomings of prior works before introducing the new approach.\n\n2.The explanations for Figure 2 and Figure 3 are not detailed enough, making the proposed method hard to understand.\n\n3.It is suggested that the paper be targeted to a computer vision conference rather than a machine learning venue, given the limited theoretical support.", "questions_for_authors": "Please refer to the weaknesses section for the detailed questions, particularly regarding the limited scalability beyond OCR tasks.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a visual in-context learning approach for fine-grained text recognition tasks, including segmentation and removal.

After the rebuttal, three reviewers (qfVE, L63m, and AZf8) provided positive comments, recognizing the paper's strong performance and novel ideas. The other reviewer (qCFq) did not respond to the authors' feedback.

The Area Chair (AC) agrees with most of the reviewers and recommends accepting the paper. Additionally, it is strongly suggested that the authors incorporate the content discussed in the feedback to make the paper more comprehensive.
Decision: Accept (poster)