Urban Socio-Semantic Segmentation with Vision-Language Reasoning
Authors: Yu Wang
Yi Wang
Rui Dai
Yujie Wang
Kaikui Liu
Xiangxiang Chu
Yansheng Li
Venue: ICLR 2026 Poster (2026)
Keywords: Remote Sensing
Semantic Segmentation
Vision Language Model
Reinforcement Learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: As hubs of human activity, urban surfaces consist of a wealth of semantic entities. Segmenting these various entities from satellite imagery is crucial for a range of downstream applications. Current advanced segmentation models can reliably segment entities defined by physical attributes (e.g., buildings, water bodies) but still struggle with socially defined categories (e.g., schools, parks). In this work, we achieve socio-semantic segmentation by vision-language model reasoning. To facilitate this, we introduce the Urban Socio-Semantic Segmentation dataset named SocioSeg, a new resource comprising satellite imagery, digital maps, and pixel-level labels of social semantic entities organized in a hierarchical structure. Additionally, we propose a novel vision-language reasoning framework called SocioReasoner that simulates the human process of identifying and annotating social semantic entities via cross-modal recognition and multi-stage reasoning. We employ reinforcement learning to optimize this non-differentiable process and elicit the reasoning capabilities of the vision-language model. Experiments demonstrate our approach's gains over state-of-the-art models and strong zero-shot generalization. The dataset and code are open-sourced under the Apache License 2.0 at https://github.com/AMAP-ML/SocioReasoner.

Published as a conference paper at ICLR 2026
URBAN SOCIO-SEMANTIC SEGMENTATION VISION-LANGUAGE REASONING
WITH
Yu Wang' 22; Yi Wang", Rui Dai
Yujie Wang',
Kaikui Liu",
Xiangxiang Chu?, Yansheng Li
Wuhan University
2 Amap, Alibaba Group
ABSTRACT
As hubs of human activity, urban surfaces consist of wealth of semantic tities. Segmenting these various entities from satellite imagery is crucial for a range of downstream applications. Current advanced segmentation models can reliably segment entities defined by physical attributes (e.g., buildings, water bod- ies) but still struggle with socially defined categories (e.g., schools, parks). this work, we achieve socio aocio-semantic segmentation by vision-languaal model reasoning. To facilitate this, we introduce the Urban Socio-Semantic Segmenta- tion dataset named SocioSeg, new resource comprising satellite imagery, digital maps, and pixel-level labels of social semantic entities organized hierarchi- cal structure. Additionally, propose novel vision-language reasoning frame- work called SocioReasoner that simulates the human process of identifying and annotating social semantic entities via cross-modal recognition and multi-stage reasoning. We employ reinforcement learning to optimize this non-differentiable process and elicit the reasoning capabilities of the vision-language model. Exper- iments demonstrate our approach's 's gains over state-of the-trt models and strong zero-shot generalization. The dataset and code are open-sourced under the Apache License 2.0 at Wuthbb.cmmAMMPPP-//ScccceC.
INTRODUCTION
Urban areas, as primary hubs of human activity, critical subject for Earth Observation (Patino & Duque, 2013). Urban land surfaces consist of rich semantic entities, and segmenting them is crucial for downstream tasks like urban planning (Zheng et al., 2025) and environmental monitoring (Yang, 2021). These entities can be broadly grouped into two types: physical semantic entities and social semantic entities. The first encompasses entities defined by physical attributes, such as buildings, water bodies, and roads. Thanks to abundant high-resolution satellite data, current segmentation models can segment these entities precisely from visual cues satellite imagery (Hang et al., 2022). The second comprises entities defined by social attributes, such as schools, parks, and residential districts. The identification of these entities is pivotal not only for urban analysis tasks like disease transmission (Alidadi & Sharifi, 2022), the 15-minute city (Bruno et al., 2024). but also for industrial mapping applications, such as inferring socio-semantic Areas of Interest (AOIs) from Points of Interest (POIs) to enhance navigation and recommendation (Shi et al., 2025). However, their boundaries and identities are shaped by social semantics rather than distinct visual appearances (Büttner, 40. Since this semantic information is difficult extract from satellite imagery alone, achieving segmentation for these socially defined entities is substantially more challenging.
Existing approaches address this challenge by incorporating auxiliary (e.g., Points of Interest) (Xiong et al., 2025; Zhang et al., 2017). These methods often employ sepa- rate model encoders extract features from different modalities and train task-specific models in fully supervised manner. However, this paradigm faces three major bottlenecks: (i) such geospatial data are often difficult to obtain due to commercial or security constraints; (ii) even when available,
geospatial data
Work done during internship at Amap.
I Project lead
1 "Corrsppnndigg author.

---

Published as a conference paper at ICLR 2026
Building
Playground
Monsion
Adminisrative
Oocaclacy Seg
Beijing Planeiariio
Sport cuitural
bullding left
5,000 Socio-names
+
corner
Socio-classes
Socio-fumctions
Referiing Seg
a. Existing tasks
b. Our SocioSeg tasks
Result
Rendering
Result
Iange
Imagee
Render ing
Resilt
Ranke
Reflezt 2
S Reasoning Model
Sy
Socio
Reasoner
Existing reasoning method
d. Our SocioReasonee
Figure 1: (a) Current works segregate physical entities. (b) Our SocioSeg identifies social entities (names, functions) via multi-modal data. (c) Existing reasoning methods employ a single-stage reasoning approach. (d) Our SocioReasoner employs two-stage reasoning strategy with render- and-refine mechanism.
the heterogeneous formats and mismatched spatial granularities require complex preprocessing and alignment with satellite imagery; and (iii) because these methods trained only on predefined cat- egories, they can handle only a limited set of social semantic classes. These limitations underscore the need for a more versatile framework that can adeptly integrate diverse multi-modal geospatial data for
segmentation.
Recent advances in Vision-Language Models (VLMs) (Achiam et al., 2023; Liu et al., 2023a; Bai et al., 2025a; Mall et al., 2024; Chu et al., 2024) offer promising pathway toward creating such framework. In the natural image domain, VLMMs have already showcased their powerful visual understanding and reasoning capabilities on complex tasks like reasoning segmentation (Lai et al., 2024; Liu et al., 2025a; Wei et al., 2025: Chu al., 2025), visual grounding (Bai et al., 2025b), mathematical reasoning (Zou et al., 2024), and geolocalization (Li et al., 2024a). While some work has begun applying VLMs to satellite imagery (Li et al., 2025b; Yao et al., 2025), these efforts predominantly still focus on reasoning about physical attributes. This leaves critical gap, social semantics, which are inherently diverse and complex, demand precisely the sophisticated reasoning processes that VLMS excel at. This natural alignment inspires explore the potential of VLMs for socio-semantic segmentation.
Motivated by the aforementioned challenges and opportunities, this paper detines and tackles socio- semantic segmentation by leveraging the reasoning capabilities of VLMs. To address the critical lack of dedicated benchmark, we introduce the Urban Socio-Semantic Segmentation dataset called SocioSeg. SocioSeg is structured with three-tiered hierarchy of tasks in increasing order of com- plexity: (i) Socio-name segmentation (c.g., "a certain university""), (ii) Socio-class segmentation (e.g., "collegee" and (iii) Socio-function segmentation (e.g.. "educational"). This design means the tasks place progressively higher demands on the model's reasoning abilities. Furthermore, re- solve the data-handling bottlenecks of previous methods, SocioSeg adopts novel geospatial data representation paradigm. Instead of using raw geospatial data, which introduces the problems of access, alignment, and heterogeneity, SocioSeg unifies them into digital map layer. This paradigm highly effective: the need for protected raw data is eliminated, and the map layer is inherently spatially aligned with the satellite imagery. Building on SocioSeg, we propose SocioReasoner, a vision-language reasoning framework that simulates the human process of identifying and annotating socio-semantic entities through cross-
2

---

Published as a conference paper at ICLR 2026
modal recognition and multi-stage reasoning. More specifically, given a textual instruction with socio-semantic concepts, SocioRcasoner employs a two-stage reasoning strategy with a render-and- refine mechanism: it first generates bounding box prompts from both satellite and map imagery to localize the target region. These prompts are then fed to the Segment Anything Model (SAM) (Ravi et al., 2024) produce initial coarse segmentation. Recognizing that segmentation from a bounding box alone can be imprecise and inconsistent with the actual human annotation process, SocioReasoner proceeds to generate point prompts on the rendered mask to refine the boundary, ulti- mately generating a high-fidelity segmentation result. This entire interactive process closely mirrors the workflow of a human annotator. Since this pipeline is non-differentiable, we employ a popular reinforcement learning algorithm, GRPO (Shao et al., 2024; Guo et al., 2025), to train the frame- work end-to-end, which also effectively elicits the VLM's latent reasoning capabilities for the social semantic segmentation task. Extensive experiments show that our approach outperforms state-of- the-art segmentation basclines and exhibits strong zero-shot generalization capabilities, highlighting the potential of combining satellite imagery with rendered map context for social semantic under- standing. In summary, our contributions are:
We introduce vocio-semantic segmentation, novel and challenging segmentation and release the benchmark SocioSeg, which establishes the paradigm of rendering hetero- geneous geospatial data into a unified map image, transforming complex multi-modal challenge into visual reasoning task.
task,
We propose SocioReasoner, a segmentation framework that mimics human annotation via a multi-stage reasoning process. This non-differentiable workflow is optimized using re- inforcement learning with a dedicated reward function, effectively eliciting the model's reasoning capabilities.
Extensive empirical evidence demonstrates the effectiveness and generalization capabilities of our approach, highlighting its potential for real-world applications.
2
RELATED WORK
2.1 SEMANTIC SEGMENTATION
Semantic segmentation fundamental task in computer vision (Voulodimos al., 2018). Early deep learning methods trained models in fully supervised manner, enabling them to recognize only a predefined set of semantic categories (Ronneberger et al., 2015; Xie et al., 2021; Zhang al., 2022). With the advancement of pre-trained models, tasks such as open-vocabulary segmentation (Ghiasi et al., 2022) and referring segmentation (Wang et al., 2022) have emerged, allowing models to identify unseen categories or segment objects based on textual descriptions. More recently, the task of reasoning segmentation (Lai et al., 2024) is introduced, where the input text describes the target's function relationship rather than its visual appearance. This demands more sophisticated reasoning capabilities from the model. Notably, a significant body of current work now employs VLM-based paradigms to address reasoning segmentation tasks (Liu et al., 2025a; You & Wu, 2025; Liu et al., 2025b). These methods feed visual prompts (e.g., bounding boxes or points) derived from VLM inference into the SAM to perform segmentation, and employ reinforcement learning to elicit the model's reasoning capabilities.
Semantic segmentation from satellite imagery follows similar developmental trajectory (Kotaridis & Lazaridou, 2021). It began with fully supervised models for extracting features like buildings (Cheng et al., 2019) and roads (Sun al., 2019), and has since progressed to explorations in open- vocabulary (Li et al., 2025a; Zhu et al., 2025) and referring segmentation (Chen et al., 2025). Re- cently, some studies also begin to tackle reasoning segmentation on satellite imagery, often by us- ing closed-source vision language model to re-frame existing segmentation categories into text that requires reasoning (Li et al., 2025b). This existing work predominantly focuses on categories de- fined by physical attributes (e.g., buildings, water bodies) or categories with distinct visual features. Socio-semantic categories (e.g., schools, parks), whose boundaries and identities are determined more by social constructs than by distinct visual cues, remain significant challenge for methods that rely solely on satellite imagery. In contrast to existing work, our paper specifically targets these socio-semantic categories within urban regions.
3

---

Published as a conference paper at ICLR 2026
2.2 MULTI-MODAL APPROACHES FOR URBAN UNDERSTANDING
The task of segmenting urban social semantic entities, which we term urban socio-semantic segmen- tation, a nascent research area. While no prior work directly addresses this task, related problems exist in the field of urban science, such as land-use classification (Xiong et al., 2025) and urban functional zone (Yao et al., 2018). These studies typically fuse multimodal data, such as Points of Interest (POIs) and road networks, with satellite imagery. Their common approach involves using separate model encoders for different data modalities and then merging the extracted features for classification or segmentation (Xiong et al., 2025; Zhang et al., 2017). However, these methods, which rely on raw multi- data, face several critical bottlenecks. They are often hampered by challenges in data acquisition (due to commercial or security constraints), the complexity of han- dling heterogeneous data formats and mismatched spatial granularities, and an inability to generalize beyond limited set of predefined categories. Crucially, this highlights the fundamental difference between our task and traditional land-use classification. While the latter typically targets fixed, closed of categories, our secinssmmantic segmentation involves fine- ane-grained attributes (over 90 categories) and specific entity names, where each instance acts as a unique class. Consequently, our task aligns more closely with open-vocabulay referring, or reasoning segmentation rather than standard classification.
SOCIOSEG DATASET
Existing semantic segmentation dataset (Wang et al., 2021; et al., 2024b) from satellite imagery has been largely confined to extracting entities defined by physical attributes. To expand the scope to social semantics, we introduce the SocioSeg dataset, which is distinguished by two key features:
Hierarchical Socio-Semantic Segmentation Task Design. As illustrated in Appendix A. Fig- ure 6, we define urban socio-semantic entities across three hierarchical levels of increasing abstrac- tion and difficulty: Socio-names (e.g.. "a certain university"), Socio-classes (e.g.. "college"), and focio-functions (c.g., "educationall). This ticred structure facilitates a progressive evaluation of a model's reasoning capabilities. Above all, SocioSeg is exceptionally rich in social semantic infor- mation, containing over 5,000 Socio-names, 90 Socio-classes, and 10 focio-functions.
Multi-Modal Data with Digital Map Representation. A key innovation of the SocioSeg dataset is its unification diverse geospatial information into a single digital map layer. This representation offers several distinct advantages. First, it overcomes data accessibility issues, as publicly available map layers replace raw multi-modal data that are often proprietary or restricted. Second, the map layer is inherently co-registered with the satellite imagery. which eliminates the need for complex data alignment. Finally, this fusion into single visual modality provides rich socio-semantic cues that are crucial for enhancing a model's social reasoning capabilities.
We construct the inputs for SocioSeg by sourcing satellite images and digital maps from the Amap public API', which provides these maps both Chinese and English versions. The digital maps ren- der only basic geospatial information, including roads and points of interest. We then collected the ground-truth socio-semantic labels for the corresponding regions. (Further details on the annotation procedure and dataset statistics are available in Appendix 1.1. As a result, the SocioSeg dataset comprises over 13,000 samples distributed across the three hierarchical tasks. Each sample consists of satellite image, digital map. and a corresponding the dataset into training, validation, and test sets using 6:1:3 ratio, ensuring that the sample counts and class distributions for each hierarchical task are consistent across all splits.
mask label. We partitioned
4 SOCIOREASONER FRAMEWORK
4.1 HUMAN-LIKE REASONING SEGMENTATION PROCESS
Prevailing sassonnng-segmentaane methods (Liu et al., 2025b; Yao et al., 2025) typically follow a single-stage pipeline: a Vision- LangugLe Model (VLM) generates visual prompts (e.g., bounding box), which are then fed into frozen SAM to produce the final mask. Because the weights of
Amap API Documentation. https:/t///// httppa://bbs.. com/ Accessed: 2025-05-14.
4

---

Published as a conference paper at ICLR 2026
SAM are fixed, these methods lack direct control over the output quality, often resulting in coarse inaccurate segmentation. In contrast, our SocioReasoner framework (as shown in Figure 2) employs two-stage reasoning strategy with soneer-nn-refingg mechanism to emulate the sequential work- flow human annotator. This multi-stage approach enhances precision and makes the model's inference steps transparent and interpretable.
Stage-1 (Localization): Emitting a set of 2D bounding boxes. Let the VLM be denoted by F. Given satellite image I,, digital map I and textual instruction to. the VLM emits of 2D bounding boxes B = (b: [b: to localize candidate target regions:
B F(1,, Ims ts).
(1)
These bounding boxes are supplied to a pre-trained segmentation model, SAM (S) (Ravi et al., 2024), to produce preliminary coarse mask M
M. = prompt B).
Stage-2 (Refinement):
Emitting both a set of bounding boxes and points. Recognizing that segmentation from bounding boxes alone be imprecise, provide visual feedback the VLM by rendering both the boxes and the coarse mask onto the inputs. A rendering function overlays B and M. onto the satellite image I, and the digital map I,, producing a pair of rendered images (Ig,r ) for re-evaluation:
Irr D(I, B, M,), I D(I B, M.).
Conditioned on (Is,r: Im.r) and the instruction tp. the VLM emits set of bounding boxes B together with points P {p)
(B, F(s,; Lm.rs (p).
Finally, the comprehensive set of prompts (bounding boxes and points) is fed back into SAM to yield the final mask My: My = S(,,, prompt = {B, P)).
(5)
By decomposing the segmentation challenge into this sequence of localization and refinement, So- cioReasoner achieves superior accuracy and provides an explicit reasoning chain. As this entire pipeline is non-differentiable, we leverage reinforcement learning to optimize the VLM's policy for generating these sequential prompts.
4.2 END TO END REINFORCEMENT LEARNING OPTIMIZATION
We optimize the non-differentiable, multi-stage prompting policy of SocioReasoner using reinforce- ment learning with Group Relative Policy Optimization (GRPO) (Guo et al., 2025). single Vision- Language Model (VLM) policy is shared across both stages and emits structured textual outputs that encode prompts for SAM. The environment parses these outputs, executes SAM with the parsed prompts, and returns scalar reward.
Stage-1 (Localization) Optimization. Given an input x; (I,, Im, 4t), the policy mg stochas- tically generates completion yı that encodes set of bounding boxes. The environment parses Y1 to obtain B, runs SAM to produce a coarse mask Mg. and returns a stage-1 reward R3 (y13 x1) comprising: (i) binary syntax reward ensure valid JSON output, (ii) localization accuracy term for the predicted boxes, and (iii) a reward for matched object count. GRPO is applicd per input by drawing G completions {v(*) G. Jg=1 computing rewards { ) g== and defining group-relative baseline by (x1) Eg R("), with advantages A(") R(() - by (x1). The stage- 1 objective is clipped PPO-like surrogate with KL regularization against a frozen reference policy Reef.
(i(0) - -
-
- Σ min (r(o)A(0) clip(r 1)
-,,++)
(6)
g=1 (e))
B KL TT( (-(x) (=)((-
where (2) no xo( vf. (x)) is the token-level importance ratio. The hyperparameters and
Toald (v(.)) (o) xx)
control the PPO clipping and KI. regularization, respectively.
5

---

Published as a conference paper at ICLR 2026
Segmentation
Segmentation
Segmentation
"wchool
bbox
"school"
bbox&points
SAM
SocioRcasoncr
SecioRcasoner
GRPO Training
Share weight
GRPO Training
Reward functions
1.. Am
to
panally
Bet Peliay
Pulicr
Foraat reward
Accuracy reward
Greap
SocioRcasoner
Length reward
Objective
Keosoning chain
phon/points
Rewards
Advantages
Figure 2: SocioReasoner Framework. Given satellite image, digital map, and textual instruc- tion, the VLM first generates bounding boxes to localize candidate regions. These boxes are fed into SAM to produce coarse mask. The boxes and mask are then rendered onto the inputs for re-evaluation. The VLM emits boxes and points, which are again fed into SAM to yield the final mask.
Stage-2 (Refinement) Optimization. Conditioned the rendered visual feedback and the coarse mask, the policy refines the prompts. We construct X₂ = (I..r: Im.r. tp. M.) by overlaying the stage-1 boxes and coarse mask using the renderer D. The policy me emits y2 that encodes bounding boxes and points. The environment parses y2 to obtain {B, P). runs SAM produce the final mask My, and returns a stage-2 reward R2(y2; comprising: (i) a binary syntax reward for valid ISON, (ii) pixel-level IoU term for M,. and (iii) reward for point length. GRPO sampling, paseline/advantage computation, and the clipped surrogate with KL regularization follow the same formulation in slage-1.
Training Schedule. Within a single reinforcement learning step, execute both stages sequen- tially: (i) sample, evaluate, and update with Li (0) using stage-1 rollouts; (ii) construct the stage-2 inputs from the stage-1 outputs, then sample, evaluate, and update with (z( This two-stage pro- cedure aligns optimization with the sequential localization-refinee refinement workflow. Detailed formula- tions of the rewards R2 and R2 are provided in the Appendix A.2.2. The overall training algorithm is summarized in Algorithm 1.
EXPERIMENTS
BASELINES AND EVALUATION METRICS
We primarily compare against three families of methods: (i) standard semantic segmentation mod- els, including the CNN-based UNer (Ronneberger et al., 2015) and the Transformer-based Seg- Former (Xie al., 2021): (ii) state-of-the-art reasoning segmentation for natural images, including VisionReasonerr (Liu et al., 2025b), Seg-RI (You & Wu, 2025), and SAM-RI (Huang et al., 2025); (iii) of-te-of-theart satellite image segmentation methods, including the open-vocabulary segmen- tation SegEarth-OV (Li et al., 2025a), referring segmentation RSRefSeg (Chen et al., 2025), and reasoning-baat based approaches Earthrt (Li et al., 2025b) and RemoteReasoner (Yao et al., 2025). Because SocioSeg provides two images (satellite and digital map), we adapt all VLM-based base- lines to accept dual-image inputs; for methods (RSRefSeg and SegEarth-RR -1) that do not support multiple images, provide only the satellite image. All baselines training split to ensure fair comparison. Additionally, given the challenging nature the SocioSeg benchmark, employ off-the-shelf Large Multimodal Models reference models to provide
re-trained the SocioSeg

---

Published as a conference paper at ICLR 2026
Socio-time
Jinan Zoo
Socie-class
"Express Delivery"
Socie-elass
"Business Office Building
"Park and greenspace
Query
Digital map
GT
VisionReasoner
SegFarth-RI
Ours stage-1
Ours stage-2
Ours Stage-1
"Jinan Zoo"
map with marked, including 'jinan Zoo'. The second image is satellite view of a large area with buildings and roads clearly visible, including the Jinan Zoo </thinle> [["bbox_2d 2d" [360, 268. 538, 383]]]
<think
image
Ours Stage-2
"Jinan Zoo
<thinlchThc bounding (bbox) coordinates [360, 383] is likely to be the area of interest. The points added to the bbox help to better represent the area of interest by highlighaing the building surrounding
key features
zoo, such greanery. </think>> (("box__dd" 2d": [360, 268, 538, 383), "points": [[452, 287]. [487, 306]]}]
Out-of-domain
Figure 3: Visualization of the SocioReasoner results. The top panel shows comparison between the results of SocioReasoner (with both stages visualized) and competitive baselines. The bottom- left panel illustrates the reasoning process of SocioReasoner. The bottom-right panel displays the visualization results of SocioReasoner on the out-of-domain dataset.
more comprehensive evaluation. Detailed experimental results for these models are provided in the Appendix A.4. For evaluation, follow previous work (Lai et al., 2024) in reporting cloU and gloU. Additionally, we employ the Fl score to assess instance-level performance.
5.2 COMPARISON WITH STATE-OF-THE-ARY METHODS
Comparison with tfate-of-the-a methods on the SocioSeg test set is presented in Figure and Ap- pendix A.6. The quantitative results are presented in Table 1, with results grouped by task for clarity. Our SocioReasoner framework consistently outperforms all baselines across all three hierarchical tasks, demonstrating its effectiveness in handling the complexities of socio-semantic segmentation. This performance gain underscores the advantage of our human-like reasoning process and the use of rendered map context in enhancing the model's understanding of social semantics. However, be- cause SocioReasoner simulates a multi-step human reasoning process, its inference time is longer compared to other methods. We provide detailed analysis of SocioReasoner'ss inference time in Ap- pendix 1. Additionally, we illustrate the accuracy for individual Socio-classes Figure 4, Our method consistently outperforms baselines the t0p-20 most frequent socio-class categories. Re-

---

Published as a conference paper at ICLR 2026
Beaketaal Armu
Amiiiaaatios
Secerr buniza
Sbospit Mall
Out Dalership
Transettaiion stations
Buises office
Ous
School
Cospechesivve Qytiessio
1/Net
Segformer
Spot
School
Spoct
Conunescial
SeglEarth OV
RSRefSeg
Pobtic Saurity
lnchiiton Consorito
SagEarth-RI
Parking Let
Carcitment
Roadenial
Edbcatooaa
VistonReasomer
Park Square
Funiskitg
Seg-R1
Legisiks Expess
Hospial
Pak grvespace
Estortial
SAM-RI
Libeay
Dand
Indeonal Park
Figure 4: Per-class accuracy comparison across Socio-classes and Socio-functions. We select the top-20 most frequent Socio-classes (left) and all Socio- (unetiouuuntiios(rihht for visualization.
garding the pecio-function tasks, our approach achieves state-of-the-ae results across all categories, with the exception of Business Office and Revidential. We attribute these failures to error propaga- tion: in certain samples, the initial bounding box localization (Stage-1) deviates significantly from the ground truth. Consequently, the point prompts generated during the refinement phase (Stage- 2) tend to exacerbate rather than correct this initial deviation, as illustrated in the failure cases in Figure 12.
Comparison with standard semantic segmentation methods. Since standard semantic segmenta- tion models incapable of processing multimodal inputs, they fail to perceive the social semantic information inherent the SocioSeg task. Consequently, under this setting, the task for standard models degenerates into binary classification problem. As observed, UNet and SegFormer exhibit inferior performance on SocioSeg compared to the other two categories of methods, whereas our approach significantly outperforms them. Additionally, we tested the performance of Socio-class and Socio-function under a multi-class semantic segmentation setting. As shown in Table 10, be- cause social semantic categories lack distinct visual features in satellite images, standard semantic segmentation methods perform even worse under this setting, further highlighting the challenging nature of the SocioSeg task.
Comparison with natural image reasoning segmentation methods. Similar to SocioReasoner, VisionReasoner, Seg-R1, and SAM-RI all support multi-image inputs and therefore perform rela- tively well on SocioSeg. Notably, SAM-R1 (Huang et al., 2025) lacks constraints on the length of the output point prompts; in our reproduction, emits a large number of point coordinates, which degrades performance. These methods freeze SAM parameters and perform single-stage inference. In contrast, our SocioReasoner framework surpasses these methods by a notable margin across all metrics. This improvement is attributable to our multi-stage reasoning process that mimics human annotation, providing reflection and refinement capabilities that lead to more accurate segmentation.
Comparison with advanced satellite image segmentation methods. SegEarth-OV completely freezes the CLIP encoder, limiting its recognition capabilities to the categories present in CLIP's pre-training data. Since SocioSeg features novel semantic categories related to social attributes, this method fails to function effectively on the SocioSeg task. RSRefSeg and SegEarth-RI, which are designed for segmenting physical attributes and support only single satellite image input, show limited performance socio-semantee tasks. However, because they are trained in fully super- vised manner without freezing the mask decoder, they achieve some performance gains. In contrast, our approach leverages multimodal reasoning, effectively integrating satellite imagery with digi- tal map context to capture nuanced social semanties. RemoteReasone adopts design similar to VisionReasoner, supports multi-image inputs, and performs well on SocioSeg. Our SocioReasoner framework outperforms RemoteRcasoner, highlighting the benefits of our two-stage localization and refinement process, which enables more precise segmentation through iterative reasoning.
5.3 ABLATION STUDIES
We ablate three core design choices of SocioReasoner: the training/inference scheme (single-stage two-stage), the impact of reinforcement learning (RL), and the number of points issued in the
8

---

Published as a conference paper at ICLR 2026
Socio-name
Socio-class
Socio-function
All dataset
Method
cloU
gloU
cloU
gloU
F1
cloU
gloU
clo
gloU
F1
UNet
10.9
9.4
8.0
12.6
11.9
11.2
11.1
10.6
10.4
11.7
10.7
10.0
Segformer
22.0
19.6
18.1
22.4
21.4
19.5
21.4
20.2
17.9
22.1
20.5
18.7
VesionReasoner
48.5
50.9
58.4
44.4
49.3
55.5
36.3
41.8
45.0
44.0
48.5
54.3
Seg-RI
46.0
48.1
50.4
40.4
44.7
45.2
34,5
39.5
36.5
41.0
45,0
45.2
SAM-RI
25.6
25.4
37.2
22.3
23.8
32.1
17.7
19.9
25.2
22.5
23.7
32.4
SegEarth-OV
3.3
3.3
0.0
3.8
3.8
0.0
4.2
4.2
0.0
3.7
3.7
0.0
RSRetSeg
27.1
25.4
30.9
30.7
30.6
35.3
28.7
28.8
30.8
29.0
28.3
32.8
SegEarth-RI
36.9
42.1
46.9
38.9
45.1
50.0
39.5
45.6
47.4
38.3
44.1
48.4
RemoteReasoner
46.6
49.5
56.1
42.9
48.0
53.9
38.0
43.5
47.2
43.2
47.7
53.3
Ours
52.6
55.7
64.6
47.6
52.8
60.1
40.6
46.9
50.3
47.9
52.8
59.7
Table I: Comparison with state-of-the-an methods on SocioSeg test set, split by task groups for read- ability. The best performance each column is highlighted bold. The second best is underlined. Baselines are re-trained on the SocioSeg training split to ensure fair comparison.
Locdintion aleU
0.6
0.5
Stege:. Refucrent
Reward
0.4
glot3
0.4
Nbxx point
0.0
100
200
0.0
100
bbox with points
bbox with poirss
Stop
SNp
Figure 5: (a) Sum reward during training. It shows the sum of rewards across training steps in the two-stage workflow. (b) Multi-stage gloU during training. shows the gloU improvement across training steps in the two-stage workflow. (c) Different number of points. It visualizes the result of SocioReasoner in the refinement stage with different numbers of points.
second stage. Results are summarized Table 2, Table 3, and Table The full results of each ablation setting are provided in Appendix A.5.1.
Impact of the training/inference scheme. the "w/o reflection" configuration, the model bypasses the two-stage workflow and instead produces bounding boxes and points in single stage, equivalent to VisionReasoner's 's one-step prompting. This setting performs the worst for two reasons: (i) without an iterative process, the model cannot self-correct after observing the coarse mask; and (ii) it must solve a complex planning-ant and--arsing problem in one shot (jointly synthesizing boxes and points in long structured output), which increases failure rates. In the "w/o refinement" ablation, use the model trained with the two-stage pipeline but halt the inference process after Stage-1. The output from this initial localization stage is used directly as the final result, completely bypassing the refinement stage. The complete pipeline ("Ours"),, which overlays stage-1 outputs and emits both boxes and points, achieves the best results. Figure 5b shows the evolution of mask IoU across the two stages during RL training: stage-1 accuracy is initially higher because the model focuses more on localization early on; as training progresses, the model increasingly leverages points improve the mask, leading to steady rise in stage-2 accuracy. This finding highlights the effectiveness of our multi-stage reasoning process, where the refinement stage contributes enhancing segmentation quality.
Impact of the number of points the refinement stage. our reward function, the parameter directly controls the number of point prompts generated in the refinement stage. We present the experimental results for different numbers of points Table and visualize the corresponding qualitative results in Figure 5c. We observe that single point prompt often fails cover the entire target, while the model struggles to learn a stable distribution for three points, with marginal performance gains compared to using two. Therefore, select points the final design choice.

---

Published as a conference paper at ICLR 2026
Table 2: Ablation of multi-stage design.
Table 3:
Ablation of point number.
All datasct
All dataset
Method
Method
cloU
gloU
F1
cloU
gloU
w/o reflection
44.0
48.5
54.3
point refinement
47.6
51.2
58.0
w/o refinement
46.4
50.8
57.5
points refinement
47.9
52.8
59.7
Ours
47.9
52.8
59.7
points refinement
48.9
52.3
58.8
Table 4:
Generalization of SocioReasoner, where ID and OOD refer to in-domain and out-of-
domain, respectively.
ID
OOD (Map Style)
OOD (New Region)
Method
cloU
gloU
F1
cloU
gloU
F1
cloU
gloU
F1
VissonReasoner (SFT)
44.1
47.2
52.1
38.8
40.9
45.5
22.5
24.7
VisionRcasoner (RI.)
44.0
48.5
54.3
42.0
44.4
51.2
32.8
34.4
35.0
Ours (SFT)
47.1
51.4
57.8
39.7
42.0
46.9
30.1
32.3
31.5
Ours (RL)
47.9
52.8
59.7
45.1
49.1
57.7
40.2
43.4
42.9
Impact of the RL. Figure 5a illustrates the reward trajectory during training. The consistent upward trend demonstrates that RL effectively optimizes SocioReasoner's human-like workflow. To further demonstrate the effectiveness of our training strategy. we compare SocioReasoner trained via RI. against Supervised Fine-Tuning (SFT) baseline. We evaluate performance not only the in- domain (ID) Amap data but also on two challenging out-of-domain (OOD) scenarios using Google Maps tiles, as reported in Table 4. The "OOD (Map Style)" setting tests robustness cartographic style shifts. Furthermore, we introduce specific "OOD (New Region)" setting evaluated on a newly constructed dataset sampled from five global cities: Tokyo (Asia), New York (North America), São Paulo (South America), London (Europe), and Nairobi (Africa). This dataset, which is detailed in Appendix A. 1.2. comprises 3,200 samples covering categories, including 24 novel classes unseen during training. While the SFT baseline suffers significant performance degradation these OOD tasks, our RL method maintains high robustness, achieving superior results on the diverse regional dataset. A similar trend is observed in the VisionReasoner baseline, where the RL-optimized version consistently outperforms the SFT variant, further corroborating the efficacy RL in enhancing reasoning capabilities. This indicates that minimizing the RL objective, which directly optimizes the non-differentiable IoU, enables the model to learn more generalized geometric reasoning policies that transfer effectively across different map styles and geographic regions.
CONCLUSION
This paper introduces the task of urban socio-semantic segmentation and present SocioSeg, the first benchmark for this challenge. SocioSeg's 's key contribution is a new paradigm that renders heteroge- neous geospatial data into unified map. transforming complex multi-modal problem into visual reasoning task. We also propose SocioReasoner, a framework that leverages Vision-Language Mod- els to mimic the human annotation process through multi-stage reasoning segmentation workflow. By optimizing this non-differentiable pipeline with reinforcement learning, we effectively elicit the model's is latent reasoning capabilities. Extensive experiments demonstrate that our approach outper- forms existing methods and exhibits strong zero-shot generalization to unseen map sources. Our work highlights the potential of VLM reasoning for complex geospatial analysis.
ACKNOWLEDGMENTE
This work was supported by Amap, and in part by the National Key Research and Development Program of China under Grant 2024YFB3909001, in part by the National Natural Science Founda- tion of China under Grant 42371321, and part by the Key Research and Development Program of Hubei Province under Grant 2025BAB024.
Ethics Statement Our research utilizes publicly accessible satellite and map data, specifically from the Amap public API, create the SocioSeg dataset. The collection, utilization, and public
10

---

Published as a conference paper at ICLR 2026
lease of this data, well our accompanying codebase, have been explicitly authorized by Amap. Consequently, both the SocioSeg dataset and the codebase are open- open-soureed under the Apache Li- cense 2.0. The manual annotation process was strictly confined to identifying and labeling public and private functional zones, without collecting or inferring any personally identifiable information (PIT).
Reproducibility Statement To ensure the full reproducibility of our findings, we have provided comprehensive implementation details throughout the paper. The construction and statistics of our SocioSeg benchmark are detailed in Sec 3 and Appendix A.1. The architecture of the SocioReasoner framework, including the multi-stage reasoning process, is described in Sec 4.1. Key details for the reinforcement learning optimization, including the reward function design and GRPO training algorithm, are presented in Sec 4.2 and Appendix A.2. In line with our commitment to open science, the SocioSeg dataset and source code will be made publicly available.
LLM clarification We clarify the use of Large Language Models (LLMs) in the preparation of this manuscript. Specifically, LLMs were employed for two main purposes: translation of initial drafts from our native language and subsequent language polishing. This process involved correcting grammatical errors, improving sentence structure, and enhancing the overall readability and flow of the text. It is crucial to emphasize that all core scientific content, intellectual contributions, and original ideas presented in this paper are exclusively the work of the human authors. This includes the formulation of the research problem. the development of the SocioReasoner framework, the creation of the SocioSeg dataset, the experimental design, and the analysis of the results. The LLM served strictly as a writing aid and was not involved in any conceptual or analytical aspect of this research.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 087774, 2023.
Mehdi Alidadi and Ayyoob Sharifi. Effects of the built environment and human factors on the spread of covid-19: A systematic literature review. Science of The Total Environment, 850:158056 2022. ISSN 0048-9697
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 2025a.
Sule Bai, Mingxing Li, Yong Liu, Jing Tang, Haoji Zhang, Lei Sun, Xiangxiang Chu, and Yansong Tang. Univg-rl: Reasoning guided universal visual grounding with reinforcement learning. arXiv preprint arXiv:2505. 250i::2355...,311,,, 2025b.
Matteo Bruno, Hygor Piaget Monteiro Melo, Bruno Campanelli, and Vittorio Loreto. A universal framework for inclusive 15-minute cities. Nature Cities, 1:633-641, 2024.
György Büttner. CORINE Land Cover and Land Cover Change Products, pp. 55-74. Springer Netherlands, Dordrecht, 2014. ISBN 978-94-007--769-3,
Keyan Chen, Jiafan Zhang, Chenyang Liu, Zhengxia Zou, and Zhenwei Shi. Rsrefseg: Referring remote sensing image segmentation with foundation models. In 2025 IEEE International Geo- science and Remole Sensing Symposium (IGARSS). IEEE, 2025.
Dominic Cheng, Renjie Liao, Sanja Fidler, and Raquel Urtasun. Damet: Deep active ray network for building segmentation. In Proceedings of the IEEE/CVF conference computer vision and paltern recognition, pp. 7431-7439, 2019.
Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang. Xiaofei Sun, Yiming Hu, Xinyang Lin. Bo Zhang, et al. Mobilevlm v2: Faster and stronger baseline for vision language model, arXiv preprint arXiv:2402.03766 037666 2024.

---

Published as a conference paper at ICLR 2026
Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.02546, 2025.
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In European conference on computer vision, pp. 540-557. Springer, 2022.
Peng Gong, Bin Chen, Xuecao Li, Han Liu, Jie Wang, Yuqi Bai, Jingming Chen, Xi Chen, Lei Fang, Shuailong Feng, et al. Mapping essential urban land use categories in china (euluc-china): Preliminary results for 2018. Science Bulletin, (83)):82-187, 2020.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Xin Guo, Jiangwei Lao, Bo Dang, Yingying Zhang, Lei Yu, Lixiang Ru, Liheng Zhong, Ziyuan Huang, Kang Wu, Dingxiang Hu, et al. Skysense: A multi-modal remote sensing foundation model towards universal interpretation for earth observation imagery. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 27672-27683, 2024.
Renlong Hang, Ping Yang. Feng Zhou, and Qingshan Liu. Multiscale progressive segmentation net- work for high-resolution remote sensing imagery. IEEE Transactions on Geoscience and Remote Sensing, 60:1-10: 102, 2022.
Yunfeng Hu and Yueqi Han. Identification of urban functional areas based on poi data: case study of the guangzhou economic and technological development zone. Sustainability, 11(5): 1385, 2019.
Haosheng Huang, Georg Gartner, Jukka M Krisp, Martin Raubal, and Nico Van de Weghe. Location based services: ongoing evolution and research agenda. Journal of Location Based Services, 12 (2):63-93, 2018.
Jiaqi Huang, Zunnan Xu, Jun Zhou, Ting Liu, Yicheng Xiao, Mingwen Ou, Bowen Ji, Xiu Li, and Kehong Yuan. Sam-rl: Leveraging sam for reward feedback in multimodal segmentation via reinforcement learning. In Advances in Neural Information Processing Systems, 2025.
loannis Kotaridis and Maria Lazaridou. Remote sensing image segmentation advances: A meta- analysis. ISPRS Journal of Photogrammetry and Remote Sensing, 173:309-322, 2021.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9579-9589, 2024.
Kaiyu Li, Ruixun Liu, Xiangyong Cao, Xueru Bai, Feng Zhou, Deyu Meng, and Zhi Wang- Segearth-ov: Towards training- free spea-vocabulary segmentation for remote sensing images. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 10545-10556, 2025a.
Kaiyu Li, Zepeng Xin, Li Pang, Chao Pang, Yupeng Deng, Jing Yao, Guisong Xia, Deyu Meng, Zhi Wang, and Xiangyong Cao. Segearth-rl: Geospatial pixel reasoning via large language model. arXiv preprint (XXv::554400944: 2025b.
Ling Li. Yu Ye, Bingchuan Jiang, and Wei Zeng. Georeasoner: Geo-localization with reasoning in street views using large vision-language model. In Forty-first International Conference Machine Learning, 2024a.
Yansheng Li. Bo Dang, Wanchun Li, and Yongjun Zhang. Glh-water: A large-scale dataset for global surface water detection in large-size highrrsstotootiiinnnn satellite imagery. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 22213-22221, 2024b.
Yansheng Li, Yu Wang, Lei Yu, Bo Dang, Gang Xu. Zhenyu Zhong, Yuning Wu, Xin Guo, Kang Wu, Zheng Li, et al. Learning to reason over multi granularity knowledge graph for zero-shot urban land-use mapping. Remote Sensing of Environment, 130:114966 2025c.
12

---

Published as a conference paper at ICLR 2026
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 11:34892-349111 2023a.
Qin Liu, Zhenlin Xu, Gedas Bertasius, and Marc Niethammer. Simpleclick: Interactive image segmentation with simple vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22290-222000 2023b.
Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg- zero: Reasoning-chain guided segmentation via cognitive reinforcement. arXiv preprint arXiv:2503.06520, 2025a.
Yuqi Liu, Tianyuan Qu, Zhisheng Zhong, Bohao Peng, Shu Liu, Bei Yu, and Jiaya Jia. Vision- reasoner: Unified visual perception and reasoning via reinforcement learning. arXiv preprint arXiv:2505.12081, 2025b.
Utkarsh Mall, Cheng Perng Phoo, Meilin Kelsey Liu, Carl Vondrick, Bharath Hariharan, and Kavita Bala. Remote sensing vision-language foundation models without annotations via ground remote alignment. In The Tweelfh International Conference on Learning Representations, 2024. URL https:/ httpss///ppereveeeee nnet/ forum? (et::iddww:tw69wwee..
OpenStreetMap contributors. Planet dump retrieve from https://plant..ommorrg https: h /www. openst org, 2017.
Jorge Patino and Juan C Duque. A review of regional science applications of satellite remote sensing in urban settings. Computers, Environment and Urban Systems, 37:1-17, 2013.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.007/ 007144 2024.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234 241. Springer, 2015.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Chuanji Shi, Yingying Zhang, Jiaotuan Wang. Xin Guo, and Qiqi Zhu. Multimodal urban areas of interest generation via remote sensing imagery and geographical prior. International Journal of Applied Earth Observation and Geoinformation, 136: 104326, 2025.
Jee- Bhe-Young Sun, Seung-Wook Kim, Sang- Won Lee, Yon Kim, and Sung-Jea Ko. Reverse and boundary attention network for road segmentation. In Proceedings the IEEE/CVF international conference on computer vision workshops, pp. 0-0, 2019.
Athanasios Voulodimos, Nikolaos Doulamis, Anastasios Doulamis, and Eftychios Protopapadakis. Deep learning for computer vision: brief review. Computational intelligence and neuroscience, 2018(1):006849,, 2018.
Junjue Wang, Zhuo Zheng, Ailong Ma, Xiaoyan Lu, and Yanfei Zhong. Loveda: A remote sensing land-cover dataset for domain adaptive semantic segmentation. arXiv preprint arXiv:2110.08733, 2021.
Weixun Wang, Shaopan Xiong, Gengru Chen, Wei Gao, Sheng Guo, Yancheng He, Ju Huang, Jiaheng Liu, Zhendong Li, Xiaoyang Li, et al. Reinforcement learning optimization for large- scale learning: An efficient and user-friendly scaling library. arXiv preprint wwXiv:2506.06122 2025.
Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11686-11695, 2022.
13

---

Published as a conference paper at ICLR 2026
Fei Wei, Xinyu Zhang, Ailing Zhang, Bo Zhang. and Xiangxiang Chu. Lenna: Language enhanced reasoning detection assistant. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1-5. IEEE, 2025.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34:12077 2021.
Shuping Xiong, Xiuyuan Zhang, Haoyu Wang, Yichen Lei, Ge Tan, and Shihong Du. Mapping the first dataset of global urban land uses with sentinel-2 imagery and poi prompt. Remote Sensing of Environment, 327:1 14824, 2025.
Xiaojun X Yang. Urban remote sensing: monitoring, synthesis and modeling in the urban environ- ment. John Wiley & Sons, 2021.
Liang Yao, Fan Liu, Hongbo Lu, Chuanyi Zhang, Rui Min, Shengxiang Xu, Shimin Di, and Pai Peng. Remotercasoner: Towards unifying geospatial reasoning workflow. arXiv preprint 1xXiv:2507.19280. 2025.
Yao Yao, Xia Li, Xiaoping Liu, Penghua Liu, Zhaotang Liang. Jinbao Zhang, and Mai. Sensing spatial distribution of urban land use by integrating points-of-interest and google word2vec model. International Journal of Geographical Information Science, 31(4):825-848 2017.
Zijun Yao, Yanjie Fu, Bin Liu, Wangsu Hu, and Hui Xiong. Representing urban functions through zone embedding with human mobility pattems.. In Proceedings of the Twenty-Seventh Interna- tional Joint Conference on Artificial Intelligence (ICAA--8), 2018.
Zuyao You and Zuxuan Wu. Seg-rl: Segmentation can be surprisingly simple with reinforcement learning. arXiv preprint 2xXiv:2506.22224 2025.
Bowen Zhang, Zhi Tian, Quan Tang, Xiangxiang Chu, Xiaolin Wei, Chunhua Shen, et al. Segvit: Se- mantic segmentation with plain vision transformers. Advances in Neural Information Processing Systems, 35:4971 1-992,,, 2022.
Xiuyuan Zhang, Shihong Du, and Qiao Wang. Hierarchical semantic cognition for urban functional zones with vhr satellite images and poi data. ISPRS Journal Photogrammetry and Remole Sensing, 332:170-184, 2017.
Yu Zheng, Fengli Xu, Yuming Lin, Paolo Santi, Carlo Ratti, Qi R Wang, and Yong Li. Urban planning in the era of large language models. Nature Computational Science, pp. 1-10, 2025.
Yanfei Zhong, Bowen Yan, Jingjun Yi. Ruiyi Yang, Mengzi Xu, Yu Su, Zhendong Zheng, and Liangpei Zhang. Global urban high-resolution land-use mapping: From benchmarks to multi- megacity applications. Remote Sensing of Environment, 298: 13758, 2023.
Qi Zhu, Jiangwei Lao, Deyi Ji, Junwei Luo, Kang Wu, Yingying Zhang, Lixiang Ru Jian Wang, Jingdong Chen, Ming Yang, al. Skysense-o: Towards open- world remote sensing interpretation with vision-centric visual-language modeling. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 4733-11744, 2025.
Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. Xixiv preprint arXiv:241 1.0011100066 2024.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 DATASET DETAILS
distribution
Socio-function distribution
Socio-name wordcloud
Soci-clas
Cormuniial service
City
Sport
Haraha
40%
Sedi-fanczies
Palcod
Teechol Paire
Industr
Center
Demnea office
212
(2390)
Tompetiiinoation
Bullcing
China
Edacaionl
People
Ishaai
Ltd
Medial
206
400
Socio-class distribution
Munber
Dataset samples
Socie-name
Socio-class
Sacio-furction
Bejing Freetten
Pest Bllssen Pork
Senica
Reikeekk
Park and
Cencir
Liberr
Figure 6: The SocioSeg dataset overview. (a) Sample distribution across the three hierarchical tasks. (b) Socio-function class distribution. (c) Socio-name word cloud. (d) Socio-class distribution. (e) Sample examples from SocioSeg, including satellite images, digital maps, and socio-semantic mask labels.
A.1.1 SOCIOSEG DATASET
The SocioSeg dataset is constructed entirely from data provided by Amap, offering comprehensive geographic coverage of all provinces and major cities across China. The input modalities, namely satellite images and digital maps, are acquired via the public Amap API. The ground-trutt labels are derived from Amap's is Area of Interest (AOI) data. Figure illustrates the data construction process.
15

---

Published as a conference paper at ICLR 2026
Sample 40k+ AOI
Get sample's is boundary
Amap dats
AOI (4000k++)
Add other AOIs in each boundary
Final datasct 13k samples
Amap
Amap (bouddry))
Verify Map Image
Verify Satellite Image
Map image
Satellite image
Figure 7: The SocioSeg dataset overview.
To clarify the semantic scope, we structure these labels into three hierarchical levels intrinsically linked to daily social activities: the Socio-Name level corresponds to the specific name of cach AOI instance; the Socio-Class level adopts the standard third-level POI taxonomy from Amap, classification widely utilized in prior studies (Hu & Han, 2019; Yao et al., 2017); and the Socio- Function level is derived from established urban functional zone definitions (Gong al., 2020; Li et al., 2025c).
To adapt this source data for research, performed several refinement steps, encompassing reformatting the vector-based AOI data into rasterized semantic masks and conducting a rigorous quality assurance process. Specifically, we manually verified the alignment between the AOI poly- gons and the actual physical boundaries, rigorously filtering the dataset from an initial pool of ap- proximately 40,000 samples down 13,000 to exclude misaligned ambiguous annotations. To further quantify the annotation quality and reproducibility, we conducted an inter-annotator agree- ment study with three independent annotators Kappa coefficient of 0.854. This ensures that each pixel is precisely classified into its correspond- ing focio-functional category, enhancing the dataset's overall fidelity and reliability. The resulting SocioSeg benchmark is thus rich in socio-semantic information, providing a robust foundation for urban aocio-semantic segmentation research.
random subset of 500 samples, yielding Cohen's
Figure 6 offers comprehensive overview of the SocioSeg dataset. Specifically, subfigure (a) illus- trates the sample distribution across the three hierarchical tasks, underscoring the dataset's balance and diversity. Subfigures (b) and (d) present the class distributions for the socio-function and socio- class tasks, respectively, showcasing the variety of categories included. A word cloud in subfigure (c) visualizes the frequency and prominence of the socio-name labels. Finally, subfigure (e) provides qualitative examples from the dataset, displaying corresponding satellite images, digital maps, and socio-semantic masks that effectively demonstrate the data's richness and complexity.
102
polc
Mayyy
330
130
220
120
Figure 8: The SocioSeg OOD (New Region) dataset distribution.
16

---

Published as a conference paper at ICLR 2026
A.1.2 SOCIOSEG OUT-OF-DISTRIBUTTT DATASET
To rigorously evaluate the generalization capabilities of our model, introduce two distinct Out- Of-Distribution (OOD) datasets: Map Style and New Region.
OOD (Map Style). This setting is designed to assess the model's robustness shifts in cartographic rendering and symbolization. We utilize the original test set from the SocioSeg benchmark. retaining the original satellite imagery and ground-truth labels. However, we replace the imput digital map modality, originally sourced from Amap, with tiles fetched from Google Maps. This substitution isolates the impact of map style, requiring the model to generalize new cartographic domain without any fine-tuning.
OOD (New Region). Figure shows the distribution of the SocioSeg OOD (New Region) dataset. This setting evaluates performance across diverse geographic environments. We constructed this dataset using OpenStreetMap (OSM) (OpenStreetMap contributors, 2017) AOI data, strietly ad- hering to the same data construction and quality assurance pipeline employed for the primary So- cioSeg dataset. To ensure global diversity, we selected five representative cities from different con- tinents: Tokyo (Asia), New York (North America), São Paulo (South America), London (Europe), and Nairobi (Africa). While the input digital maps are sourced from Google Maps, the ground truth labels were derived from OSM and underwent manual filtering to ensure high fidelity. The resulting dataset comprises 3,200 samples focused on the socio-class task. It encompasses 80 distinct cate- gories, importantly including 24 novel categories that were not present in the training set, thereby posing significant challenge for sero-shot generalization.
A.2
IMPLEMENTATION DETAILS
A.2.1 GRPO OPTIMIZATION DETAILS
We train SocioReasoner with the two-stage end-to-end GRPO algorithm. The training process is summarized in Algorithm 1. In contrast to the single-stage training of existing methods, SocioRea- soner's 's process includes two rounds of RL sampling and policy updates, all while utilizing a shared of model parameters.
A.2.2 REWARD FUNCTION DESIGN
Format Reward Functions. form reasoning channel and a machine-parsable answer channel:
The policy generates a structured textual output y containing a free-
(think) (/think) (answer) JSON (/answer).
The answer channel must contain valid ISON array of objects. In stage- each object specifies a bounding box: { ("bbox_2d": [x1, , y x2, y2]]. In stage-2, each object is augmented with list of points: {"bbox.2d":
I...I, "points":
J.. We define binary [[x,]]
format reward, Rioe (y) E {0.1), 1}, which is 1 if and only if the output is syntactically correct and adheres to the stage-specific schema. the format reward is 0, the total reward for the episode is also O, overriding all other components.
Stage-1 (Localization) Reward. Given the ground-truth set of boxes B = {b3}/=1 and the predicted set B K we define:
Format reward f (y) as defined above.
Accuracy reward via Hungarian matching with an IoU threshold of 0.5. Let ToU b') be the standard box IoU, We form a binary match matrix M ([oUU(,, by) > 0.5), solve the linear assignment problem on the cost matrix M, and denote the number of matches as N The accuracy reward is
Nm max J)
R(1) (y; B+)=
E [0, 10.
Length reward that encourages predicting the correct number of instances:
RLd (): exp (- 2 K JI/J), J 0.
(8)
17

---

Published as a conference paper at ICLR 2026
Algorithm Two-stage end-to-end GRPO Training for SocioReasoner
Require: Training dataset Praain VLM policy Re; frozen SAM S; renderer D; reference policy
Trefi group size G; PPO clip KL weight 3; optimizer with learning rate 7, number of RL steps T
1: Initialize 30 <- 00
2: for step = 1 to 7 do
3: Sample mini-batch B C Draiin
4: for all (I,, Im. tb) do
5: X1 [I., Im, tb)
Stage-1: Localization
6:
for g = to do
Sample completion y (-(-1x)
7:
8:
Parse bounding boxes Bg9) from yi (assign syntax reward invalid)
M S(I,, prompt B(s))
9:
Compute 17(5)
11:
end for
b1 + G - R(
12:
Compute advantages 1 +- R((0) b for all
13:
Update policy wo with GRPO fx1, yl ( J=--= using clip and KL weight
Select +- arg maxy R( (or sample proportional to exp(R
15:
16:
I,r + D(I,, B(0"), MM
Im,r < -D((Im: B(42) M
X2 (Ie.r, Im.r, to ! (9
18:
Stage-2: Refinement
19:
for g = to do
Sample completion y2 (g) ~ x-)xxx
20:
Parse (B(s), p(o)} from (0) (assign syntax reward if invalid)
22:
( + S(I, prompt (B p(
Compute (2(5)
23:
24:
end for
by -- E R
25:
Compute advantages A
R( by for all g
26:
27:
Update policy my with GRPO on (2 A
1g=1. using clip and KL weight |
28:
end for
29:
To.id +
b Refresh behavior policy for next step
end for
The total stage-1 reward the unweighted sum of these components:
R = R(I) (y) R(1) (y: B²) + R (y: B*).
(9)
Stage-2 (Refinement) Reward. For each predicted group (one bbox plus its point list), we execute SAM with the prompts to obtain mask M, and compare it to the ground- truth mask M*
Format reward R
(y) defined above.
Accuracy reward as pixel loU:
ace xy = IOU(M, e [].
(10)
Length reward that encourages concise, informative interactions. For group with points, we define Gaussian-shaped score peaking at two points:
C'
(2) (y) = G
Σ (n)) E [0.
(11)
9=1
18

---

Published as a conference paper at ICLR 2026
Table 5: All ablation of multi-
Socio-name
Socio-class
docio-function
All dataset
Method
cloU
gloU
F1
cloU
gloU
F1
cloU
gloU
F1
cloU
gloU
F1
w/o reflection
48.5
50.9
58.4
44.4
49.3
55.5
36,3
41.8
45.0
44.0
48.5
54.3
w/o refinement
50.5
53.1
61.2
46.2
51.0
58.1
40.3
45.7
48.1
46.4
50.8
57.5
Ours
52.6
55.7
64.6
47.6
52.8
60.1
40.6
46.9
50.3
47.9
52.8
59.7
Table 6: All ablation of point number.
Socio-name
Socio-class
docio-function
All dataset
Method
cloU
gloU
F1
cloU
gloU
F1
cloU
gloU
F1
cloU
gloU
F1
point refinement
51.6
53.4
61.2
47.6
51.2
59.0
40.0
45.7
49.5
47.6
51.2
points refinement
52.6
55.7
64.6
47.6
52.8
60.1
40.6
46,9
50,3
47.9
52.8
points refinement
53.2
54.7
65.0
48.9
52.6
59.7
41.8
46.6
49.8
48.9
52.3
58.8
with μ = 2 and σ = 2. G' the number valid groups. This encourages using small number of informative points rather than many redundant ones.
where r(n) = exp
2n
The total stage-2 reward is the sum of these components:
xx = (2) (y) + R (y; + R22) (y).
(12)
A.2.3 EXPERIMENTAL SETTINGS
For all our Reinforcement Learning (RL) based models, namely VisionReasoner,, Seg-RI, SAM-RI, and RemoteReasoner, we adopt unified training configuration. We set the rollout batch size to 128 and the group size to 8. The models are optimized using the Adam optimizer with a learning rate of 1 10-4 6, For the Proximal Policy Optimization (PPO) algorithm, the clipping parameter is set to 0.5, and the Bullback-Leible (KL) divergence weight ,3 is configured to 0.005. All RL. models trained for 250 steps within the ROLL framework (Wang al., 2025).
key aspect of our methodology the handling of visual inputs. Since all RL-bbased methods built upon Qwen2.5-VL-3b. which natively supports multi-image inputs, we provide both the satellite imagery and digital maps as visual input. For the Supervised Fine-Tuning (SFT) version of our model, we construct the supervision signal using the bounding box of the ground-true mask, along with three points randomly sampled from within the mask's area.
In contrast, for the baseline models UNet, SegFormer. SegEarth-OV, RSRefSeg and SegEarth-R.. we followed the original authors' implementations. We utilized their publicly available source code and pre-trained models, which are then fine-tuned on the SocioSeg dataset. As these architectures do not support multi-image inputs, only the satellite imagery is used as the visual input for these models. All models are trained on a high- high-pperformance computing cluster equipped with 16 NVIDIA H20 GPUs.
A.3 USER PROMPT TEMPLATE
The user prompt templates utilized in our experiments are shown in Figure 9. SocioReasoner em- ploys a two-stage reasoning process; consequently, we designed two distinct prompt templates to accommodate the different input and output formats of each stage. For our baseline model without the reflection mechanism, as well as the GPT and Qwen models, we adopt a single-stage prompt template. This template is adapted from the one used by VisionReasonerr, with modifications to meet the specific requirements of our task. For our SFT model, we use this same base template but remove the chain-of thouhht components. For all other RL-based comparative methods, used the original prompt templates provided by their respective authors, prepending each with the instruction, "You will be given images. The first is map and the second is corresponding satellite image."
19

---

Published as a conference paper at ICLR 2026
Stage-1 1
User Prompt
"You be given two images, The
map and the second corresponding satellite image."
"Please find 'Query]' with bboxs."
"Compare differencc between object(s) and find
closely matched object(s)."
"Ourput the thinking process in <thiks <hthink> and final unswer Samswer </answer" tags.
"Please use Fnnglish. Output the bhox(es) in ISON format."
"ie. <think>thinking process here <ithink>"
"canswerr ([r [["bbox_ee 2d": boxl.byl..x22.y222 ("bbox_2d") (bx3.by..bxx.by4] )
Jansswrr""
Stage-2 User Prompt
"You will be given two images. The first a map and the sccond is a corresponding satellite image."
"Now some bbox(s) and the results after SAM segmentation for 'QQuery] have been rendered on images."
"The found bbox(s) are: (Bhoxs))"
"Please add some points appropriately cach bbox to better represent area interest." "
"Output thinking process <thinks
tags."
"ic. <thinke> thinking process here </think>"
"aanwerr [[" ("bbox 2d" b..bbyyyyb22) "points": [ 2ppp:lyyyyp222222pppppp... 3
Single-stage User Prompt
"You will he given images, The first map and the second corresponding sarellite image." "Please find 'QQeery' with bboxs and some points appropriately to each bbox to better represent the area of interest.
closely matched object(s))" "Output thinking process in <think> </think>> final answer sauswer> </answor> tags.
"Compare the difference between object(s) and find
"Please use English. Output the bbox(es) in JSON format."
'ie. -think-thinking process <thhnnk>"
"canswerr ( ("bbox_ 2d": b.x1.,yl.xx2.bb211 "points": Jispslgg.1pss222222ssssssss Vanswerr"
Figure 9: The two prompts above are the user prompt template for SocioReasoner, which adopts two-stage reasoning process to mimic human annotation. The prompt below is the single-stage prompt used for the baseline without reflection and zero-shot GPT and Qwen models.
Table 7: Comparison with state-of- aff-the-art methods on the SocioSeg OOD (New Region) dataset.
OOD (New Region) Dataset
Method
cloU
gloU
UNet
10.0
7.3
7.1
Segformer
18.8
14.7
14.1
ResionRe
32.8
34.4
35.0
Seg-RI
28.8
28.8
26.2
SAM-RI
14.8
14.5
19.0
SogEarth-OV
3.2
3.2
0.0
RSRefSeg
12.4
10.3
13.8
SegEarth-R1
20.7
18.7
21.2
RemoteReasoner
27.5
29.8
28.1
Ours
40.2
43.4
42,9
A.4 SOCIOSEG BENCHMARK
GPT-5, GPT-o3 and Qwen2.5- 20ven2.5-VLL72h VL--2b are evaluated as baselines without any fine-tuning, As shown Table 9, their performance is substantially lower than that of our trained model, indicating that even large-scale VLMs struggle with the complexities of socio-semantic segmentation without task- specific training. Notably, Dwwnn25-VL-33 fails to produce valid bounding box outputs in our ex- periments, resulting in performance. This underscores the importance of specialized training and the effectiveness of our reinforcement learning approach in eliciting the reasoning capabilities necessary for this task.
20

---

Published as a conference paper at ICLR 2026
Table 8: Inference time comparison (seconds per sample).
VisionReasoner
SegR1
SAMRI
RSRefSeg
SegEarth-R1
RemoteReasoner
Ours-rl
Ours-sft
1.33
1.07
2.52
0.16
0.35
1.13
2.71
0.41
Table 9: SocioSeg benchmark.
Socio-name
Socio-class
Socio-function
All dataset
Method
Backbone
cloU
gloU
cloU
gloU
cloU
gloU
cloU
gloU
GPT-5
Not disclosed
16.1
16.1
14.9
15.1
12.2
12.5
14.7
15.0
GPT-o3
Not disclosed
22.6
22.9
20.9
22.7
16.1
17.3
20.3
21.7
Wven2.5-VL-30
Qwen2.5- 3b
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Qwen2.5-VL-72
Qwen2.5 -VV-22b
27.1
29.5
21.8
27.2
20.4
24.4
23.1
27.5
A.5
MORE QUANTITATIVE RESULTS
A.5.1 MORE ABLATION STUDIES AND INFERENCE TIME COMPARISON
We provide the comprehensive quantitative results of our ablation studies in Table 5 and Table 6, presenting detailed metries across all three hierarchical task levels. Furthermore, Table reports the performance of all comparative methods on the OOD (New Region) dataset, demonstrating that method consistently achieves the best results across all evaluation metries. Finally, present a computational efficiency analysis in Table 8, comparing the average inference time per sample (in seconds) against the baselines. While our model incurs a higher latency due to its iterative two-stage reasoning mechanism, this computational trade-offis is justified by its superior segmentation accuracy.
A.6 MORE VISUALIZATIONS
We first present the trend of the reward function during the training process, as shown in Figure 10. As can be seen, the reward function gradually converges training progresses, indicating that the model continuously improves its decision-man makiin quality. Next, we provide additional qualitative results comparing SocioReasoner's performance on the three hierarchical tasks, as illustrated in Figure 11. These examples clearly demonstrate the advantages of SocioReasoner's performance across different tasks, especially its accuracy and robustness in complex scenarios. Furthermore, we showcase more inference examples from SocioReasoner in Figure 13. These examples further validate SocioReasoner's capability in processing multi-modal inputs.
A.7 FAILURE CASES
Finally, we present visualization of representative failure cases encountered by SocioReasoner in Figure 12. These instances, all characterized by a Generalized IoU (gloU) below 0.1, primarily stem from two distinct eиrог modes: (i) Localization Failure (Cases 1 and 2), where the predicted bounding box fails to locate the target region entirely, likely due to the model being distracted by the dense and visually cluttered urban environment. In such cases, the high density and low resolution of the Points of Interest rendered on the map make it difficult for the VLM to discern the correct lo- cations of the semantic entities; and (ii) Boundary Imprecision (Cases 3, 4, and 5), where the model correctly identifies the general location of the semantic entity but fails to generate a geometrically accurate enclosure. This issue can be attributed the complexity of geographical features in satellite imagery, where the boundaries of buildings, roads, and other urban elements are often ambiguous and irregular, making it difficult for the VLM to infer precise bounding boxes. Furthermore, certain semantic categories inherently possess highly abstract definitions, whose physical boundaries may not be clear-cut or fixed, which further compounds the difficulty of accurate localization. These challenges underscore the inherent difficulty of mapping abstract social concepts to precise physical coordinates in complex satellite imagery. This suggests that while our reasoning framework effec- tively bridges the modal gap, future research should further focus enhancing the fine- grained spatial reasoning capabilities of Vision Language Models (VLMs) and mitigating spatial ambiguity to achieve more robust performance in dense urban scenarios.
21

---

Published as a conference paper at ICLR 2026
Table 10: Multi-class semantic segmentation setting.
Socio-class
focio-function
Method
cloU
gloU
F1
cloU
gloU
F1
UNet
0.1
0.1
0.7
0.4
SegFormer
8.4
9.5
15,5
4.2
4.2
8.0
A.8 APPLICATION
In the industrial sector, the practical value of our method lies in its ability to infer the corresponding spatial extent, specifically the Area of Interest (AOI), from given Point of Interest (POI). This provides essential visualization data for mapping applications and is of significant importance for coordinate-basee recommendation systems. For instance, deriving precise AOIs enables superior Location-Baze Services (LBS) (Huang et al., 2018) recommendations, such as for dining, retail, and entertainment venues, thereby enhancing overall accurate AOIs allow mapping applications to optimize route planning and navigation, providing more reliable route suggestions and spatial context, especially within complex urban environments.
experience and satisfaction. Furthermore,
In academic contexts, particularly for paradigms like the 15-minute city, precise AOIs assist urban planners and policymakers comprehensively understanding urban spatial structures and functional distributions, which in turn optimizes public resource allocation and elevates the quality of urban life. For example, current 15-minute city frameworks (Bruno et al., 2024) are predominantly cal- culated using POIs, whose spatial boundaries are fundamentally ambiguous. By acquiring precise AOIs, it becomes possible to rigorously evaluate actual spatial accessibility and service coverage, thereby providing more scientific foundation for data-driven urban planning.
A.9 LIMITATIONS AND FUTURE WORK
While the SocioSeg dataset provides robust benchmark for urban socio-semantic segmentation, we acknowledge a limitation regarding the handling of single satellite image may contain multiple instances of the same Socio-class or Socio-function (e.g., multiple office buildings). In our dataset, the ratio of single-instance to multi-instance samples is 0.89:0.1 11, with an average of 1.17 instances per image.
targets. In real-world scenarios,
During our experiments, we observed that SocioReasoner, along with other VLM+SAM based base- lines (such VisionReasoner and RemoteReasonent tends to converge toward identifying and seg- menting single dominant instance. This behavior reflects a known bottleneck in the spatial reason- ing and multi-target localization capabilities of current Vision-Language Models. It is important to note, however, that our comparative evaluations remain strictly fair, as all methods were trained and evaluated under identical conditions and constraints this dataset.
In future work, we aim to address this single-instance bias. Recent studics suggest that scaling up the parameter size of the base VLM (e.g.. 7B parameters or larger) significantly enhance multi-instance perception and fine- ane-grained spatial reasoning. Integrating larger models or develop- ing specialized multi-instance generation constraints within the reinforcement learning reward func- tion represents a promising direction to further improve socio-semantic segmentation performance in dense urban environments.
22

---

Published as a conference paper at ICLR 2026
Stago-1 Reward
Length Reward
Stage= Accuracy Roward
1.0
0.8
0.6
0.4-
0.2
150
200
0.0
50
100
150
200
1000
170
200
Suup
Stage-2 Format Reward
Stuge-2 Length Reward
Stage-2 Accuracy Reward
1.0
1.
Ro000
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.2
0.0
100
0.8
35
100
Supp
Step
Stpp
Figure 10: The rewards visualization during the training process.
Sall Par
1he Optone
Hary Endlin Neate
"Agurrsa
R5RaSox
Soglinth-k
Figure 11: All method Comparisons of SocioReasoner across the three hierarchical tasks.
23

---

Published as a conference paper at ICLR 2026
Digital Map Label
Satellite Image with Label
SocioR SocioRcasoner Result
Case
Case 2
Case
Case
Case 5
Figure 12: Failure cases of SocioReasoner.
Oan Suger
Abvve
"gai rage Donu braak ou
veree Thoon fwrm Clidingg herserg
dioine
Riex Sears
Leaiis dooning Lendoe sars buldega
ohaae Syrunigs corten ooou rukked
The cieely mnsted dlees san
Query
Ihow Jn
seg
Quary
pM on eag)
creak Fear
name. adde
park of sage lighightt
Mow
creak
Dightl map
"acka de
Iod tracuan arge
sekving
Topeoo Daliey
obeak The imnge shown deaded mis with vemmi bertin
weldeg inuctite charly
maryy culligg acresss
pendie
(Pho sot sargt
Day
monnd of leng" parn
Mko help to betar spreart ina maserily plighnng te
Xij "ueral (Hics Abl -eTE
Haly
vihiod regiico docies deivery conttots
Dupanina
Digisiing
SMj EDCA 441 elzgo
tolk roslery Door sanus locders
Puries
modth aegg proder tomatie perpoeiie of
uee envstess catthe des you progecine
andanccinn vikueir
Queyy
Query
dheak wating hew plbnad voadeees eeeeee pMe
440 Bay une hetres Thes pulss adli ia leauum
Digitl map
[Thiec pao neg "pdte"" 2a14,
Dightl
(TTX eurge
Denc
porade gaatrs corningoo uniocd
Tirk
saben
rodicces
lacae wader boundain of auut des repaas
(Pbbe (IN 3t 2010)
cas
(Sl) dany
pore
4n) Bloued conndran (UA
(i 20, sai yatral 1315 (k1) (60 JAL
Digitll
Figure 13: More inference examples of SeaioReasoner.
24


Reviews:
[{"review_id": "lXB8isPrFs", "reviewer": "Reviewer_uPcU", "summary": "This paper studies urban socio-semantic segmentation from satellite imagery. The authors contribute a benchmark dataset, SocioSeg, integrating satellite imagery with digital maps, and a SocioReasoner framework that uses vision-language models with a multi-stage human-inspired reasoning process optimized via reinforcement learning. Experiments including ablation studies and zero-shot tests show improvements over existing methods.", "questions": "Are the comparisons between fine-tuned and zero-shot baselines fair? How do you ensure that these comparisons accurately reflect the model's true performance without inflating the reported gains?\n\nHow well does the proposed method generalize to new social semantic categories or to other cities? Does it require substantial manual tuning to adapt to these new scenarios?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The SocioSeg dataset uses a hierarchical structure from Socio-name to Socio-function and unifies diverse geospatial data into a single map, enabling easier multi-modal reasoning. \n\n2. The SocioReasoner framework simulates human-like annotation with sequential localization and refinement and integrates vision-language models with SAM under reinforcement learning.", "weaknesses": "1. The main contribution of this paper lies in the introduction of a new dataset, with the primary performance improvements stemming from the dataset and reinforcement learning. Overall, the innovation appears to be limited.\n\n2. The evaluation lacks per-category breakdown and qualitative error analysis so it is unclear which entities are actually improved. \n\n3. The paper shows quantitative improvements but lacks qualitative failure analysis or discussion of where the model fails.\n\n4. Stronger ablation comparing RL vs. supervised fine-tuning could clarify necessity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M9Jo7yzk0z", "reviewer": "Reviewer_5c3k", "summary": "The paper introduces SocioSeg, a new dataset for what the authors call urban socio-semantic segmentation: the task of identifying _socially_ defined entities (i.e. schools, parks, and hospitals) from satellite imagery. The authors differentiate this new taks from traditional segmentation tasks that focus on physical or visually distinct features (e.g., buildings or water bodies), this work addresses the challenge of detecting socially meaningful categories that require contextual and semantic reasoning.\n\nThe paper also introduces SocioReasoner, a vision-language reasoning framework that intends to mimic human annotation behavior through cross-modal understanding and multi-stage reasoning. The framework leverages reinforcement learning to handle the non-differentiable aspects of this reasoning process, enabling the model to refine its interpretation of social semantics based on feedback.\n\nExperiments show that SocioReasoner outperforms all compared models and demonstrates robustness when applied to images from another domain. Despite these good results, the paper still has some important flaws (which I will detail later), which make me inclined to reject the paper as it is.", "questions": "- What is the difference between the proposed socio-semantic segmentation task and land use segmentation?\n- How do you justify the use of vision-language models (VLMs) for this setup? What motivates their suitability for detecting socially defined entities?\n- How were the textual prompts for the VLM chosen or constructed? Were they manually designed, or automatically derived from the dataset labels?\n- The paper emphasizes that the proposed framework mimics human reasoning. Could you clarify what aspects of the approach correspond to reasoning, and provide evidence or examples that support this claim?\n\n- It is stated that an advantage of the proposed dataset is that raw geospatial data is unified as a digital map layer. How exactly is this layer obtained, and in what way does it overcome the limitations of existing data sources?\n- Is training on the SocioSeg dataset performed in a supervised or self-supervised manner? What are the labels or objectives used during training?\n- Could you include some representative examples or visualizations of the final dataset to illustrate the types of entities and annotations it contains?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Novel dataset. The paper introduces SocioSeg, a new dataset for urban socio-semantic segmentation, focusing on identifying socially defined categories (e.g., schools, parks, hospitals) from satellite imagery.\n\n2. Methodological proposal. The proposed SocioReasoner framework combines vision-language reasoning and reinforcement learning to tackle the proposed semantic segmentation task. The approach is conceptually interesting and aligns with current research trends in multimodal and reasoning-based vision models.\n\n3. Overall well-written and clear. The paper is well-organized and clearly written. All sections are easy to follow.", "weaknesses": "1. **Lack of justification for the new task.** While the paper motivates the challenge of detecting socially meaningful categories, it does not convincingly articulate the concrete real-world relevance or practical need for socio-semantic segmentation. The connection to downstream applications (e.g., urban analytics, policy planning, or social impact studies) could be made more explicit. Moreover, it is not clear how this socio-semantic classes are defined and they seem quite arbitrary. Is there any definition for socio-semantic classes? What is the difference with land use segmentation?\n\n2. **Insufficient motivation for using VLMs.** The use of vision-language models is not well justified. The paper assumes that VLMs are naturally suited for reasoning about social semantics, but provides little evidence to support this claim [1], while literature tends to say the opposite. Current VLMs tend to excel at visual-text association rather than genuine reasoning, and the paper does not clearly demonstrate that SocioReasoner achieves the latter.\n\n[1] Huang, C., Zhu, Y., Zhu, S., Xiao, J., Andrade, M., Chopra, S., & Kira, Z. (2025). Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models. arXiv preprint arXiv:2506.07936.\n\n3. **Missing essential baselines.** The experiments would be stronger with comparisons to simple supervised segmentation baselines, such as DeepLabv3+, Segformer or U-Net trained directly on SocioSeg. Without these, it is difficult to isolate the benefits of the reasoning framework from other factors like model capacity or data scale. \n\n4. **Limited empirical analysis.** The evaluation focuses mainly on performance metrics, with limited ablation or qualitative analysis. It would be helpful to see results that explicitly test reasoning ability or analyze the contribution of individual components (e.g., reinforcement learning, multi-stage reasoning, or language inputs). Is it really necessary to integrate a language prompt?\n\n5. **Ambiguous claim of reasoning.** The paper frequently refers to “reasoning,” but the evidence for such capability is indirect. Additional experiments, examples or interpretability analyses would be necessary to substantiate this claim.\n\n6. **Unclear dataset creation and task formulation.** The process of constructing the SocioSeg dataset is not described in sufficient detail. It is unclear how the social semantic labels were obtained, validated, or aligned with the satellite imagery and digital maps. The paper mentions pixel-level labels and hierarchical structures but does not specify whether these were manually annotated, derived from external GIS sources, or generated automatically. Important dataset details are missing, such as the number of images, the number of annotated objects, number of classes, etc.\nSimilarly, the precise inputs and outputs of the model remain ambiguous. It is not entirely clear whether the model takes only raw satellite imagery, or also uses auxiliary map data or textual metadata during training and inference. This lack of clarity makes it difficult to fully understand what the task setup is, how supervision is provided, and how reproducible the dataset and results would be.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qihsPgXIWO", "reviewer": "Reviewer_5GRV", "summary": "This submission presents work at the intersection of (socio-) semantic segmentation and vision-language models for earth observation and remote sensing. It introduces \"SocioSeg\", a new dataset and \"SocioReasoner\", a vision-language reasoning framework for that task. To accomplish this task, the authors propose adapting a given vision-language model using reinforcement learning. In their experiments, the authors demonstrate fair performance increases as compared to baselines.\n\nAlthough I very much like the idea, I have concerns about the contribution of this work with respect to (i) the methodological novelty of the presented approach, (ii) the mismatch and relevance for the ICLR community, (iii) the weak experimental evaluation.\n\nI outline my concerns in more detail below.", "questions": "- **(Q1)**: Is the reference to \"RSRefSeg (Mall et al., 2024)\" the right one, or is this a typo or a wrong citation?\n- **(Q2)**: Since the results of VisionReasoner are the second-best ones, could you outline the methodological differences between your approach and VisionReasoner (I was not able to find any of the SOTA models you compared against in the related work section of your submission, that's the reason I am asking)?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- **(S1)**: this paper focuses on an interesting interaction of vision-language/reasoning and remote sensing/earth observation.\n\n- **(S2)**: this paper is easy to read and follow, given the systematic build-up of the paper.\n\n- **(S3)**: I appreciate the author's contribution in curating and annotating the earth observation and remote sensing dataset.", "weaknesses": "- **(W1)**: Methodological novelty of the presented approach: this work combines several components of previously published work to address a new task for a specific domain (being remote sensing and earth observation). However,  I see the introduced dataset being a contribution, which doesn't change the methodological novelty of the presented approach.\n\n- **(W2)**: Mismatch and relevance to the ICLR community: I think this work might be more suitable for publication at a computer vision conference or a remote sensing/earth observation journal, given its very domain-specific domain.\n\n- **(W3)**: Weak Evaluation: I acknowledge the author's evaluation of their approach against other work. Unfortunately, all SOTA approaches mentioned in this submission - except one - are coming from non-published arXiv preprints. Only \"RSRefSeg (Mall et al., 2024)\", has been published (at ICLR 2024). However, having read the paper by Mall et al, I was not able to find anything about \"RSRefSeg\". The approach proposed by Mall et al., 2024 is a remote sensing vision-language model referred to as \"GRAFT\" in the original paper and not \"RSRefSeg\". This leaves me very confused, since I do not know if this is a typo or if the authors speak about another paper and the citation is wrong.\n\nIndependent of this, I would suggest evaluating the proposed approach on another dataset, such as the ones used in Mall et al., 2024. This work evaluated model performance on EuroSAT and BigEarthNet. This way, one would have comparable results of the approach, modulo the proposed dataset being specific (I fully understand that this submission focuses on socio-segmentation)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JUWo7G114p", "reviewer": "Reviewer_icS2", "summary": "The paper introduces a new task: “urban socio-semantic segmentation,” which aims to segment socially defined entities in cities — things like parks, schools, logistics centers, “business office buildings,” “park and greenspace,” etc. These are different from classic land-cover / land-use segmentation targets (roads, buildings, water) because they’re not purely visual; they’re defined by function and human use.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**SocioSeg dataset:**\n\nMulti-level hierarchy (name / class / function) is well thought out. It naturally scales difficulty from “find this exact named facility” to “find anything that is educational.”\n\n**Strong empirical results:**\n\nSocioReasoner > strong baselines (VisionReasoner, RemoteReasoner, Seg-R1, etc.) across all subtasks, ~+4 gIoU absolute vs best baseline on average.\nCareful baselines: they retrain baselines on SocioSeg where possible, and explain exceptions (e.g., RSRefSeg can’t take 2-image input, so only satellite is given).", "weaknesses": "- Ground truth and annotation quality/reproducibility of labels\n\nThe socio-semantic “truth” is derived from Amap AOI data. The paper says they rasterized AOIs, QA’d and dropped bad samples, etc.\nBut: we don’t yet see quantitative inter-annotator agreement or error estimation. Are AOI polygons always aligned with functional reality? \n\n- Geographic diversity\n\nAll training data is from Amap within China.\nYes, they test on Google Maps tiles (style/domain shift), but it’s still presumably Chinese geography. We don’t know if the ontology or appearance generalizes to cities in, say, Europe, Africa, or North America where zoning, building morphology, and POI taxonomies differ. \n\n- Dependence on commercial basemaps / terms of service\n\nThe method leans on digital map rasters rendered from Amap (or Google Maps at test time). The paper frames this as “we solve multi-modal fusion by turning everything into an image.”\nBut practically, this offloads data scarcity and licensing problems onto whoever deploys the model: you still need a high-quality, up-to-date basemap layer with POIs, which in some regions is proprietary, restricted, paywalled, or censored. This could limit “real-world” access.\n\n- Evaluation scope\n\nMetrics are cIoU / gIoU only. It would help to also see precision/recall on instance counts for socio-name tasks (did it find the right named facility and only that facility?) because over-segmentation vs under-segmentation communicates different failure modes (especially important for urban planning use).\n\nRuntime overhead: they admit SocioReasoner inference is ~2.7s/sample, which is slower than e.g. RSRefSeg at 0.16s.\nThis is fine for planning, but maybe not for city-scale tiling at high throughput.\n\n- Limited OOD (out-of-distribution) evaluation\n\nThe paper’s out-of-distribution (OOD) evaluation only tests different map styles (Amap → Google Maps), which mainly reflects visual domain shift, not true reasoning generalization.\nIt doesn’t test cross-city, cross-year, or cross-socio-semantic generalization — scenarios that would actually demonstrate whether the RL-tuned model exhibits emergent reasoning capabilities.\nWithout such broader OOD studies, it’s unclear whether the reinforcement learning contributes genuine reasoning emergence or simply improves style robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FSD5xqPDDr", "reviewer": "Reviewer_Rsq7", "summary": "This paper introduces a new task called \"socio-semantic segmentation\", identifying socially defined urban areas (like schools, parks, or residential districts) from satellite imagery, which is challenging because their boundaries are based on function, not just visual appearance. \n\nThe authors introduce a new dataset, SocioSeg, which pairs satellite imagery with corresponding digital map layers and pixel-level labels for social entities, organized in a hierarchy of increasing complexity. (contribution) \n\nThe authors propose a novel framework, SocioReasoner, which uses a Vision-Language Model (VLM) to reason like a human annotator. It first localizes a target area using both satellite and map images, then iteratively refines the segmentation mask. Since this process isn't differentiable, they use Reinforcement Learning to train the model end-to-end.\n\nExperiments show that SocioReasoner outperforms existing state-of-the-art models and demonstrates strong zero-shot generalization.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "**The task is interesting**:  Identifies and formalizes a significant, underexplored challenge in geospatial analysis and provides a dedicated dataset (SocioSeg) to foster research.\n\n**Innovative Methodology**: The multi-stage, reasoning-based approach (SocioReasoner) effectively mimics human annotation logic. The use of Reinforcement Learning to optimize this non-differentiable pipeline is a clever and practical solution.\n\n**Solid Empirical Results**: The framework not only achieves superior performance on the new benchmark, highlighting its robustness and potential for real-world application. \n\n**Writing**: The writing is clear and easy to follow (presentation).", "weaknesses": "The current experimental results are not sufficiently rich. More experiments verified on other datasets wil strengthen the contribution. \n\nThe space utilization of figures can be improved. For example, there is too much space in figure 2 and the elements are sparse, which is not professional. \n\nI will consider to increase the score if the authors: \n- clarify how this work concretely differs from or improves upon prior research.\n- verify on broader related benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)