Alice Benchmarks: Connecting Real World Re-Identification with the Synthetic
Authors: Xiaoxiao Sun
Yue Yao
Shengjin Wang
Hongdong Li
Liang Zheng
Venue: ICLR 2024 poster (2024)
Keywords: Synthetic Data
Object re-ID
Benchmarks
Primary area: datasets and benchmarks
TL;DR: This paper introduces Alice benchmarks for "synthetic to real" domain adaptive object re-identification.

Abstract: For object re-identification (re-ID), learning from synthetic data has become a promising strategy to cheaply acquire large-scale annotated datasets and effective models, with few privacy concerns. Many interesting research problems arise from this strategy, e.g., how to reduce the domain gap between synthetic source and real-world target. To facilitate developing more new approaches in learning from synthetic data, we introduce the Alice benchmarks, large-scale datasets providing benchmarks as well as evaluation protocols to the research community. Within the Alice benchmarks, two object re-ID tasks are offered: person and vehicle re-ID. We collected and annotated two challenging real-world target datasets: AlicePerson and AliceVehicle, captured under various illuminations, image resolutions, etc. As an important feature of our real target, the clusterability of its training set is not manually guaranteed to make it closer to a real domain adaptation test scenario. Correspondingly, we reuse existing PersonX and VehicleX as synthetic source domains. The primary goal is to train models from synthetic data that can work effectively in the real world. In this paper, we detail the settings of Alice benchmarks, provide an analysis of existing commonly-used domain adaptation methods, and discuss some interesting future directions. An online server has been set up for the community to evaluate methods conveniently and fairly. Datasets and the online server details are available at https://sites.google.com/view/alice-benchmarks.

Published as a conference paper at ICLR 2024
ALICE BENCHMARKS: CONNECTING REAL WORLD RE- IDENTIFICATION WITH THE S SYNTHETIC
Xiaoxiao Sun', Yue Shengjin Wang', Hongdong Li', Liang Zheng!
The Australian National University 2 Tsinghua University
(first-noe -aame. last -name) (anu. edu. au! wgsgjetsiingha..etu.oo
ABSTRACT
For object re-identification (re-ID), learning from synthetic data has become promising strategy to cheaply acquire large-scale annotated datasets and effective models, with few privacy concerns. Many interesting research problems arise from this strategy, egg, how to reduce the domain gap between synthetic source and real-world target. To facilitate developing more new approaches in learning from synthetic data, we introduce the Alice benchmarks, large-scale datasets providing benchmarks as well evaluation protocols to the research community. Within the Alice benchmarks, two re-Il) tasks are offered: person and vehicle re-1l). We collected and annotated two challenging real-world target datasets: AlicePerson and
captured under various illuminations, image resolutions, etc. As an important feature of our real target, the clusterability of its training set is not manually guaranteed to make it closer the real domain adaptation test scenario. Correspondingly, we reuse existing PersonX and VehicleX as synthetic source domains. The primary goal to train models from synthetic data that work effectively in the real world. In this paper, we detail the settings of Alice benchmarks, provide an analysis of existing commonly-used domain adaptation methods, and discuss some interesting future directions. An online server" has been set up for the community evaluate methods conveniently and fairly.
INTRODUCTION
Synthetic visual data provide an inexpensive and efficient way to obtain a large quantity of annotated data for facilitating machine learning tasks. It allows for both foreground objects and background context to be edited conveniently order to generate diverse image styles and contents. Thanks to these benefits, synthetic data have been increasingly widely used in computer vision research and applications to improve the data acquisition process (Bak et al. 2018 Fabbri et al. 2018 Gaidon cases of synthetic data include scene semantic segmentation (Hu et al.
al. 2016). Successful
2019 Ros et al.] 2016: Lin et al. 2020 Xue et al.) (021). object detection (Gaidon et al. (016))
tracking (Fabbri al. 2018), re-11) (Sun & Zheng. 2019 Yao et al. 2020 Wang al. 2020 2022
Li al. Zhang et al. 2021 Wang et al. 2022), pose estimation (Chen et al. 2016). self-driving
vehicles (Dosovitskiy et al. 2017) and other computer vision tasks (Raistrick al. (023b).
In this paper, we introduce a series of datasets, "Alice benchmarks" (hereafter "Alice" consisting of a family of synthetic and real-world databases which facilitate research in synthetic data with applications to real-world object re-ID, including person and vehicle re-ID tasks. There are two advantages of using Alice henchmarks. Firstly, humans and vehicles are the study objects of the two tasks and will cause privacy concerns without proper de-identification (like blurring face or license plate). Learning from synthetic data offers an effective solution to this problem by decreasing the need for real-world data. Secondly, it is expensive to rely on human labor to accurately annotate person/vehicle interest across cameras. In comparison, using synthetic data allows us to cheaply obtain large amounts of accurately annotated images.
detaill can he found at htttss //ss/iiiae google e, www.ciooovii..oo ew/al i ce-bbenchaa- "The name "Alice" is inspired by the notion that the model learns from synthetic environment, similar to how Alice explores the virtual world in "Alice in Wonderland".

---

Published as a conference paper at ICLR 2024
synthetic data
real-world data
synthetic data
real-world data
(A) person re-ID
(B) vehicle re-ID
The current version of Alice benchmarks supports two research tasks: person and vehicle re-ID. The source domains are synthetic images from the PersonX and VehicleX datasets. The data of the target domains are real-world images we collected from varying conditions.
Figure 1:
Specifically, Alice focuses on domain adaptation (DA) under the setting of "synthetic to real" (Syn2Real). We aim for re-ID models trained on the synthetic data to generalize well to real data. For the source domain, we reuse two existing synthetic datasets, PersonX (Sun & Zheng 2019) and VehicleX (Yao et al., 2020). Specifically, the two synthetic datasets are fully editable. We combine the two engines and provide an editing interface, whereas researchers can generate their own source dataset with an arbitrary number of images under manually controllable environments. For the target domain, we introduce two new real-world datasets, AlicePerson and Alice They are collected under diverse environmental conditions including resolution, illumination, scene type etc. Fig. shows sample images. All images of those two datasets and annotations of the training sets have been released to the community
[ Labels of the test datasets are not publicly available, and only limited online tests will be allowed in a certain period. We hope these hidden testing sets can offer a unified, fair, and challenging benchmark for Syn2Real DA research object re-ID.
Using Alice, existing DA methods can be tested, including those using style-level alignment (Deng et al. 20), feature-level alignment (Long et al. 2015) and pseudo-label based methods (Zhong et al. 2020 Zheng et al.) | He et al. 2022). In this regard, our benchmarks offer a new platform for re-ID research. Moreover, an important advantage of Alice is that the obtained synthetic data are fully editable. We provide easy-to-use editing tools and user interfaces for exploring different aspects of the data, such as lighting conditions and viewpoints Some recent methods (ao et al. 2020 2022 Kar et al.) (019) automatically adjust the content of synthetic data, aiming reduce the gap between the synthetic and real data. In this paper, we refer to this type of method as content-level DA.
Another key feature of the Alice benchmarks is the unassured clusterability of the target training sets. Existing domain adaptation benchmarks for re-ID typically possess a target training set with strong clusterability, i.e., an individual ID often appears in multiple cameras from various viewpoints within the training set. For instance, when adapting the domain for person re-ID, the common practice is to employ the training sets of Market-1501 (Zheng et al. 2015 or MSMTI7 (Wei et al. 2018). As these training sets were originally designed for effective supervised training, they typically guarantee a single ID across multiple cameras (i.e., views), thus enabling robust clusterability. This characteristic facilitates pseudo-label based DA methods in achieving high scores with relative ease. However, such strong clusterability should be considered dataset bias, as does not often occur in real-world settings. In this study, we propose the use of AlicePerson and AliceVehick Vehicle datasets, which lack such intense clusterability. As will be demonstrated in our experiment, the weak clusterability leads to a relatively smaller accuracy improvement when pseudo-label based DA methods are employed.
To provide a baseline analysis, we evaluate some common DA methods at the content level Yao ee al. 2020) 2022). pixel level (Deng et al.) 2018a), feature level (Vu al.) 2019 Tsai et al.) 2018). and based on pseudo labels (Fan et al. 2018 81 Zhong et al., 2019 2020). We report their accuracy not only on the Alice target datasets, but also on some existing weal-world datasets, such Market- 1501 (Zheng et al. 2015) and VeRi-776 (Liu et al. 2016). With extensive experiments, we identify a number of interesting findings, such as that current state-of-the-ard pseudo-label based methods are less effective on our benchmarks when the clusterability of the target domain is not guaranteed.
In addition datasets and benchmarking, Alice provides an online evaluation platform for the community. An executable version of the system will be set up to accept submissions. Currently, two aforementioned re-ID tasks are available. In the future, we will continue building up new
privacy information: manually blur the human faces and license plates when applicable. Alice is distributed under license CC BY-NC.0 which restricts its use for nom-emmmercial purposes.
2

---

Published as a conference paper at ICLR 2024
training/testing settings for the two existing tasks and add new learning tasks. As far as we know, our server will be the first online evaluation server in the re-ID community. In summary, Alice will benefit the community in the following ways: 1) exploring the feasibility of reducing system reliance on real-world data, alleviating privacy concerns; 2) facilitating the study and comparison of DA algorithms by providing challenging datasets and unifying evaluation platform; 3) sustaining long-term "service" for more tasks. In addition, we will discuss some potential research problems and directions made available by Alice. For example, how does "synthetic to real" compare with "real to real" regarding the domain gap problem?
This paper presents the following contributions to the computer vision community.
We provide the Alice benchmarks, an online service for evaluating domain adaptation algorithms of object re-ID that perform learning from synthetic data and testing on real- world data. Two challenging real-world datasets have been newly collected as target sets for person and vehicle re- identification.
We provide preliminary evaluations of some existing common domain adaptation methods our two tasks. Results and analysis provide insights into the characteristics of synthetic and real-world data.
We discuss some new research problems that are made possible by Alice. We also raise questions on how to better understand the Syn2Real domain adaptation problem.
2 RELATED WORK
Synthetic Data. We have seen many successful cases using synthetic data in real-world prob- lems, such as image classification (Peng et al. 2017 Yao et al.) 2022), scene semantic segmenta- tion (Sankaranarayana
et al. 2018 Xue et al. 2021). I, object tracking (Gaidon al. 2016 Yao et al.,
2023b; Liu et al. (002)). traffic vision research Geiger et al. 2012
Li et al. 2018), object re- (Sun & Zheng, 2019| Yao et al. 2020; Zhang et al., 202 Wang et al. 2022) and other computer vision
tasks (Raistrick et al. (023b). For example, the GTAS (Richter et 2016)) dataset, extracted from the game with the same name, is well-known for domain adaptive semantic segmentation. Together with datasets like Virtual KITTI (Gaidon et al., 2016) and synthetic2real (Peng al. 2018), they facilitate the research in style and feature-level DA. Wang et al. (2022) build the ClonedPerson dataset by cloning the whole outfits from real-world person images virtual 3D characters. We continue this area of research by introducing Alice, aiming to further explore the potential of synthetic data.
Domain Adaptation. Domain adaptation is long-sttndinn problem. and many attempts have been made to understand the domain gap (Torralba & Efros 2011: Perronnin et al. (010). Recently, efforts have been made to reduce the impact of domain gap (Saenko al., [010 Deng al. 2018a
Lou et al., 2019 Yao et al. 2023a). Common strategies contain feature- Long et al. 2015).
pixel- (Zhu al. 2017 Deng et al. 2018b) and pseudo-label based (Fan al. 2018 Zhong
et al. 2019 Song et al.) 2020b; Zheng et al. 202T: He et al. 2022) domain adaptation. Learning
from synthetic data often appears together with developing domain adaptation methods. Many
studies (Saito et al.) 2018 Chen et al.) 2019: Wang et al. 2019a] Bak et al. [0118] Yao et al.] 2020
Song et al.) 2020b) have been conducted investigating the Syn2Real domain gap. For example, most
existing work uses domain adaptation methods (Wang et al. 2019h Peng et al.) 2019 Wang et al.)
2022) to change the image style of the synthetic data to resemble that of the real-world data. On the other hand, some work (Kar et al. 2019 Yao et al. 2020 Ruiz et al. 2018) proposes new DA strategy for adapting the content of images. In this paper, several domain adaptation strategies will be evaluated on Alice provide baseline analysis for object re-ID tasks.
3 SYNTHETIC DATA IN ALICE
The synthetic data used in Alice are generated by the existing publicly available PersonX (Sun & Zheng 2019) and VehicleX (Yao et al. 2020) engines, used for person and vehicle re-ID, respectively.
PersonX has 1,266 person models that are diverse in gender (547 females and 719 males), age, and skin color. Meanwhile, the PersonX engine provides very flexible options to users, including the number of cameras, the camera parameters, scene conditions and other visual factors, such as illumination, which can be modified based the demands of different researchers.
3

---

Published as a conference paper at ICLR 2024
Table 1: Statistics of synthetic source data used in Alice benchmarks. The "arbitrary" means the number of images, cameras and scenes can be set based on the demand of users.
datasets
IDs/classes 1,266 & 1,362
image
scene
# camera arbitrary
# camera network
PersonX & VchicleX
arbitrary
arbitrary
customizable
VehicleX is designed for vehicle-related research. There are 1,362 vehicles of various 3D models. The environment setting is similar to PersonX. Attributes including vehicle orientation, camera parameters, and lighting settings, etc., are controllable.
PersonX and VehicleX are fully editable, and some statistics of them are shown in Table Here, "arbitrary" means a user can create an arbitrary number of images under user-specified environments. In other words, the settings, such scene style, number of cameras and data format (image or video) of the dataset are customizable in those two engines. Note that while we use them to generate source data for domain adaptation in this paper, the synthetic data can also be used for other research purposes (more discussions are provided in Section the Appendix).
REAL-WORLD DATA IN ALICE
Alice benchmarks currently have two re-ID tasks and thus two real- datasets, named AlicePerson and Alice Vehicle,,,, respectively, as target sets. They are collected, annotated and checked manually, and are summarized Table 2 Details of them are described below.
The AlicePerson dataset can be considered the successor to the Market- 1501 dataset (Zheng et al. 2015). However, the settings of the AlicePerson dataset are more challenging for recognition compared to those of the Market- 1501 dataset in a number of aspects: 1) there are more challenging samples (low-resolution, occlusion and illumination changes) in the new dataset, 2) the ratio of distractors is increased, such the increases in ratios of "distractor" and "junk" in gallery, 3) the training data of AlicePerson does not have identity annotation. Note that this means each ID is assured to appear in multiple cameras, enabling unassured clusterability.
AlicePerson Dataset:
Specifically, five cameras (three 1920x 1080 HD) cameras, one 1440x 1080 HD camera and one 570 SD camera) were used during dataset collection. There are 42,760 bounding boxes (bhoxes) in AlicePerson, in which 13,19 bboxes are unlabeled training data (camera IDs are available) and 29,562 bboxes are annotated validation and testing data. The validation and testing data contains large number of "distractors". Referring to the annotation method of the Market-1501 dataset, we use Faster R-CNN (Ren et al. 2015)) to detect the identities and select "good" images based on their IoU (Intersection over Union is larger than 50%) with the "perfect" hand-drawn bboxes. Moreover, "bad" detected bboxes (ratio of loU < 20%) are labeled as "distractor". We also add some images out of the annotated identities in the gallery set as "distractor" to simulate more realistic test situation.
The Alice dataset is built for vehicle re-ID and includes data from sixteen cameras on an urban road. The main road is straight road with some left and right branches in the middle, and the layout the sixteen cameras is shown in Fig. 2
Alice Vehicle Dataset:
As can seen, the cameras are positioned in various directions, with the fields of view (FoVs) of some cameras overlapping, such as cam8 and cam9. We kept the original distribution of the surveillance cameras on this road, such as their locations and rotations, reduce undesired biases. All camera resolutions are 1920x 1080. The training data are randomly selected from the target domain and do not have identity labels.
During annotating validation and test data, we labeled vehicles driving from caml cam16 in the opposite direction. Along the travel path of the vehicle, we gradually annotate it until it disappears from our sixteen cameras. To keep the diversity of the directions. also labeled some vehicles from the central cameras (e.g., cam8) and annotated their movement towards both sides of the road.
5 EVALUATION PROTOCOLS
Source Domain. We provide two types of synthetic data in the source domain, i.e., 3D models (Table in Unity and images (Table captured in the virtual environment. The first type of source data the 3D model. The provided models are fully editable and can generate an arbitrary number of images by arbitrary number of cameras, for the two tasks. The second type of source data (the
4

---

Published as a conference paper at ICLR 2024
source: synthetie
Heems
PersonX
VehicleX
410
image
carerra
image
14,760
45,538
larget: real-world
Heems
AlicePerson
VliceVehicle
16
image
13,198 (31%)
17,286 (63R)
training
anoo.?
N
class
N.A
N.A
image
3,978
(9%)
1,752 (6%)
validation
class
100
50
image
25,584 (60%)
8,552 (31%)
testing
anno.?
(hidden)
Y (hidden)
# class
hidden
hidden
Table 2: Statistics summary of source and target images in Alice benchmarks.
Figure 2: Camera topology in Vehicle, (A): Geolog- ical positions of cameras. (B): Camera field of views.
images) is given in the same way as existing synthetic datasets. These images are generated by setting environment attributes to certain values, which are randomly sampled from uniform distributions. We generated 14,760 and 45,538 images from PersonX and VehicleX engines, respectively.
Target Domain. For each task, we divide the real-world target data into three splits, ie,, training. validation and testing (details are shown in Table
The training sets contain 13, 198 and 17,286 the two tasks, respectively. Since we focus on unsupervised domain adaptation (UDA), the target training set should not have labels. To this end, we randomly select images from the target domain without annotating them. By random, we mean the number of identities for object re-ID is unknown (may contain outlier identities). Mcanwhile, the number of samples per identity for object re-ID also varies-sonee some have very few samples, and others would have much larger number of samples. For the re-ID tasks, having unknown number of target identities is more realistic and poses new challenges for existing algorithms. As be demonstrated in Section 6.3 many state-of-the-art UDA methods have lower performance on AlicePerson and Vehicle test sets than on existing benchmarks, such as Market- 1ikket--1001. and VeRi-776 (Zheng al. 2015 2017).
Each task includes a fully annotated validation set for the model and its hhper-paramene parameter selection, containing 7,978 (from 100 identities) and 1,753 (from 50 identities) images, respectively.
The testing sets in AlicePerson and Vehicle have 25,584 and 8,552 images, respectively. Their annotations hidden and only accessible to the organizers. Meanwhile, we keep the number of IDs unknown for object re-ID tasks to make the settings to be more consistent with the real-world scenario Model performance on the test set will be calculated through our server.
Evaluation Metric, For object re-ID, cumulative matching characteristics (CMC) and mean average precision (mAP) are used for evaluation, which are standard practices in the community (Zheng et al.) 2015 Liu et al. 2016). In CMC, rank-k t accuracy represents the probability that queried identity appears in the top-k ranked candidates list, and ranks 1, and are commonly used.
6 BASELINE EVALUATION
6.1 METHOD DESCRIPTION
Content-level domain adaptation. A distinct characteristic of synthetic data is that we can edit its content using the interface tools. This strategy has the potential reducing the content gap between real and synthetic data. In this paper, evaluate attribute descent (Yao et al.) 2020). a robust and stable method in this area. Specifically, for person and vehicle re-ID, there are five attributes to be optimized, including orientation (0* - 359"), light direction (from west (' to east 180"), light intensity, camera height and camera distance. During learning, the attributes are modeled by single Gaussian distributions Gaussian Mixture Model (GMM), and the difference in data distribution
5

---

Published as a conference paper at ICLR 2024
A
source set: PersonX
B
source set: VehicleX
90
-1.45
mean average precision (%6
improvement by attribute descent
mein averaaee precision (%)
random attr.
52
45
30
10:01
43.2643.19
+1,55
1.5
40.19
1.34
8.75
10224
10.25
7.
11,92
14.36
11.52-0.0
0.00
14.43
16:12
19:30
15
0.89
-2.34 W
X
126142.5
33,00
6:22
IDE Cyc&GGN
SPGAN
PUL ECN
UDA
MMT
IDE CycleGAN
SPGAN
PUL
ECN
UDA
MMT
CyeeCAAN
IDE
SPGAN PUL
UDA
MMT
CyllGAN SPGAN PUL
IDE
UDA
MMT
target set:
AlicePerson
Market-1501
Vehieee
Veri-776
Figure 3: Performance of domain adaptation methods from synthetic to real-world datasets. A: person re-ID using PersonX as the source. B: vehicle re-ID using VehieleX as the source. Under each target dataset, we use light green to indicate the accuracy of various methods and dark green to show the improvement brought by attribute descent combined with these methods.
is measured by the Fréchet Inception Distance (FID). The objective of this optimization problem is to minimize FID between real data and synthetic data.
Pixel-level domain adaptation. Two methods in this direction are evaluated, i.e., CycleGAN ZZu al. 2017) and its improvement in the re-ID task, SPGAN (Deng et al., 2018a). Specifically, CycleGAN learns two mappings, one from source to target and the other from target to source, and cycle consistency loss keep the similarity of original and cycle-transferred images. SPGAN is loss
designed for object re-II), which is realised based on CycleGAN, while a developed to preserve the cross-domain similarities. In our experiments, the settings from (Deng eet al., 2018 and (Yao et al. 2020) are used for person and vehicle domain adaptations, respectively,
Both content-level and pixel-level DA methods provide us with images that have a lower domain gap with the target domain. We then employ important baseline approaches for each task to evaluate the DA method, including ID-disrrimieative embedding (IDE) (Zheng et al. 2016), the bast- convolution baseline (PCB) (Sun et al. (2018) and Multi-Scale Interaction Network (MSINet) (Gu
ct al. 2023). When applying PCB in vehicle re-ID, images are divided into vertical stripes.
Pseudo-label based methods assign labels to the unlabeled target data based on an encoder. Then, through an iterative process, the labels will be refined, and the encoder updated. Here, the encoder is usually pre-trained on the source domain. Specifically, for object re-ID, PUL (Fan al., (2088). ECN (Zhong et al. 2019). UDA (Song et (0220a) and the pecent-state-oo-thhe-e MMT (Ge et al. 2020) chosen to be evaluated. We use OpenUnRelIDJJo to implement UDA and MMT. For ECN, we train StarGAN (Choi et al.) 2018) to generate the CamStyle for AlicePerson and run the experiments through the code released by ECN authors. PUL uses the re-implemented code. For vehicle re-ID, the image size is set to 256 x 256.
6.2 EFFECTIVENESS OF CONTENT-LEVEL DA METHOD
Attribute descent is found to generally improve accuracy. As an example, the mAP improvement in person re-ID is shown in Fig. 3 A. When attribute descent is used together with other methods, such as IDE, CycleGAN, SPGAN, PUL., ECN, UDA and MMT, it creates an improvement in mAP of +0%, +2.04%, +1.34%, 0.52%, +1.92% +1.52% and +0.60%, respectively, under the setting When Market-1501 is used as target sets, attribute descent also brings obvious improvements. On the Vlicele and VeRi-776 datasets, the use of attribute descent helps to create a better source dataset to improve vehicle re-ID results (Fig. 3 B). The mAP improvement on IDE, CycleGAN, SPGAN, PUL, UDA and MMT is +0.89%, +2.34% +1 +1.39% +2.61 %, +2.57% % and 10%,,, respectively for
Combining attribute descent with pixel DA are more effective than using attribute descent alone. As shown in Fig. A, "Pessseoo--i-iiePPrrrri when combined with CycleGAN, attribute
https://ggi www.:////ppmmmmmmmmmmmm

---

Published as a conference paper at ICLR 2024
Table 3: Comparison of Methods on AlicePerson and Market-1501 datasets. We show results of (1) direct transfer, (2) content DA, (3) pixel DA and (4) pseudo labels. Except for content DA, all the other methods use PersonX with random attributes as the source domain. 1mAP means the improvement of each method compared with the baseline IDE. rank-1 (R1) accuracy (%), rank-5
(R5) accuracy (%), rank- (1))) accuracy (%), and mAP (%) are reported.
PersonX > PAliceeeroon
PersonX
Market-
Method
RT
R5
RTO
mAP
1mAP
R1
R5
RTO
MAP
fmAP
IDE
15.16
26.50
33.29
6.22
36.22
51.99
59.56
14.88
0
(I)
PCB
15.03
27.62
34.61
6.29
0.07
37.29
54.66
62.20
15.29
0.41
MSINet
16.41
32.17
40.94
7.27
1.05
33.40
$1.66
59.65
14.34
-0.54
(2)
Attr. desc. IDE
16.08
27.42
32.82
6.41
0.19
38.98
54.81
62.17
16.79
1.91
CycleGAN
22.54
38.10
45.29
8.75
2.53
38,30
56.74
64.10
16.12
1.24
SPGAN
25.84
42.25
51,55
10.24
4.02
44.80
64.04
71.29
19.30
4.44
PUL
20.50
33.75
42.19
7.80
1.58
68.76
85.72
90.53
46.97
32.09
ECN
37.24
56.16
63.61
14.36
8.14
71.23
85.96
94.21
43.26
28.38
(4)
UDA
20.76
33.03
38,96
7.44
1.22
68,50
82.93
88.45
43.18
28.30
MMT
21.16
35,60
43.05
8.89
2.67
91.83
97.30
98.37
79.46
64.58
descent leads to +2.04% mAP improvement over using CycleGAN alone. In comparison, attribute descent offers an improvement of 10% mAP on the IDE baseline. The same trend can be observed when using 1500 as the target set. Similarly, for vehicle re-ID, the effect of attribute descent also becomes more prominent when used together with pixel DA. For example, attribute descent combined with SPGAN yields +2.39% mAP improvement over SPGAN itself (Fig. 3 B), and using attribute descent alone offers an improvement of +0.89% on the IDE baseline mAP score. A probable reason is that synthetic data is too simple to train robust model for complex real-world tasks. Style transfer increases the complexity of synthetic data by adapting image appearances and introducing noise, which is beneficial for training a more robust model. In this case, the advantages of content DA are more obviously apparent on the target task.
Attribute descent is less effective when used alone or with pseudo-label methods. As shown in Fig. A and B, attribute descent offers +0.19% and +0.89% mAP improvements over the IDE baseline on AlicePerson and Alice Vehicle, respectively. Compared to the same IDE baseline, the use of attribute descent improves mAP by +1 +1.91 %, and + +1.02% on Market- market-1501 1501, and VeRi-776. respectively. The benefits of attribute descent are stable but not significant, especially when compared to the improvements resulting from the combination of attribute descent and pixel DA. The reason could be, object re-ID works using small bounding boxes for learning, which do not contain much of the environmental context. Therefore, the use of content adaptation only results in relatively small improvements. However, as previously mentioned, combining content DA with pixel DA yields higher accuracy, indicating that the joint application of these techniques leads to more substantial improvements. For pseudo-label methods, a reason may exist in addition to the previous one. Pseudo- label methods like PUL MMT usually use an encoder pre-trained the source to compute pseudo labels. Essentially, these methods are not directly trained with source data. While content DA improves the quality of the source data, the improvement not significant enough affect the source pre-trained model. In other words, pseudo-label methods are less sensitive to the quality of source data: without content adaptation, the pre-trained encoder would still be good enough to mine high-quality pseudo labels. Similar observations have been made in existing work (Ge et al. 2020)
6.3 EFFECTIVENESS OF TRADITIONAL DA METHOD
Pseudo-label methods are less effective on AlicePerson and Alice Vehicle than on existing target datasets like Market-150h In Table 33 pscudo-label based methods yield a significant improvement in accuracy for setting, which is consistent with what has been reported in the literature. In comparison. content DA (an mAP score of 16.79%) and pixel DA (an mAP score of 19.30% mAP) are far less effective than pseudo-label methods. However, for "PersonX-+Alicee which is lower than the pixel-level SPGAN (an mAP score of 10.24%). The contrasting results are attributed to the data structure of the target datasets. In fact, the training sets in Market-1500 and VeRi-776 are originally intended for supervised learning, where each identity possesses a similar number of samples distributed evenly across cameras. Such underlying data structure is friendly to
-Mraeett- 1501". For example, the mAP score of MMT is 79.46% under this
MMT only obtains an mAP of 8.89%,
7

---

Published as a conference paper at ICLR 2024
Table Method evaluation on Vehicle and VeRi-776. We use VehicleX as the source domain. Similar to the previous table, we show results of (1) direct transfer, (2) content DA, (3) pixel DA and (4) pseudo labels. Notations and evaluation metrics are the same as those the previous table.
VchicleX Vchiccl
VehicleX
VoRi-776
Method
R1
R5
mAP
tmAP
RI
R5
MAP
1mAP
IDE
19.63
34,51
9.08
0
29.92
46.36
11.13
(1)
PCB
18.85
32.18
8.89
-0.10
28.61
42.37
11.36
0.23
MSINet
25.37
38.84
10.33
1.25
40.41
54.05
15.81
4.68
(2)
Attr desc. IDE
22.11
37,49
9.97
0.89
30,44
44.40
12.15
1.02
CycleGAN
25.58
44.37
11.85
2.77
42.49
60.13
17.53
6.40
(3)
SPGAN
29.41
49.04
13.83
4.75
46,96
63.11
19.10
7.97
PUL
37.77
50.74
15.73
6.65
59.77
68.47
23.00
11.87
(4) UDA
36.22
47.41
15.33
6.25
61.20
71.33
23.33
12.20
MMT
60.81
71.01
27.45
18.37
80.36
86.29
36.82
25.69
clustering-like algorithms such as pseudo-label mining. In comparison, the training set of AlicePerson is much more randomly distributed. For example, many identities have very few examples, and there no guarantee that images of each identity span multiple cameras. As result, this poses new challenges for data clustering in pseudo-label methods.
For Vehicle re-ID, there is huge difference between the results of mining pscudo labels on the VeRi-776 and AliceVehicke datasets (Table 4), but the differences in improvements (fmAP) are obvious. PUL enhances the mAP score on AliceVehicle by 6.65%, which is approximately half of the 11.87% observed VeRi-776. MMT improves +25.69% and +18.37% in mAP VeRi-776 and AliceVehicle, respectively. The improvement magnitude decreases from Alice Vehicle to VeRi-776, showing that the settings of AliceVchicle are more challenging for pseudo-label methods than those of VcRi-776. We suggest that the differences in performance on the person re-ID dataset are sharper than vehicle re-1l) dataset because changes in the appearances of cars limited compared the changes in people. Despite the randomness the training set Alice Vehicle, the clustering will not be much weaker than that of the VeRi-776, so there is no huge difference in the final results.
6.4 VISUALIZATION DA METHODS
To show the effect of content and pixel DA, as has been a focus of discussion above, here provide visualization examples of these two methods. For object re-ID, Fig. shows image examples before and after the content/pixel adaptation has taken effect. Specifically. we have the following observations. Firstly, attribute descent allows for key attributes of the synthetic data to tend toward the target domain. For example, the cameras of the target domain mostly of the same height as people, and attribute descent can effectively adjust source cameras to this angle. In vehicles, the target images are mostly in side view, and accordingly, the edited source images exhibit similar trend. Secondly, SPGAN causes obvious changes bringing synthetic images closer to the target style. For example, the skin color synthetic persons and the illumination/tone of vehicles becomes more realistic and closer to the target. Thirdly, when content and style alignment both exist, the source data exhibit the highest similarities to the target domain, which explains the superior performance of attribute descent in conjunction with pixel DA in Fig. 3 A and B.
7 DISCUSSION AND FUTURE WORK
This section will further discuss some open questions. challenges and interesting research problems for future investigations. More discussions can also be found in the Appendix.
Benefits of learning from synthetic data. Major advantages of synthetic data include the following, The first is low cost. When annotating AlicePerson, takes three people 8-9 hours each to annotate 80 IDs across cameras. In comparison, after crafting and modifying 3D person models, generating synthetic training data only takes minutes. Second, labels synthetic data are perfectly accurate.
In less exploited domain, synthetic data is controllable and customizable. For example, many tasks (e.g., semantic segmentation and object detection) require an agent to robust in various scenes under low lighting extreme weather. Such data usually difficult acquire in the real world, but
8

---

Published as a conference paper at ICLR 2024
randocn learsed
El domain
A3 randees
B2 leamned
compin
CI SPGAA
D1 SPGAN & learned att.
C2 SPGAN
D2 SPGAN & learned atc.
Figure 4: Sample images on person (left) and vehicle re-ID (right) before and after content/pixel domain adaptation. There are source images generated by (A1-2) random attributes, (B1-2) attribute descent, (C1-2) SPGAN and (D1-2) SPGAN & attribute descent. E1-2: show samples from the target domain. We find that attribute descent changes the viewpoint and illuminations etc., visual factors, of the objects, while SPGAN increases image blurring and adapts the image color to the target style.
can be easily generated through graphic engines. Moreover, synthetic data allows us to evaluate the robustness of algorithms by setting up various test sets with changing visual factors. In addition, synthetic data can alleviate concerns about data privacy and ethics.
Challenges of organizing synthetic datasets and improvement directions. Obtaining a variety of 3D object/scene models is the key to the successful use of synthetic data. The ideal goal is the development of virtual 3D world from which we can generate various synthetic data for a variety of computer vision tasks. However, the challenge is the lack of synthetic assets, such as 3D objects and texture materials. Besides manually crafting 3D models, would be beneficial resort 3D research, e.g., automatically generating 3D models based on real-world images (Wu) al. 2020), and automatically changing the appearance of 3D models acquire diverse data. Once the 3D models of synthetic engine are ready, it will be fast and inexpensive to generate as much data as needed. the future, we will gradually expand the Alice benchmarks to include more tasks and provide more complex and higher-quality synthetic data.
Generalization of conclusions drawn based on the Alice benchmarks. In most cases, we believe that conclusions drawn based on the Alice benchmarks are generalizable to other target datasets as well. In Table 3 and 4 in addition to the Alice targets, we also use existing datasets as the target domain and report the results of different methods. It is evident that there similar trends of results between Alice and other existing datasets. For example, SPGAN (Deng et al. 2018a) outperforms CycleGAN, and MMT (Ge et al. (020) obtains higher results than UDA (Song et al.) 2020a),, both which are consistent with reports in the literature. An exception is a comparison between pseudo-label methods and pixel DA methods: the former inferior for AlicePerson, which contradicts prior expectations. We have discussed this point in Section 6.3
CONCLUSION
This paper introduces the Alice benchmarks, including a series of datasets and an online evaluation server facilitate research on "synthetic to real"* domain adaptation. Our current version includes two tasks: person and vehicle re-ID. For each task, fully editable synthetic data and newly collected real-world data are used as the source and target respectively. We provide insightful evaluation and discussion of some commonly used DA methods on the content and pixel level, as well as some pseudo-label methods. We have discussed some interesting future research problems that are enabled by this platform, which may bring new and exciting ideas the community. The Alice project is long-term project. our future work, we will try to include more tasks beyond the existing re-ID task by welcoming open-source collaboration, with the aim of covering more objects in the visual world.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We thank all anonymous reviewers and AC for their insightful comments and suggestions in improving this paper. This research was funded in part by the ARC Discovery Project (DP210102801) to Liang Zheng, and the ARC Discovery Grant (DP220100800) to Hongdong Li.
REFERENCES
Slawomir Bak, Peter Carr, and Bean-Francois Lalonde. Domain adaptation through synthesis for unsupervised person re-identification. In Proceedings of the European Conference on Computer Vision, 2018.
Igor Barros Barbosa, Marco Cristani, Barbara Caputo, Aleksander Rognhaugen, and Theoharis Theoharis, Looking beyond appearances: Synthetic training data for deep enns in re-identification. Computer Vision and Image Understanding, 167:50-62, 2018.
Wenzheng Chen, Huan Wang, Yangyan Li, Hao Su, Zhenhua Wang, Changhe Tu, Dani Lischinski. Daniel Cohen- Or, and Baoquan Chen. Synthesizing training images for boosting human 3d pose estimation. In 2016 Fourth International Conference on 3D Vision, pp. 479-488, 2016.
Yuhua Chen, Wen Li, Xiaoran Chen, and Luc Van Gool. Learning semantic segmentation from synthetic data: A geometrically guided input-output adaptation approach. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1841-1850, 2019.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung- Woo Ha, Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE Computer Vision and Pattern Recognition, pp. 8789 8797, 2018.
Steve Dias Da Cruz, Oliver Wasenmuller, Hans-Peter Beise, Thomas Stifter, and Didier Stricker. Sviro: Synthetic vehicle interior rear seat occupancy dataset and benchmark. In The IEEE Winter Conference Applications of Computer Vision, 2020.
Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, et al. Robothor: An open simulation-to-real embodied ai platform. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2020.
Weijian Deng, Liang Zheng, Qixiang Ye, Guoliang Kang, Yang, and Jianbin Jiao. Image- image domain adaptation with preserved self self-similarity and dissimilattti for person re-identification. Proceedings of the IEEE Computer Vision and Pattern Recognition, 2018a.
Weijian Deng. Liang Zheng, Qixiang Ye, Yi Yang, and Jianbin Jiao. image domain adaptation for person re-identification. arXiv preprint arXiv:1811. 2018b.
image-
Anh-Dzung Doan, Abdul Mohsi Jawaid, Thanh-Toan Do. and Tat-Jun Chin. G2d: from gta to data. arXiv preprint
2018.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. arXiv preprint arXiv:1711.03930 2017.
Matteo Fabbri, Fabio Lanzi, Simone Calderara, Andrea Palazzi, Roberto Vezzani, and Rita Cucchiara. Learning to detect and track visible and occluded body joints in a virtual world. In Proceedings of the European Conference Computer Vision, 2018.
Hehe Fan, Liang Zheng, Chenggang Yan, and Yi Yang. Unsupervised person re- re-identification: Clustering and fine-tuning. ACM Transactions on Multimedia Computing. Communications, and Applications, 14(4):1-18, 2018.
Adrien Gaidon, Qiao Wang, Yohann Cabon, and Eleonora Vig. Virtual worlds proxy for multi- object tracking analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
10

---

Published as a conference paper at ICLR 2024
Yixiao Ge, Dapeng Chen, and Hongsheng Li. Mutual mean-teaching: Pseudo label refinery for unsupervised domain adaptation on person re-identification, In Proceedings of the International Conference Learning Representations, 2020.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2012.
Jianyang Gu, Kai Wang, Hao Luo, Chen Chen, Wei Jiang, Yuqiang Fang, Shanghang Zhang, Yang You, and Jian Zhao. Msinet: Twins contrastive search of multi-scale interaction for object reid. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 19243-19253, 2023.
Tao He, Leqi Shen, Yuchen Guo, Guiguang Ding, and Zhenhua Guo. Secret: Self-consistent pseudo label refinement for unsupervised domain adaptive person re-identification. Proceedings of the AAAI conference artificial intelligence, volume 36, pp. 879-887, 2022.
Yunzhong Hou, Liang Zheng, and Stephen Gould. Multiview detection with feature perspective transformation. In Proceedings of the European Conference on Computer Vision, 2020.
Yuan-Ting Hu, Kong-Shuo Chen, Kexin Hui, Jia-Bin Huang, and Alexander G Schwing. Soil-vos: Semantic amodal instance level video object segmentation-a synthetic dataset and baselines. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Amlan Kar, Aayush Prakash, Ming-Yo Yu Liu, Eric Cameracci, Justin Yuan, Matt Rusiniak, David Acuna, Antonio Torralba, and Sanja Fidler. Meta-sim: Learning generate synthetic datasets. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4551-4560, 2019.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474 2017.
He Li, Mang Ye, and Bo Du. Weperson: Learning generalized re-identification model from all- weather virtual data. In Proceedings of the 29th ACM international conference on multimedia, pp. 3115-3123, 2021.
Xuan Li, Kunfeng Wang, Yonglin Tian, Lan Yan, Fang Deng, and Fei-Yue Wang. The paralleleye dataset: A large collection of virtual images for traffic vision research. IEEE Transactions on Intelligent Transportation Systems, 20(6):2072 208444222,,24 2018.
Kevin Lin, Lijuan Wang. Kun Luo, Yinpeng Chen, Zicheng Liu, and Ming-Tine Sun. Cross-domain complementary learning using pose for multi-person part segmentation. JEEE Transactions on Circuits and Systems for Video Technology, 2020.
Xinchen Liu, Wu Liu, Huadong Ma, and Huiyuan Fu, Large-scale vehicle re-identification in urban surveillance videos. In Proceedings of the IEEE International Conference on Multimedia and Expo, pp. 1-6, 2016.
Yuchi Liu, Zhongdao Wang, Xiangxin Zhou, and Liang Zheng. A study of using synthetic data for effective association knowledge learning. Machine Intelligence Research, 20(2):194-206 2023.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In Proceedings of International Conference on Machine Learning, pp. 97-105, 2015.
Yihang Lou, Yan Bai, Jun Liu, Shiqi Wang. and Ling- Yu Duan. Embedding adversarial learning for vehicle re-identification. IEEE Transactions on Image Processing, 28(8):3794-3877, 2019.
Keith Man and Javaan Chahl. A review of synthetic image data and its use in computer vision. Journal of Imaging, 8(11):310, 2022.
Matthias Mueller, Vincent Casser, Jean Lahoud, Neil Smith, and Bernard Ghanem. Sim4ev: A photo-realistic simulator for computer vision applications. International Journal of Computer Vision, 08 2017.
11

---

Published as a conference paper at ICLR 2024
Jinjia Peng, Huibing Wang, Tongtong Zhao, and Xianping Fu. Cross domain knowledge transfer for unsupervised vehicle re-identification. In Proceedings of the IEEE International Conference on Multimedia & Expo Workshops, pp. 453-458, 2019.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprini wrXivv/710.000 2017.
Xingchao Peng, Ben Usman, Kuniaki Saito, Neela Kaushik, Judy Hoffman, and Kate Saenko. Syn2real: A new benchmark for synthetic-to-real visual domain adaptation. arXiv preprint vrXiv:/806.00755 2018.
Florent Perronnin, Jorge Sánchez, and Thomas Mensink. Improving the fisher kernel for large-scale image classification. In Proceedings of the European Conference on Computer Vision, 2010.
Aayush Prakash, Shaad Boochoon, Mark Brophy, David Acuna, Eric Cameracci, Gavriel State, Omer Shapira, and Stan Birchfield. Structured domain randomization: Bridging the reality gap by context-aware synthetic data. Proceedings of the International Conference on Robotics and Automation, pp. 7249-7255 2019.
Alexander Raistrick, Lahav Lipson, Zeyu Ma. Lingjie Mei, Mingzhe Wang, Yiming Zuo, Karhan Kayan, Hongyu Wen, Beining Han, Yihan Wang. Alejandro Newell, Hei Law, Ankit Goyal, Kaiyu Yang. and Jia Deng. Infinite photorealistic worlds using procedural generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12630-12641, 2023a.
Alexander Raistrick, Lahay Lipson, Zeyu Ma, Lingjie Mei, Mingzhe Wang, Yiming Zuo, Karhan Kayan, Hongyu Wen, Beining Han, Yihan Wang, et al. Infinite photorealistic worlds using procedural generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12630-12641, 2023b.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real- real-time object detection with region proposal networks. In Proceedings of Advances in Neural Information Processing Systems, pp. 91-99, 2015.
Stephan R. Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In Proceedings of the European Conference on Computer Vision, 2016.
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE Computer Vision and Pattern Recognition, 2016.
Nataniel Ruiz, Samuel Schulter, and Manmohan Chandraker. Learning to simulate. In Proceedings of the International Conference on Learning Representations, 2018
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In Proceedings of the European Conference on Computer Vision, 2010.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Computer Vision and Pattern Recognition, pp. 1723-3732,22 2018.
Swami Sankaranarayanan, Yogesh Balaji, Arpit Jain, Ser Nam Lim, and Rama Chellappa. Learning from synthetic data: Addressing domain shift for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Liangehen Song, Cheng Wang, Lefei Zhang, Bo Du, Qian Zhang, Chang Huang, and Xinggang Wang. Unsupervised domain adaptive re-identification: Theory and practice. Pattern Recognition, 102: 107173, 2020a.
Liangchen Song, Yonghao Xu, Lefei Zhang, Bo Du, Qian Zhang, and Xinggang Wang. Learning from synthetic images via active pseudo- heuudolabeling. IEEE Transactions on Image Processing, 2020b.
Xiaoxiao Sun and Liang Zheng. Dissecting person re-identification from the viewpoint of viewpoint. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
12

---

Published as a conference paper at ICLR 2024
Xiaoxiao Sun, Yunzhong Hou, Weijian Deng, Hongdong Li, and Liang Zheng. Ranking models in unlabeled new environments. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11761-11771, 2021.
Yifan Sun, Liang Zheng, Yi Yang, Qi Tian, and Shengjin Wang. Beyond part models: Person retrieval with refined part pooling (and strong convolutional baseline). In Proceedings of the European Conference on Computer Vision, pp. 480-496, 2018.
Zheng Tang, Milind Naphade, Stan Birchfield, Jonathan Tremblay, William Hodge, Ratnesh Ku- mar, Shuo Wang, and Xiaodong Yang. Pamtri: Pose-aware multi-task learning for vehicle re-identification using highly randomized synthetic data. In Proceedings of the IEEE International Conference on Computer Vision, pp. 21 211-220, 2019.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In Proceedings of the European Conference Computer Vision, 2011.
Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming- Hiung-ssuan Yang, and Manmohan Chandraker. Learning to adapt structured output space for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 7472-7481, 2018.
Tuan-Huu Hann-Hug Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Adver- sarial entropy minimization for domain adaptation in semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2517-2526, 2019.
Qi Wang, Junyu Gao, and Xuelong Li. Weakly supervised adversarial domain adaptation for semantic segmentation urban scenes. IEEE Transactions on Image Processing, 28(9)4477-4386, 2019a.
Qi Wang, Junyu Gao, Wei Lin, and Yuan Yuan. Learning from synthetic data for crowd counting in the wild. In Proceedings of the IEEE Computer Vision and Pattern Recognition, pp. 8198 8207, 2019b.
Yanan Wang, Shengeai Liao, and Ling Shao. Surpassing real-world source training data: Random 3d characters for generalizable person re-identification. arXiv preprint 2XXx:22006.12774 2020.
Yanan Wang, Xuezhi Liang, and Shengcai Liao. Cloning outfits from real-world images to 3d characters for generalizable person re-identification. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4900-4909, 2022.
Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. Person transfer gan to bridge domain gap for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 79-88, 2018
Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi. Unsupervised learning of probably sym- metric deformable 3d objects from images in the wild. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1-10, 2020.
Suncheng Xiang, Yuzhuo Fu, Guanjie You, and Ting Liu. Unsupervised domain adaptation through synthesis for person re-identification. In Proceedings of the IEEE International Conference on Multimedia and Expo, pp. 1-6, 2020.
Zhenfeng Xue, Weijie Mao, and I. Liang Zheng. Learning simulate complex scenes for street scene segmentation. IEEE Transactions on Multimedia, 2021.
Yue Yao, Liang Zheng, Xiaodong Yang, Milind Naphade, and Tom Gedeon. Simulating content consistent vehicle datasets with attribute descent. In Proceedings of the European Conference on Computer Vision, 2020.
Yue Yao, Liang Zheng, Xiaodong Yang. Milind Napthade, and Tom Gedeon. Attribute descent: Simulating object-centric datasets on the content level and beyond. TPAMI, 2022.
Yue Yao, Huan Lei, Tom Gedeon, and Liang Zheng. Large-scale training data search for object re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pp. 15578,, 2023a.
13

---

Published as a conference paper at ICLR 2024
Yue Yao, Xinyu Tian, Zheng Tang. Sujit Biswas, Huan Lei, Tom Gedeon, and Liang Zheng. Training with product digital twins for autoretail checkout. arXiv preprint arXiv:2308.09700; 2023b.
Tianyu Zhang, Lingxi Xie, Longhui Wei, Zijie Zhuang, Yongfei Zhang, Bo Li, and Qi Tian. Unre- alperson: An adaptive pipeline towards costless person re-identificatton. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11506-11515 2021.
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification: benchmark. In Proceedings of the IEEE International Conference on Computer Vision, 2015.
Liang Zheng, Zhi Bie, Yifan Sun, Jingdong Wang, Chi Su, Shengjin Wang. and Qi Tian. Mars: A video benchmark for large-scale person re-identification. In Proceedings of the European Conference Computer Vision, pp. 868-884,10 2016.
Yi Zheng, Shixiang Tang. Guolong Teng, Yixiao Ge, Kaijian Liu, Jing Qin, Donglian Qi, and Dapeng Chen. Online pseudo label generation by hierarchical cluster dynamics for adaptive person re- identification. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8371 ,381,2021
Zhedong Zheng, Liang Zheng, and Yi Yang. Unlabeled samples generated by gan improve the person re-identification baseline vitro. In Proceedings of the IEEE International Conference on Computer Vision, 2017.
Zhun Zhong, Liang Zheng, Zhiming Luo, Shaozi Li, and Yi Yang. Invariance matters: Exemplar memory for domain adaptive person re-identification. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Zhun Zhong, Liang Zheng, Zhiming Luo, Shaozi Li, and Yi Yang. Learning to adapt invariance in memory for person re-identification. IEFE Transactions Pattern Analysis and Machine Intelligence, 2020.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEFE International Conference on Computer Vision, pp. 2223-2232, 2017.
14

---

Published as a conference paper at ICLR 2024
APPENDIX
In the Appendix, we present the summary of existing synthetic datasets (Section Aj. the dataset annotation procedure (Section[B) [ sources of datasets and DA method (Section C well as discussion and future work (Section
and Section D). as
A SUMMARY OF EXISTING SYNTHETIC DATASETS
Table 5 presents some synthetic datasets published in recent years, designed for various purposes. This indicates synthetic data has emerged powerful tool, offering flexibility in data design and addressing challenges related to data scarcity and privacy concerns. In line with this, our research endeavors to push the boundaries of synthetic data in the field of object re- re-identification.
Table 5: Example synthetic datasets released in recent years.
name
task
data format
engine
SOMAset Barbosu et al.
2018
person re-1D>
image
Unreal
SyRI (Bak al.
2018
person ID
image
Blender
PersonX Sun Zheng
2019
person ID
image/model
Unity
RandPerson Wang et ml
2020
person ro-ID
image/vidoo
Unity
UnrvalPerson Zhang al.
2021
person 1--1
image
Unreal
ClonedPerson Wang et
2022
person ID-1D
image/model
Unity
WePerson
Li al
2021
person re
image/model
Script Honk V
GPR (Xinng ct
2020)
person re-ID)
image
GTA V
VehieleX Yao
2020
vehicle ro-ID
image/model
Unity
PAMTRI (Tang et al.
2019
vehicle ro-1D>
image
Unreal
SYNTHIA Ruu et al.
2016
segmentation
image
Unity
GTAS Richter al.
2016
segmentation
image
| GTA
SceneX Xue et
202
segmentation
image/model
| Unity
SAIL-VOS (Hu ct
2019)
segmentation
imago/vidoo
GTA
GCC (Wag
2019b)
counting
image
GTA
G2D DDaa al.
2018)
SIM
imuge/engine
GTA
SDR (Prikash et al
2019
car detection
image/model
Unreal
CARLA (Dosovitskiy et al
2017
depth ostimation & segmentation
image/model
Unreal
Sim4CV Mueller cl
2017
navigation tracking
image/model
Unreal
Virtual KITTI Gaidon et al.
2016
object tracking & detection & segmentation
image/video
Unity
synthetic2real (Peng al.
2018
classification & detection
image
CAD
VisDA (Peng et al.
2017
classification & segmentation
image
CAD GTA V
A12-THOR (Kolve et
2017
navigation
imago & model
| Unity
RoboTHOR Deitke et
2020
navigation & tracking
image
Unity
Meta Sim Kar al.
2019
detection
image
|
Unreal
SVIRO Cruz et
interior vehicle sensing
image
|
Blender
MultivieXX (Hou al
2020
malti-view & detection
image
Unity
B DATA ANNOTATION
Figure 5 displays the GUI, where Top and Bottom are examples for annotating different datasets. Annotators collaborate based on the video recording timestamps. Each annotator oversees 1-2 cameras, collectively identifying individuals with the same ID. After annotation, we conducted multiple rounds of checks on the data to ensure the accuracy of annotation. Our data is annotated
15

---

Published as a conference paper at ICLR 2024
Labeingg options
Leaf
Diox
New
Provices
tboe
Presken
coml ong
mar
carl, 1,0000
GUI for AlicePerson
Labeling aptions
Labeling optons
Cucara
Seeg Skipe
Load
ibox
Newr
Pleviess
Sime
Stad by
Ui
Caseal
Ing ramat can't 2eRR22
GUI for Vehicle
Figure 5: Graphical user interface (GUT) used in annotation process. AlicePerson (Top) and AliceVehicl (Bottom). Annotators work together according to the time of video recording. Each annotator charge of 1-2 cameras, looking for the same ID together.
by a data annotation company, which has a legitimate business license and meets the required wage levels (24 AUD Hour) of the government.
Test set statistics are shown in Fig. For AlicePerson, we can observe that the 4th camera captures larger number of unique IDs compared to the other cameras from A of Fig. (6 Additionally, is noticeable that most IDs appear in 4 or cameras. For Vehicle, each camera includes about 300-800 images, and most IDs appear in 6 cameras. In the test set, do not have ID only appears in one camera except for person on distractor images.
We would like to provide more discussion of our datasets. Beyond the clusterability, our real-worla datasets exhibit several significant differences when compared to existing datasets like Market-15011 1) Target training data more consistent with real-world distributions because they are not manually selected by humans. 2) AlicePerson and
more challenging datasets because existing methods get lower improvements on them than on existing datasets. 3) The test labels of AlicePerson and AliceVehicke are hidden, which will offer unified, fair, and challenging benchmark.
Meanwhile, datasets are only part Alice Benchmarks. Alice provides: 1) an online evaluation plat- form which far as we know is the first online evaluation server for person/vehicle re-identificatin;; 2) evaluations and analysis of different domain adaptation methods, which can provide some insights into learning from synthetic data; discussions of interesting questions and future directions.
16

---

Published as a conference paper at ICLR 2024
A
images O-#
# images
IDs
12
xtoo
iimage
cameras
# image each cameer
# images'appeared camera number of each ID AlicePerson
# images
IDs
images
IDs
90
Dimaggs
images
12
# cameraa
30
2
3
5 6 7
8
10
11
12
13 14
15
16
image and ID of each camera Vehccle
amagee/appeared camera number each AliceVehicle
Figure 6: Statistics of the AllicPeeroon/Aliceeeiie son and Vehicle, we display the number of images and object IDs that have appeared in that camera. and D: For each ID of AlicePerson and Vehicl,, we show the number of images and how many cameras the ID appeared. Note that the number of IDs in the test set is hidden for online evaluation. # is the notation of "the number of".
test sets. A and C: For each camera of AlicePer-
C LINKS TO DATASETS
We make the real data we collected for Alice benchmarks publicly available on the links below:
Alice-train:
httpps//drive..go wttp///oo////////eooooo http::/www.p......oo.o//oo/o///o
Alice-validation-test
1ing
The synthetic data we reused is also available on these links:
PersonX: http:///w..s.ttt..c..////t//ttts
VehicleX: httppps//gtthu...ccoo///pppeeee/
D
SOURCE CODE OF DA METHODS
Attr: desc.: http:t/s//ggtttooooo////////saaa
IDE: http:///chhaaaacoo//////////eced
PCB: http::///ttiiiica..a///////aaaaa
and httpa///wwaaaa/////aa/aaaaaaaaaa
ECN: httpss//gghhhb.ocomm///huuuooggn
UDA: [ttpss//ggthhicccommm//ermm/m//r
MMT: httpps///gtttbbb..coo///mmmm/m/p
E
DISCUSSION AND FUTURE WORK
Understanding the domain gap between the synthetic and the real. The domain gap is usually studied with real-world datasets. For example, Torralba et al. (Torralba & Efros 201 analyze and summarize different types of domain bias, such as the selection bias (different collection sites, websites, etc) and the caption bias (different viewpoints, resolutions etc). In the context "synthetic to real", the domain gap may be different from that in "real to real" DA. For example, the difference between 3D object models and real-world objects may create a domain gap that does not occur under the "real to real" setting. It is still largely unknown whether existing "real to real" DA methods
17

---

Published as a conference paper at ICLR 2024
Table 6: FID values between target and synthetic data with (Attr. desc) and without (Random)
content-level DA. A lower FID value means a smaller distance between synthetic and real data.
Target datasets
Random
Attr, desc
Market-1501
104.78
83.24
AlicePerson
134.00
121.75
VeRi-776
80.22
57.45
VeiceVe
78.05
56.07
can handle such new problems. Also, simulated data has lower diversity than real-world data in complex environments. However, given that synthetic data can be simulated in large amounts, would it compensate for its lower data diversity? In this regard, synthetic data makes an inferior source domain to real data. In other words, given the same target domain, which should we choose source: real data or editable synthetic data?
interesting question would be whether
Designing "synthetic to synthetic" DA evaluation protocols for comprehensive evaluation of DA methods, dissecting the domain gap and providing references for "real to real" DA. The evaluation and understanding of DA methods are limited by the scale and variation of both the source and the target data. This problem can be overcome by controllable and customizable synthetic data. For example, we can generate source and target data with similar styles but very different content distributions for semantic segmentation. Conversely, we can synthesize data with similar content distributions but very different styles. With various data settings, we can conduct well- directed and comprehensive evaluations of domain adaptation methods, which will also be helpful for understanding the domain gap. On the other hand, although quantitatively defining the domain gap caused by changes in various visual factors is rather infeasible, it is possible to analyze the domain gap in higher capacity by using synthetic data. For example, the illumination of source data be gradually changed for a fixed target set to analyze the influence of illumination differences on domain adaptation. Similarly, we can study the relative changes of various visual factors between the source and target domain, which may bring us a better and deeper understanding of the domain gap.
Need for new pseudo-label methods for object re-ID under more realistic settings. The object re-ID results in Tables 3 and 4 show that state-of-the-art pseudo-label methods, such as MMT, demonstrate high performance when using the Market-1501 and VeRi-776 as target sets, but do not work well on the Alice benchmarks (refer Section 6.3). Given the decreased performance of pseudo-label methods, we should rethink the clustering strategies used for more realistic target sets. It would be interesting to study how to better leverage the camera style of the target set to assist with data clustering. Adopting this strategy, ECN yields the highest results for AlicePerson (Table (3)
New strategies to effectively use synthetic data to bridge the gap with the real world. This paper discusses several existing strategies such as pixel DA and the relatively new content DA. For the latter, there are still many open questions, e.g., the design of new similarity measurements for distributions apart from the FID score used in (Yao et al., 2020 Sun et al. 202T) (Table some results of the FID values). Morcover, the characteristies of good training set are still largely unknown. In this regard, having a smaller domain gap between training and testing sets might not be the only objective. Other indicators such as appearance diversity even noise worth investigating. Furthermore, a variety of annotation types can be obtained from synthetic data, such as pixel-level semantics, bounding boxes and depth. These annotations may facilitate research in multi-task learning, which would also potentially improve real-world performance. In addition, it is an interesting task to explore how to combine simulation 3D models and real images in learning.
Is it possible to evaluate the quality of generated data and its effect on enhancing results? Yes, but is a complex issue. In this paper, we offer some preliminary discussions. The assessment of synthetic data quality includes two primary dimensions: annotation quality and image quality. In terms of annotation quality, synthetic data have near-perfect label accuracy, largely thanks to an automated annotation process. This involves creating images with annotations derived from the IDs of 3D models of people and vehicles. Once these models are set up, a system automatically generates annotations, significantly reducing the chance of labeling errors. Consequently, the reliability of labels in our synthetic data is very high. Image quality, on the other hand, is affected by various factors, such as resolution and the sophistication of the 3D models used. Sun & Zheng (2019) have shown in Figure 4 in their paper that utilizing high-resolutian images leads to improved outcomes. Furthermore, while Man & Chahl (022) explored the impact of using different graphics engines
18

---

Published as a conference paper at ICLR 2024
like Unity and Unreal on data quality, they conclude that the choice of the engine plays lesser role compared to the quality of the 3D models themselves. Therefore, we believe that the generation of high-quality synthetic data is predominantly influenced by the quality of 3D assets over the choice of graphics engine. Despite the discussion above, understanding how the quality of synthetic data affects the results presents highly complex question and warrants separate investigation.
Can we finally get rid of real-world data when proposing new models? While state-of-the-ar techniques in computer vision have been developed on real-world datasets, it would be interesting to study whether we can resort to pure synthetic data (training + testing) in model development, underpinned by the increasing ethics concerns in artificial intelligence (AI). Several challenges are yet to be resolved. Firstly, diverse, complex, and realistic synthetic data needs to be created. Secondly, necessary to confirm that training and testing on synthetic data are analogous to real-world performance. Thirdly, we need to explore the optimum composition of test data so as to comprehensively evaluate model performance. Apart from these challenges, there are many task-specific problems to be considered. At this point, it has not been determined how real-world test data can be replaced with synthetic data, but we will consider this to be an option with future higher-fideline data generators and stronger theoretical support.
What are the potential limitations of using synthetic data to enhance learning and inference in real-world scenarios? Beyond the commonly acknowledged domain gap between synthetic and real-world data, a significant challenge in leveraging synthetic data lies in the complexity of creating specific datasets for targeted tasks, like those in medical imaging. Nonetheless, there is promising movement within the research community towards generating more varied synthetic datasets that encompass a wider array of objects. An illustrative example the Infinigen dataset (Raistrick et al. 2023a), demonstrating the growing capabilities of synthetic data generation. Therefore, even with these limitations, the scope for using synthetic data broadening. We expect its application across an increasing number of scenarios in the future.
What is the relationship between the control of synthetic scene creator and its influence on the content gap with real-world data? Initially, human intervention in adjusting the content of simulated scenes can effectively minimize the content gap by exerting control over various factors, such as lighting conditions, object placement, and background variations. However, there are significant limitations, including the high cost associated with generating large-scale datasets. This constraint has significantly motivated the research community to explore machine learning methods aimed at automating the reduction of content gaps. Furthermore, content-level DA operates in a manner akin to manual control of visual attributes within scenes, although it remains far from perfect. This approach employs the Fréchet Inception Distance (FID) to evaluate the quality of synthetic data and subsequently adjusts the parameters governing visual attributes in the simulation environment. This iterative process aims to progressively diminish the content gap and enhance alignment between synthetic and real data distributions.
The experiments were conducted on a server equipped with four RTX-2080TI GPUs and a 16-core AMD Threadripper CPU @ 3.5Ghz.
19


Reviews:
[{"review_id": "LAZVmlQ5Sj", "reviewer": "Reviewer_dCYN", "summary": "This paper presents the Alice benchmarks, a resource designed for advancing research in \"synthetic to real\" domain adaptation, particularly in the context of person/vehicle re-identification (re-ID). The primary aim is to train models from synthetic data that perform effectively in real-world scenarios. The paper outlines the benchmark settings, analyzes common domain adaptation methods, and sets up an online server for convenient and fair method evaluation. The Alice project is a long-term effort with plans to expand to cover more objects in the visual world through open-source collaboration.", "questions": "None", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1)\tThis paper offers a good benchmark for \"synthetic to real\" domain adaptation in the domains of person and vehicle re-identification, significantly enhancing progress in these areas.\n2)\tLots of experiments have been constructed, and many interesting findings have been obtained.", "weaknesses": "1)\tThe authors should consider the impact of different synthetic data generation strategies on domain adaptation effects, such as using different graphics engines, different attribute distributions, different noise levels, etc. \n2)\tThe quality of the generated data and labels is not clearly discussed. I recommend measuring the quality of the generated data and the impact of improving quality on the results. \n3)\tThe title of this paper is biased. It focuses on the person/vehicle re-identification, while the title uses object re-identification. The scope of the title is too large. Although the authors claim that more objects will be added later, for the purposes of this article, it is limited to the field of person/vehicle re-identification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xC0ge4PlBw", "reviewer": "Reviewer_txUp", "summary": "In this work authors present a new benchmark, namely Alice for well-known re-identification of people and Vehicles. Specifically, the dataset is constructed with the domain adaptation in mind. This provides a great opportunity for DA research and more consistent basis for comparison. In the dataset authors tried to ensure that the collected real-world dataset resembles the unconstrained data in the wild. Additionally, the benchmark comes with an online evaluation system for a fair comparison.", "questions": "-", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper is well-written, and the motivation is quite clear. The paper has a very nice structure and flow. A unified evaluation framework is gathered to ensure fairness across results. Additionally, authors provided baseline results based on well-knows domain adaptation works and techniques. This provides unique opportunity to enhance domain adaptation research. Moreover, authors showed how established techniques such as unsupervised clustering of images that result in reasonable performance gains in existing benchmarks does not translate to their dataset highlighting the undesired biases that sometime are over-looked.", "weaknesses": "Despite not being quite clear at this point, it would be nice if authors could provide some discussion on the potential limit that synthetic data can benefit the learning and inference on real-world data. Specifically whether there are some inherent barriers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tGjTBEQu4E", "reviewer": "Reviewer_ZhRK", "summary": "This paper proposes a new benchmark for synthetic-to-real object reid. The new benchmark involves two real-world data, i.e., AlicePerson and AliceVehicle, and provides some analysis on them.", "questions": "Beside the introduce of the strong clusterability issue, I can not find any new things in this benchmark. Please see the weakness for details.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The strong clusterability in current data actually brings a risk to build a robust UDA reid method, which deserves atttention.", "weaknesses": "1. The synthetic data in benchmark are from existing datasets, i.e., PersonX and VehicleX, while the real-world data (AlicePerson and Alice Vehicle) in benchmark shows less advantage compared to current off-the-shelf reid datasets. The strong clusterability issue is the only difference in my opinion. However, for AlicePerson, I do NOT think adding distractor and junk is a good way to solve the strong clusterability issue compared. For AliceVehicle, in Table.4, I do NOT find much difference between AliceVehicle and VeRi776 on the strong clusterability issue.\n2. The UDA methods compared in this benchmark (Fig.3&Tab.3&Tab.4) are out of fashion, new methods should be involved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I3RxY7zWkA", "reviewer": "Reviewer_6iq6", "summary": "The paper focuses on the research of synthetic data for object re-identification. Although using the name of object re-identification, it only focuses on persons and vehicles. The main contribution of the paper is the proposed Alice benchmarks, which include both synthetic and real-world data. Specifically, an application is training a model with fully-labeled synthetic data and then tuning it on the unlabeled real-world data. A significance of the paper is its real-world data do not have pre-define “cluster” architecture, which is more practical. The authors try to regard it as a successor \n of the famous Market benchmark.", "questions": "Please see the weakness.,", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Focusing on synthetic data is good; which does not invade privacy;\n2. Good writing and easy to follow;\n3. Some insightful discussions in DISCUSSION AND FUTURE WORK", "weaknesses": "Although with these strengths, I think the paper is still far away from the acceptance of ICLR 2024.\n1. The contribution is not enough. I fully understand the importance of the benchmark. But as far as I am concerned, the paper’s biggest significance is the “un-clustered” data architecture. Some other contributions as benchmarking existing works and discussion. They cannot be considered as significance due to (1) no significance contribution beyond other works. (2) no novel method is proposed to customize the new-labeled synthetic to real dataset.\n2. The domain adaptation seems not suitable for synthetic to real datasets. The gain of synthetic data is easy and straightforward. Therefore, we should generate more diversed synthetic data to achieve domain generalization rather than domain adaptation. Please refer to ClonePerson and RandPerson for this kind of research;\n3. The authors discuss the legitimate in data-labeling, however, the main step that incurs the privacy problem is capturing data. How can your data captured in Figure 2 protects the privacy of persons and vehicles. As far as I concern, the passerbys cannot know their cars or themselves are trained for AI models. A license from local government is not enough, it seems to be the human right matters;\n4. The authors try to provide some understanding, however, these understandings are not well-defined by mathematics. Instead, they are some intuitive ones without proof. Some understanding in theory is expected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)