IDTraffickers: An Authorship Attribution Dataset to link and connect Potential Human-Trafficking Operations on Text Escort Advertisements
Authors: Vageesh Kumar Saxena
Benjamin Ashpole
Gijs van Dijck
Gerasimos Spanakis
Venue: EMNLP 2023 Main (2023)
Keywords: Human Trafficking
Authorship Attribution
Natural Language Processing
Dataset
Benchmarks

Abstract: Human trafficking (HT) is a pervasive global issue affecting vulnerable individuals, violating their fundamental human rights. Investigations reveal that a significant number of HT cases are associated with online advertisements (ads), particularly in escort markets. Consequently, identifying and connecting HT vendors has become increasingly challenging for Law Enforcement Agencies (LEAs). To address this issue, we introduce IDTraffickers, an extensive dataset consisting of 87,595 text ads and 5,244 vendor labels to enable the verification and identification of potential HT vendors on online escort markets. To establish a benchmark for authorship identification, we train a DeCLUTR-small model, achieving a macro-F1 score of 0.8656 in a closed-set classification environment. Next, we leverage the style representations extracted from the trained classifier to conduct authorship verification, resulting in a mean r-precision score of 0.8852 in an open-set ranking environment. Finally, to encourage further research and ensure responsible data sharing, we plan to release IDTraffickers for the authorship attribution task to researchers under specific conditions, considering the sensitive nature of the data. We believe that the availability of our dataset and benchmarks will empower future researchers to utilize our findings, thereby facilitating the effective linkage of escort ads and the development of more robust approaches for identifying HT indicators.

IDTraffickers: An Authorship Attribution Dataset to link and connect Potential Human-Trafficking Operations on Text Escort Advertisements
Vageesh Saxena Law & Tech Lab
Benjamin Bashpole
Bashpole Software, Inc. bashpole @ idrraffickers.com
Maastricht University
v.saxena @ maastricttuivvrrstt..ll
Gijs Van Dijek
Gerasimos Spanakis Law & Tech Lab Maastricht University
Law & Tech Lab
Maastricht University
vijd.vkk maastrichttuniverstyy..l
@ maastrichtuniversity..ll jerrs
Abstract
deception, and debt bondage to force them into commercial sex (ILO, 2012). These operations in various locations such as massage businesses, brothels, strip clubs, and hotels (EUROPOL, 2020). Women and girls comprise a significant portion of HT victims, particularly in the commercial sex industry (ILO, 2012). Despite being advertised on- line, many victims have no control over the content of the advertisements (ads). Around 65% of HT victims are advertised online for escort services in the United States (POLARIS, 2020). However, the large number of online escort ads makes manual de- tection of HT cases infeasible, leading to numerous unidentified instances (POLARIS, 2018).
Human trafficking (HT) is pervasive global issue affecting vulnerable individuals, violat- ing their fundamental human rights. Investi- gations reveal that many HT cases are associ- ated with online advertisements (ads), particu- larly in escort markets. Consequently, identi- fying and connecting HT vendors has become increasingly challenging for Law Enforcement Agencies (LEAs)). To address this issue, we introduce IDTraffickers, extensive dataset consisting of 87,595 text ads and 5,244 vendor labels to enable the verification and identifica- tion of potential HT vendors online escort markets. To establish a benchmark for author- ship identification, we train DeCLUTR-sine small model, achieving macro-Fi Fl score of 0.8656 in a closed-set classification anvironment. Next, we leverage the style representations extracted from the trained classifier to conduct author- ship verification, resulting in a mean r-precision score of 0.8852 in an open-set ranking environ- ment. Finally, to encourage further research and ensure responsible data sharing, we plan release IDTraffickers for the authorship at- tribution task to researchers under specific con- ditions, considering the sensitive nature of the data. We believe that the availability of our dataset and benchmarks will empower future researchers utilize our findings, thereby fa- cilitating the effective linkage of escort ads and the development of more robust approaches identifying HT indicators !.
Researchers and law enforcement agencies (LEAs) rely sex trafficking indicators (lbanez and Suthers, 2014; Ibanez and Gazan, 2016; Lugo- Graulich and Meyer, 2021) identify HT ads. However, these investigations require linking ads individuals or trafficking rings, often using phone numbers or email addresses to connect them. Our research reveals that only 37% (202,439 out 513,705) of collected ads have such contact in- formation. Moreover, manual detection of HT cases is time-consuming and resource-intensive due to the high volume of online escort ads. To address this, researchers and LEAs are exploring automated systems leveraging data analysis (Ke- skin et al., 2021), knowledge graphs (Szekely et al.. 2015; Kejriwal and Szekely, 2022), network the- ory (Ibanez and Suthers, 2014, 2016; Kejriwal and Kapoor, 2019; Kosmas et al., 2022), and machine learning (Dubrawski et al., 2015; Portnoff et al., 2017; Tong et al., 2017; Stylianou et al., 2017; Al- vari et al., 2017: Shahrokh Esfahani et al., 2019; Wiriyakun and Kurutach, 2021: Wang et al., 2019). recent literature review by (Dimas et al., 2022) highlights current trends on various research fronts for combating HT.
1 Introduction
Human trafficking (HT) global crime that ex- ploits vulnerable individuals for profit, affecting people of all ages and genders (EUROPOL, 2020; UNDOC, 2020). Sex trafficking, a form of HT, in- volves controlling victims through violence, threats,
'Our code implementation is publicly available at http://gtthh..oommmaaccccttcttto

---

Uhstfickan Detamt
unique
Bequasss
Veaking privato advertiamment
Assigring Vender
PAUSS Prodect
communiite
Montificttionn
Pre-trinee Checigeist
Clasitter
opremuntiies by connectngg phone
Networkk
Rocort
Glessheer
CNNLLSTN ORF (Phone Number
Opennoa
Text
Rookeage Escort
Trikning Batasse
Datasal
Trakning Dasoestt
69
(4)
Figure 1: (i) ITraffickers: Preparing authorship dataset from Backpage Escort Market, (ii) Authorship Identifi- cation Task: identifying HT vendors in closed-set environment, (iii) Authorship Verification Task: Verifying HT vendors using advertisement similarity in open-set environment.
Although most of the abovementioned studies were conducted on online ads from the Backpage escort market, none analyzed authorship features to link and connect these trafficking operations. In the absence of phone numbers, email addresses, and private identifiers, such authorship techniques can become key to connecting vendor communi- ties and analyzing the language, style, and content of escort ads. Therefore, as illustrated in Figure 1, this research focuses on bringing the following contributions to bridge the gap between authorship techniques and HT:
involved in trafficking operations.
(ii) Authorship Benchmarks: On escort mar- kets, multiple vendors and communities often share a single account and post numerous ads. More- over, some vendors create multiple accounts to avoid detection by LEA and expand their busi- ness. To address these challenges, we first establish an authorship identification (Green and Sheppard, 2013; Benzebouchi al., 2019; Abbasi et al., 2022; Manolache et al., 2022; Saxena al., 2023) bench- mark through a closed-setting classification task (Vaze et al., 2022) (Figure (ii)). Given a specific text, the objective of the classifier is to predict the vendor that posted the advertisement. Furthermore, using the style representations from our trained classifier, we also establish an authorship verifica- tion (Stamatatos, 2016; Halvani al., 2016; Benze- bouchi al., 2018; Manolache al., 2022: Saxena et al., 2023) benchmark through an open-setting text-similarity-bbaed ranking task (Figure 1(ii)). Given two ads, we compute the cosine similarity between the style representations to analyze the patterns in writing style and determine if they came from the same vendor.
(i) Authorship dataset: Through this research, we release IDTraffickers, an authorship attribution dataset of 87.5K text ads collected between Decem- ber 2015 and April 2016 from the United State's s Backpage escort market 2. Analyzing the language and content of these escort advertisements can pro- vide crucial insights into the authorship traits and patterns associated with trafficking operations. Fur- thermore, by developing authorship attribution ap- proaches on such a dataset, we can uncover recur- ring patterns and use them to link ads from poten- tial HT communities, thereby bridging the gap in identifying and connecting individuals or groups
Related Research
"Establishing ground truth for escort ads associated with IIT is challenging as the undetected crimes are hard to identify. Even data based on arrest records may not encompass all un- detected ads. To ensure the inclusion of HT ads, we analyzed Backpage escort website ads, which previous investigations have linked to HT ads (Callanan et al., 2017). Our primary objective is not to directly identify HT ads but to provide a tool that assists law enforcement in prioritizing resources hy effectively detecting potential indicators of HT ads.
Linking Escort Advertisements: Identifying HT operations in escort ads is challenging task that requires multidisciplinary approach combin- ing expertise in linguistics, data analysis, machine lcarning, and LEAs collaboration. Recent advance- ments in natural language processing (NLP) have
2

---

led researchers identify indicators using entity linking approaches (Alvari al., 2016; Nagpal et al., 2017a; Whitney et al., 2018; Alvari et al., 2017: Li et al., 2022) for automatic detection of HT operations. However, these indicators can only be studied within cluster of ads linked with indi- vidual vendor accounts. Previous work by Cham- bers et al. (2019) proposed using neural networks extract phone numbers to connect these escort ads. Nonetheless, our research demonstrates that only 37% of the ads in our dataset contained phone numbers. While clustering approaches (Lee et al., 2021; Vajiac et al., 2023a; Nair et al., 2022; Va- jiac et al.. 2023b) can assist in connecting near- duplicate ads, they fail to establish connections in paraphrased and distinct ads. Therefore, we focus research on leveraging authorship techniques analyze unique writing styles within escort ads and establish connections with individual vendors.
tection (Alazab al., 2013; Jones al., 2022), and linking vendor accounts on darknet markets (Ekam- baranathan, 2018; Tai al., 2019; Manolache al., 2022: Saxena et al., 2023). However, to our knowl- edge, none of the existing studies focus on connect- ing vendors of HT through escort ads. In this re- search, we address this gap by introducing a novel dataset, IDTraffickers, which enables us to high- light the distinctions between language in existing authorship datasets and escort ads. Furthermore, we demonstrate the capabilities of authorship at- tribution approaches in establishing connections between escort advertisements and HT vendors us- ing authorship verification and identification tasks.
Dataset
The data in this research is collected from online posted escort ads between December 2015 and April 2016 on the Backpage Market, classified ads website similar to Craigslist on the surface web Although the market listing hosted everything from apartments to escorts, a report by Fichtner (2016) suggested that 90% of Backpage's revenue came from adult ads. Another report by Callanan et al. (2017) suggests that Backpage hosted escort listings concerned with the trafficking opera- tions of women and children across 943 locations, 97 countries, and 17 languages. In this research, we accumulated 513,705 advertisements spread across 14 states and 41 cities in the United States.
Authorship Attribution in NLP: In the past, re- search has established many machine-learning ap- proaches to analyze text styles and link distinctive writing characteristics to specific authors. These approaches encompass TF-IDF- clustering and classification techniques (Agarwal et al., 2019; izzet Bozkurt et al., 2007), conventional convo- lutional neural networks (CNNs) (Rhodes, 2015; Shrestha et al., 2017), recurrent neural networks (RNNs) (Zhao et al., 2018; Jafariakinabad al., 2019; Gupta et al., 2019), and contextualized trans- formers (Fabien et al., 2020a; Ordoñez et al., 2020; Uchendu et al., 2020; Barlas and Stamatatos, 2021). Moreover, researchers have recently demonstrated the effectiveness of contrastive learning approaches (Gao al., 2022) for authorship tasks (Rivera-Soto et al., 2021; Ai et al., 2022). These advancements have led to applications in style representational approaches (Hay et al., 2020; Zhu and Jurgens, 2021; Wegmann et al., 2022), which currently rep- resent the state-of-the-art (SOTA) for authorship tasks. Consequently, several datasets (Conneau and Kiela, 2018; Andrews and Bishop, 2019; Beven- dorff et al., 2020, 2023) have been established to facilitate further research in this area.
600
500
2000
2000
1000
100
(K SENTLLE
CHARLEN
(S)AD
Figure 2: (A) Total number of tokens per ad (sent-len). (B) Total number of characters per ad, and (C) Number of ads per vendor (class-frequent distributions.
Authorship Attribution and Cybercrime: Nu- merous authorship attribution studies have been successfully applied to the fields of forensic (Yang and Chow, 2014; Johansson and Ishister, 2019; Belvisi et al., 2020) and cybercrime investigations (Zheng et al., 2003; Rashid et al., 2013), spam de-
'While we perform the pre-processing and restructuring of data for the authorship task, we would like to acknowledge Bashpole Software, Inc. for sharing the scraped data with us. To get access to our data, please visit our GitHub repository and follow the instructions provided in the link.
3

---

Preprocessing: First, we begin by merging the ti- le and description of the text ads using the "[SEP]" token, as illustrated in Figure I[i]. Figure 2(A) and figure 2(B) show that most ads our dataset (approximately 99%) have a sentence length be- low 512 tokens and 2,000 characters. To generate ground truth, i.e., vendor labels, we employ the TJBatchExxtactor Nagpal et al. (2017b) and CNN- LSTM-CRF classifier Chambers et al. (2019) to extract phone numbers from the ads. Subsequently, we utilize NetworkX Hagherg al. (2008) ere- atc vendor communitics based on these phone num- bers. Each community is assigned a label ID, which forms the vendor labels. For evaluation purposes, ads without phone numbers are discarded, resulting in remaining dataset of 202,439 ads. Following the findings of Lee et al. (2021), which indicate that the average vendor of escort ads has 4-6 victims, we remove entries from vendors with fewer than five (average of 4-6) ads. The overall outcome of this process a dataset comprising 87,595 unique ads and 5,244 vendor labels. Most of these vendors have an ad frequency of under 1,000 4
Differences between Existing Authorship and IDTrafficcerss dataset: To understand the dif- ferences between the existing authorship dataset and the IDTraffickers, we examine the part- of-speech (POS) and wikification (Szymanski and Naruszewicz, 2019) distributions between IDTraffickers. PAN2023 (Bevendorff et al., 2023), and the Reddit-Convvrsatien (Wegmann al., 2022). The POS distribu- spacy-
dataset
tion is parsed through the RoBERTa- transformers tagger (Montani al., 2020), whereas the wikification is carried out using the Amazon ReFinED entity linker (Ayoola et al., 2022).
0.
0.2
Geography
Advertisements
Vendors
East
24,000
5,029
Ree!
12000
1906
oniic
West
22,556
2,576
North
3.124
254
South
27,871
2,291
Figure 3: POS- distriittibt distribution for IDtraffickers.
Normalized POS- PAN2023 and
Central 21,124
2,928
Overall 87,595
5,244
Reddii-Conversations datasets.
Table 1: Total number of unique advertisements and vendors across US geography
Figure 3 presents a comparative analysis of the POS (part-of-speech) distributions among three datasets. The results reveal that the IDtraffickers dataset hibits a higher frequency of punctuations, emojis, white spaces, proper nouns, and numbers than the other datasets. The punctuations, emojis, white spaces, and random characters represent approx- imately 47% of all POS tags in the IDtraffickers dataset. In contrast, these tags only account for 10.6% and 12.4% of all tags in the PAN2023 and Reddit conversation datasets, respectively. This dis- crepancy sheds light on the substantial noise within our dataset, highlighting the need for fine-tuning for domain adaptation.
After generating the vendor labels, we took mea- sures to safeguard privacy by masking sensitive in- formation within the ad descriptions. This included masking phone numbers, email addresses, age de- tails, post ids, dates, and links to ensure that none of these details could be ceverse-engineered thereby minimizing the potential misuse of our dataset. De- spite our efforts to extract escort names and lo- cation information using BERT-NER, RoBERTa- NER- entity recognition techniques, and the approach described by (Li et al., 2022), we encoun- tered a significant number of false positives. Un- fortunately, our attempts to mask this information resulted in further noise in our data. Consequently, we decided to forgo this approach.
In addition to examining POS distributions, we also investigate the wikifiability, the presence of entities with corresponding Wikipedia men- tions, on a per-advertissmeet basis. Figure 4 pro- vides insights into the wikifiability across three datasets: IDTraffickers, PAN2023, and Reddit Con-
'For a more comprchensive understanding of our dataset, we encourage readers to find a detailed explanation in the datashect attached in appendix A.4.
4

---

versational. Notably, the IDTraffickers dataset ex- hibits higher level of wikifiability compared to the PAN2023 and Reddit Conversational datasets. However, a closer examination in figure 5 reveals that the majority of recognized entities in the ID- Traffickers dataset are primarily related to locations, escort names, or organizations. This observation aligns with the nature of the ads, they often in- clude information such as the posting's location, the escort's ' name, and nearby landmarks.
perform traditional stylometric approaches, statis- tical TF-IDF, and conventional RNNs and CNNs on authorship tasks (Kumar et al., 2020; Fabien et al., 2020b; Ai et al., 2022; Saxena et al., 2023). Hence, we establish our baselines through closed- set classification experiments using the distilled ver- sions of BERT-cased (Sanh et al., 2020), RoBERTa- base (Liu et al., 2019), and GPT2 (Li et al., 2021), smaller versions of, AIBERTa (Lan al., 2020) and DeBERTa-v3 (He et al., 2023), and architectures trained on contrastive objective such as MiniLM (Wang et al., 2020) and DeCLUTR (Giorgi et al.. 2021). To account for domain differences, we also fine-tuned a RoBERTa-base language model (LM) on the IDTraffickers ads for the language task. Then, we extract the sentence representations from trained LM and employ mean-pooling for the closed-set classification task 5, In our re- search, we refer this model the LM-Classffer. Finally, we evaluated our results against a clas- sifier trained on style representations (Wegmann et al., 2022), which currently represents the state-of- the-art (SOTA) in authorship tasks. We employed several metrics for evaluation, including balanced accuracy, micro-F1, weighted-F1, and macro-F1 scores. However, we emphasize the performance of our classifiers on the macro-Fi score due the class imbalance (Figure 2(C)) in our dataset
0.2
Figure 4: Wikifiability: No. of entities per advertise- ment with Wikipedia mentions in the IDtraffickers. PAN2023 and Reddit-Conversatiin datasets.
4.2 Authorship Verification: Ranking Task
The closed-set classifier effectively identifies known vendors presented to it during training. However, it cannot handle inference for unknown vendors. Given the daily frequency of escort ads, it is impractical for law enforcement agencies (LEAs) to repeatedly train network whenever new vendor emerges. To address this limitation. leverage the trained classifier to extract mean- pooled style representations from the ads and uti- lize FAISS (Johnson al., 2019) for similarity search. Specifically, we employ k-means clustering on the style representations. Our test set ads serve as query documents, while the training set ads act as index documents. By employing cosine similar- ity, we identify the K closest index documents for each query document. Since treat the author-
0.8
OPg
OMG
PoWizN
Evenyt
Home ce
Figure 5: Wikkientities-distribbuiii Extracted entities from the wikification of IDtraffickers, PAN2023 and Reddit- Condes-Convessationn datasets.
4
Experimental Setup
'Please note that we experiment with mean, max, and mean max pooling strategies for both author verification and identification tasks. However, the best results are obtained using the mean-pooling strategy
4.1 Authorship Identification: A Classification Task
Researchers have consistently demonstrated that the transformers-bate based contextualized models out-
"The training setup and hyperparameter details are de- scribed in the appendix section A.I.
5

---

ship verification task as a ranking task, we evaluate the effectiveness of this similarity search operation using Precision @ K, @ecall@ K, Mean Average Precision (MAP@ K) (Pothula and Dhavachelvan, 2011; Jin et al., 2021), and average R-Precision scores (Beitzel et al., 2009) metrics.
similar content types, the model struggles to ef- fectively adapt the noise in our data and distin- guish between different writing styles. In contrast, DeCLUTR, trained to generate universal sentence representations, excels at capturing stylometric pat- terns and associating them with individual vendors. Next, we train the LM until convergence to achieve perplexity of 6.22. When compared to the end-to- end DeCLUTR baseline, training classifier over the trained representations of our LM only provides us with % increase in performance. We believe that the higher perplexity in the LM illustrates its inability to adapt to high noise in our data. Consid- ering this minor improvement, we conclude that the additional training for our LM is not worthwhile. Consequently, we establish the DeCLUTR archi- tecture as the benchmark for vendor identification on the IDTraffickers dataset.
5 Results
5.1
Authorship Identification Task
Table 3 showease the performance of trained our classifier baselines, evaluated on the test IDTrafff- iekers dataset.
Models
Acc.
Micro-F1 Distilled Models
Weighted-F1
Macro-F1
0.9110
0.9147
0.9143
0.8467
RoBERTs
0.9199
0,9230
0,9229
0.8603
GPT2
0.9132
0.9172
0.9166
0,8500
Smaller Models
ALBERT
0.7832
0.7891
0,7925
0.6596
D
0.8703
0.8757
0,8756
0,7825
5.2 Authorship Verification Task
T5
0.9157
0.9192
0.9190
0.8535
Contrastive Learning Models
Table 2 presents the open-set author verification
miniLM
0.8888
0.8934
0.8935
0.8101
results for the DeCLUTR and Style-Fmbedding models before and after being trained on the ID- Traffickers dataset. In this experiment, we conduct a similarity search on the training dataset for each query in the test dataset. The results show the av- erage performance across all queries, along with the standard deviation. As can be observed, sub- stantial performance difference exists between the models before and after training, emphasizing the importance of our dataset. The zero-shot perfor- mance of the available checkpoints red proves inconclusive for the author verification task.
DeCLUTR
0.9230
0.9261
0.8656
Style-Emb
0.8887
0.8936
0.8932
0.8112
HT Language Model
LM-Classifier
0.9294
0.9317
0,9316
0.8726
Table 3: Balanced Accuracy, Micro-F1, Weighted-Fl, and Macro-Fl performances of the transformers-base basse classifiers on the author identification task.
As illustrated, the DeCLUTR small classifier out- performs all other baselines for the authorship iden- tification task. The success of the Style-Embedding model comes from its ability to latch onto con- tent correlations. However, since escort ads have
The Precision@ metric measures the model's 's
@1
110
20 Precision& K
@500
Style
0,0442
0.20
0,0410
-
0.16
0.0391
+ 0.15
0.966 + 0.13
0.0299 +.11
0.019- - 0.10
0.0270 - 0.08
0.0227 + 0.07
DECLUTTR
0.3198
0.46
0,2883
0.39
0.2671
0.36
0.2278
0.32
0.1837 0.27
0.1693
0.26
0.1277
0.21
0.0893
0.15
Styll
0.9616 1 0.19
0.9437
0.19
0.9124
1 0.21
0,8175 1 0.27
0.6818 1 0.33
0.6328
- 0.35
0,4815 a 0.36
0.3551
1 0.26
DeCLUTR
0.17
0.36
0.36
R
Style
0,0023
0.01
0.0063
0.04
0.0091
0.05
0.0146
0.07
0.0233
0.09
0.0269 = 0.10
0.0394 1 0.12
0.0580 1. 0.15
DeCLUTR
0.242
0.06
0.0567
0.12
0.0792
0.16
0.1136
0.20
0.1539
0.24
0.1676
0.25
0.2122
0.29
0,2590 1 0.31
Style
0,0828
0.09
0.2348
0.24
0.3485
0.32
0.5092
1. 0.37
0.6522
1: 0.37
0.6945
= 0.36
0.7909
: 0.32
0.8600
4: 0.27
DeCLUTT
0.8366
0.09
0.2397
0.25
0.3563
0.32
0.5192
0.37
0.6653
0.37
0.7041 0.36
0.7988
0.32
0,8064
0.27
MAP@к
Style
0.21
0.0700
0.21
0.0712 + 0.21
DeCLUTT
0.3198 + 0.46
0.3687
0.45
0.3681
0.45
0.3750
0.44
0.3794 + 0.44
0.3803 - 0.44
0.3823 + 0.44
0,3833 + 0.44
Style
0.9616 : 0.19
0.9687
0.16
0,9698
0.15
0.9706
1 0.15
0.779 1 0.14
0.9700 = 0.14
0.9710 : 0.14
0.9710 4 0.14
DeCLUTR
0.11
0.9735
0.14
0.9746
0.14
0:9552
0.13
0.9755
0.13
0.9755
0.13
0.9756
0.13
0.9736 1 0,13
R-Precision X
Style
0.0199 0.07
DeCLUTR
0.1641 1. 0.23
Style
0.22
DeCLUTR
0.8850 1. 0.20
Table 2: Precision@ @ K, Recall@ K, MAP@ K, and R-Precision @ X scores for the DeCLUTR and Style-Emme models before and after being trained on the IDTraffickers dataset
6

---

ability to identify relevant vendor ads within the top-K predictions, whereas Recall K quantifies its capability to identify relevant recommended ads from the entire set of relevant ads in our training dataset. High precision at smaller K values shows that our trained model retrieves relevant ads effec- tively within the top-K retrieved items. While not all vendors in our dataset have similar number of ads, we know they all have least five ads. There- fore, we focus on precision performance for K values less than or equal 5. On the other hand, increasing Recall with higher values indicates that our trained model retrieves a large proportion of relevant ads for a query from the entire set of available relevant ads. Given that all vendors in our dataset have at least five ads each, we emphasize the recall performance of our trained model for K values greater than equal to In addition to Precision @ K and Recall@ K, we also evaluate the performance of our trained models using MAP@ K, which considers the ordering of the retrieved items. It calculates the average precision across all rele- vant items within the top- K positions, considering the precision at each position. As can be observed, our retrieval system prioritizes ads from the same vendor when presented with query advertisement from vendor A. The high MAP scores achieved by our trained models for all K values indicate that the retrieved ads are effectively ranked, with the most relevant ones appearing at the top of the list.
Legend:
il NegativeL Neatal ill Postive
Predicted Lokel Labd
True
Attiluttton Label
Arrihuation Store
Word Impoctance
MDd poo Chicago
INK
Come and
of
LABEL_
LABEL_
this good good 1 Allx 18S ENN BNN
XN Toster 4ge AGE apt Lasation: reee 1111, Chicagg Northuo ty ape luag to friesd
POST
La Posilive
Tour
Labd
Arttbution
Emspartenee
As bey h @ on geg) il Carme and some this good good Allex
XS KINNN NEIN NN Posi ons sgge
LABEL_ (1.00)
LAINEL G
3J0
AGI Lacotton
ever 10111,
Chicago SE PI I
mmmn gasdl Chicago
INK
(a) True Positives model attributions for Vendor 6
Legend-ill Chaeaee Cinnsill FINANe
Anviburinn
Arribattie
NEMO Jeparno 20 w p WONN -NN0--NN
(8.0))
LAMEE
233 27th- Call mereevee
Legend: Negutive E Neatil Prsttive
Thir Libd
Predined Attibuttion Label Label
Artribution Score
Ward Impartance
PE
Jepareme 722
Hot (707777777777
4318
A810 (0.65)
LABEL_4310
1.85
7232727 2232 22??
E. Noaret 71 V ZEN NINN
BINN IN 2 Wes chester esc
(b) False Positives model attributions for Vendor 742
Figure 6: Model Explanations from the trained
DeCLUTR-smie small classifier
Finally, we evaluate our trained models using the R- R-Precision metric, which focuses on precision when the number of retrieved ads matches the num- ber of relevant ads (X) for a vendor. This metric disregards irrelevant items and solely concentrates on accurately retrieving relevant ones. The results show that the DeCLUTR-small models achieve an approximate 88% precision for retrieving relevant ads among the top-ranked items. In other words, our vendor verification setup effectively captures and presents the most relevant ads for given query, achieving high precision at the rank equal to the number of relevant ads. Consequently, we establish the DeCLUTR-sINE small model as the benchmark for the vendor verification task on the IDTraffickers dataset. However, there is significant fluctuation in standard deviation, indicating that precision varies depending on the vendor and ad frequency. Fur- ther investigation confirms this observation, with lower Precision @ K, Recall@ K, MAP@ K, and R- R-Precision @ X values for vendors with fewer ads.
5.3 Qualitative Analysis
To generate interpretable results, transformers-interpreet (Pierse, 2021) build upon Captum (Kokhlikyan et al., 2020) to compute local word attributions in our text ads using the DeCLUTR-small classifier benchmark, indicating the contribution of each word to its respective vendor prediction. Figures 6a and 6b showcase True Positive and False Positive predictions from our trained classifier. Furthermore, for each query in the test dataset, employ FAISS to identify the most similar ad in the training dataset. The figures depict positive (in green), zero, and negative (in red) word attribution scores for queries and their corresponding anchors associated with their vendor label.
use
Similar writing patterns and word attributions in the True Positive explanations confirm that the ads
7

---

are associated with the same vendor. This inference is supported by the consistent use of " @" and "/" preceding the digits of the masked phone number "/NN N / NN N/ NN NN." Conversely, the False Positive explanations shed light on instances where the model generated incorrect predictions, likely due to significant content and writing style similar- ities between vendors 4310 and 742. Both vendors frequently used continuous sequence of "?" and mentioned Japanese services in their ads. Further examination reveals several instances where the classifier predicted the wrong vendor classes due to similar strong resemblances between the ads. This finding emphasizes the importance of care- fully evaluating the quality of our classification labels. Note that vendor labels are established us- ing the extracted phone numbers mentioned in the ads, which enabled us connect ads and form ven- dor communities. In some cases, ads mentioned multiple phone numbers, aiding us forming these connections. For others, the absence of this infor- mation led us to create a new vendor label. The hallucinations observed in our explanations result from this label assignment process, indicating the possibility of two vendors being the same entity.
tag associated with the respective vendor The visualization clearly illustrates that both vendors employ distinct grammatical structures and word at- tributions in their advertisements, suggesting they are separate entities. This analysis enables law enforcement agencies (LEAs) to enhance their un- derstanding of the connections between multiple vendors without solely relying on the ground truth.
Conclusion
In this research, we attempt to bridge the gap be- tween connecting escort ads to potential HT ven- dors by introducing IDTraffickers, an authorship attribution dataset collected from the Backpage es- cort market advertisements (ads) within the United States geography. The dataset contains 87,595 ads with a text sequence of title and description and 5,244 vendor labels. Since these ads lack ground truth for the authorship task, we generate the labels by connecting the phone numbers. First, we estab- lish a benchmark for the authorship identification task by training a DeCLUTR-small classifier with macro-FI of O. 0,8656 in a closed-set classification environment. Then, we utilize the style represen- tations from the trained classifier to perform an open-set ranking task and establish a benchmark with an r-precision score of 0.8852 for the author- ship verification task. Our experiments reveal a massive difference between the language in ID- Traffickers and existing authorship datasets. By performing the authorship identification task, we allow our classifier to adapt and benefit from this domain knowledge for the authorship verification task. Furthermore, we utilize the local and global feature attribution techniques to perform qualita- tive analysis on the trained classifier. Finally, our analysis reveals that most misclassifications in the trained classifier occur due to the possibility of mul- tiple labels attributing to the same vendor. How- ever, despite the lower performance, the classifier succeeds in generating style representations that allow us to identify these vendors through the rank- ing task. We believe that the availability of our dataset, benchmarks, and analyses will empower future researchers and LEAs to utilize the findings, aiding in the effective linkage of escort ads and developing more robust approaches to identifying HT indicators.
1993
mare
Iway
Figure 7: Word attribution collected over POS- distribution for ads of vendor 11178 and 11189
Inspired by Rethmeier et al. (2020), figure 7 em- ploys global feature attributions to examine the discrepancies in writing styles between two spe- cific vendors, namely vendors 11778 and 11189. The analysis involves collecting word attributions and part-of-speech (POS) tags for all the adver- tisements from both vendors. The resulting bar plot presents the normalized POS density activated within the vendor ads and scatter points highlight- ing the two most attributed tokens for each POS
Please note that we generate the plots using Plotly, which offers infinite zooming capubilities for any number of seatter points. However, we only display the two most attributed tokens for better clarity and visibility in the paper.
8

---

7 Limitations
fickers dataset, and applying these to their specific data context.
Assumption: This rescarch relies upon the clas- sification task to adapt and benefit from domain knowledge. We assume each class label repre- sents different vendor in the classification process. However, our qualitative analysis reveals misclassi- fication by the trained classifier due to heavy resem- blance in writing style and content, indicating the possibility of multiple vendors being the same en- tity. While we cannot establish an absolute ground truth to validate our hypothesis, this presents sig- nificant challenge. Nevertheless, we acknowledge that training a classifier with better-quality ven- dor labels would enhance the performance of our benchmarks.
Larger Architectures: Due to limited compu- tational resources, conducted our experiment using distilled and small transformers-bart ar- chitecture. is worth noting that training the model on larger architectures has the potential to improve overall performance. Additionally, this research utilizes pre-trained representations to initialize our classifier architectures. However, incorporating data
supervised contrastive finetuning enhance the generation of stylometric representa- tions, leading better performance. Therefore, in the future, we plan to introduce supervised con- trastive pre-training baseline into our experiments.
Dataset's Age: Despite several active online es- cort platforms like Craiglist, YesBackpage, and Classified ads, we focus our research on Backpage ads spanning December 2015 to April 2016. This choice stems from documented human HT activi- ties on the Backpage website (Callanan et al., 2017). We opted not to incorporate other active websites due to the absence of HT information in data from other escort websites. Please note that research does not aim to track assist currently ongoing human trafficking investigations. Instead, we aim to showcase the potential of authorship attribution techniques to aid law enforcement agencies in re- source prioritization. Given this research objective, the age of the dataset holds smaller significance.
Zero-Shot Performance: While we ensure not to use ads from the training dataset as queries for the ranking task, is worth noting that the clas- sifier was trained on the same dataset for the au- thorship identification task. To gain insights into the zero-shot capabilities of our trained representa- tions, we intend evaluate the authorship verifica- tion benchmark unseen data. In order to achieve this, we plan expand our data collection efforts, potentially sourcing data from other escort markets. This expansion will allow us to assess whether our model can generate stylometric representations that are universally applicable.
Scalability: In essence, different people write differently, and since the authorship task relics on identifying individual writing styles and perpetra- tors of human trafficking spans all racial, ethnic, and gender demographics (UNODC, 2023; Polaris, 2023); generalizing such approaches to vendors' existing years apart is not straightforward. Re- garding using our approach on other escort web- sites, our methodology employs text ads iden- tify and establish connections among vendors as- sociated with online escort ads. While the funda- mental principles of our research can be theoreti- cally extended to any active online escort website, the generalization capability of our approaches de- pends upon variations in writing styles, the use of chat-GPT-llik automated systems, the presence of different languages, and data noise. In such in- stances, we recommend our readers improve the trained model through further fine-tuning, leverag- ing transfer learning techniques from the IDTraf-
This research employs local and global feature attribution techniques for qualita- tive analysis. However, it is important to address the limitations of these approaches (Das and Rad, 2020; Krishna et al.. 2022). Local feature attribu- tion techniques
Explainability:
susceptible adversarial at- tacks and network sparsity. Moreover, they lack consideration for the broader context and depen- dencies the data, and their explanations may not be entirely intuitive. Similarly, global feature attri- bution techniques suffer from a lack of granularity and contextual information. Both methods exhibit disagreements and significant inconsistencies in their explanations when applied to different XAI frameworks (Saxena et al., 2023). Therefore, in the future, we plan to develop more dependable explainability approaches that can help us better understand model behavior and ultimately foster trust amongst LEAs.
9

---

aim to only 8 Broader Impact
of these advertisements exist after the seizure of Backpage Escort Markets in 2016, we find it un- likely that anyone can exploit the personal data in our advertisements to harm these individuals. Furthermore, given the legitimate interest of our research, our masking methodology aligns with the spirit of the General Data Protection Regulation (GDPR) article 6, the the lawfulness of processing, which acknowledges the importance of balancing privacy concerns with societal benefits. In this con- text, our research contributes to the betterment of socicty by facilitating studies aimed at addressing critical issues such as human trafficking and en- hancing law enforcement efforts to reduce harm and save lives.
Data Protocols: We collected our dataset from the Backpage Escort Markets, posted between De- cember 2015 and April 2016 across various loca- tions in the United States. In a related work (Krotov and Silva, 2018), the ethical implications of web scraping are characterized using seven guidelines. Adhering to these guidelines, we confirm that no explicit prohibitions are stated in terms of use pol- icy on the Backpage website against data scrap- ing. Furthermore, we intend to make our dataset available through the Dataverse data repository to mitigate any potentially illegal fraudulent use of our dataset. The access the data will be sub- ject to specific conditions, including the require- ment for researchers to sign non -on-disclosure agrec- ment (NDA) and data protection agreements. These agreements will prohibit sharing the data and its use for unethical commercial purposes. Consider- ing that the data was collected before the seizure of the Backpage escort markets, we confident that it does not pose any substantial harm to the website or its web server.
Finally, intend to grant access the ID- Traffickers dataset (link attached in our GitHub repository) via the Dataverse data portal. How- ever, this access is subject to stringent restrictions. Researchers seeking access must undergo an appli- cation process, necessitating approvals and assur- ances. These assurances encompass non-disclosure and data protection agreements, ensuring responsi- ble and secure and release of the data.
Privacy Considerations and Potential Risks:
Legal Impact: We cannot predict the specific impact of our research on the law enforcement pro- cess. Through this research, sist Law Enforcement Agencies (LEAs) in compre- hending vendor connections in online escort mar- kets. Hence, we strongly recommend that LEAs and researchers not solely depend on our analy- sis as evidence for criminal prosecution. Instead, they should view our findings as tools to aid their investigations, not as direct evidence.
While we acknowledge the potential privacy con- cerns associated with the information within escort advertisements, we have extensive measures mit- igate these risks. These precautions include mask- ing personal information, including phone numbers, email addresses, age details, post IDs, dates, and links. This masking is executed to prevent any pos- sibility of reverse- eengeeeenggnn,,iingg thus mitigating the potential for misuse of our dataset. Unlike other encryption techniques, our masking process elimi- nates personal information. For instance, email address like xzy @ gmail.com is transformed into <EMAILID-221 phone number such +00 00 0000 000 becomes +NN NN NNNN NNN, and link like www.google.c comegleeccmmii. com is rendered as <LINK> within our dataset. Furthermore, as explained in the preprocessing section 3, we also experiment with various entity recognition techniques to try masking escort names and the posted locations mentioned in the advertisements. However, due to noise in our data, encountered challenges in accurately masking these segments, resulting in false positive entity predictions. Nevertheless, con- sidering that previous research indicates escorts of- ten use pscudonyms in their advertisements (Carter et al., 2021; (ago-Graulich) and no public records
Environmental Impact: We conducted all our experiments on private infrastructure, 100-SXM2- 32GB (TDP of 300W), with carbon efficiency of 0.432 KgCO2eq/kWh. The training process for es- tablishing all the baselines in our research took a cumulative 191 hours. Based on estimations us- ing the Machine Learning Impact calculator pre- sented in (Lacoste et al., 2019), the total estimated emissions for these experiments amount to 24.62 kgCOzeq.
Acknowledgement
This research is supported by the Sector Plan Dig- ital Legal Studies of the Dutch Ministry of Edu- cation, Culture, and Science, and Bashpole Soft- wares, Inc. Finally, the experiments were made
10

---

possible using the Data Science Research Infras- tructure (DSRI) hosted Maastricht University,
Nacer Eddine Benzebouchi, Nabiha Azizi, Monther Ald- wairi, and Nadir Farah. 2018. Multi classiffer system for authorship verification task using word embed- dings. In 2018 2nd International Conference on Nat- ural Language and Speech Processing (ICNLSP). pages 1-6. IEEE.
References
Ahmed Abbasi, Abdul Rehman Javed, Farkhund lqbal, Zunera Jalil, Thippa Reddy Gadekallu, and Natalia Kryvinska. 2022. Authorship identification using ensemble learning. Scientific reports, 12(1):9537.
Nacer Eddine Benzebouchi, Nabiha Azizi, Nacer Ed- dine Hammami, Didier Schwab, Mohammed Chi- heb Eddine Khelaifia, and Monther Aldwairi. 2019, Authors' writing styles based authorship identifica- tion system using the text representation vector. In 2019 16th International Multi-Conference on Sys- tems, Signals & Devices (SSD), pages 371-376. IEEE.
Lucky Agarwal, Kartik Thakral, Gaurav Bhatt, and Ankush Mittal. 2019. Authorship clustering using if-idf weighted word-embeddings. In Proceedings of the 1 11th Forum for Information Retrieval Evalu- ation, FIRE' 19, page 24-29, New York, NY, USA. Association for Computing Machinery.
Janek Bevendorff, Mara Chinea-Rios, Mare Franco- Salvador, Annina Heini, Erik Körner, Krzysztof Kredens, Maximilian Mayerl, Piotr Pundefinedzik, Martin Potthast, Francisco Rangel, Paolo Rosso. Efstathios Stamatatos, Benno Stein, Matti Wieg- mann, Magdalena Wolska, and Eva Zangerle. 2023. Overview of pan 2023: Authorship verification, multi-author writing style analysis, profiling cryp- tocurrency influencers, and trigger detection: Ex- tended abstract. In Advances Information Re- trieval: 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2-6, 2023, Proceedings, Part III. page 518-526, Berlin, Heidelberg.
Bo Ai. Yuchen Wang, Yugin Tan, and Samson Tan. 2022. Whodunit? learning to contrast for authorship attribution.
Mamoun Alazab, Robert Layton, Roderic Broadhurst, and Brigitte Bouhours, 2013. Malicious spam emails developments and authorship attribution. pages 58- 68.
Hamidreza Alvari, Paulo Shakarian, and J. E. Kelly Snyder. 2017. Semi-supervised learning for detecting human trafficking. Security Informatics, 6(1):1.
Janek Bevendorff, Bilal Ghanem, Anastasia Giachanou, Mike Kestemont, Enrique Manjavacas, Martin Pot- thast, Francisco Rangel, Paolo Rosso, Günther Specht, Efstathios Stamatatos, Benno Stein, Matti Wiegmann, and Eva Zangerle. 2020. Shared tasks authorship analysis pan 2020. In Advances in In- formation Retrieval, pages 508-516, Cham. Springer International Publishing.
Hamidreza Alvari, Paulo Shakarian, and J.E. Kelly Sny- der. 2016. A non- non-parametric learning approach to identify online human trafficking. In 2016 IEEE Con- ference on Intelligence and Security Informaties (ISI). pages 133-13 138.
Nicholas Andrews and Marcus Bishop. 2019, Learning invariant representations of social media users. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLL pages 1684- 1695, Hong Kong, China. Association for Computa- tional Linguistics.
Lukas Biewald. 2020. Experiment tracking with weights and biases. Software available from wandb.com.
Brian Callanan, Matt Owen, Mark Angehr, Andrew Polesovsky, Rachael Tucker, Philip Alito, Joshua Debold, Margaret Daum Will Dargusch, Brandon Reavis, Kelsey Stroud, and Adam Henderson. 2017. Backpage.com' knowing facilation of online sex trafficking. United States Senate.
Tom Ayoola, Shubhi Tyagi. Joseph Fisher, Christos Christodoulopoulos, and Andrea Pierleoni. 2022. Re- FinED: An efficient zero-shot-capabe approach to end-to-end entity linking. In NAACL.
Pelham Carter, Matt Gee, Hollie Mcllhone, Harkeeret Lally, and Robert Lawson. 2021. Comparing manual and computational approaches to theme identifica- tion in online forums: A case study of a work special interest community. Methods in Psychology. 5: 10005.
Georgios Barlas and Efstathios Stamatatos, 2021. A transfer learning approach to cross-domain author- ship attribution. Evolving Systems, 123)6625-6443
Steven M. Beitzel, Eric C. Jensen, and Ophir Frieder. 2009. Average R-Prrcision, pages 195-195. Springer US, Boston, MA.
Nathanael Chambers, Timothy Forman, Catherine Gris- wold, Kevin Lu, Yogaish Khastgir, and Stephen Steckler. 2019. Character-basse based models for adversar- ial phone extraction: Preventing human sex traffick- ing. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019). pages 48-56, Hong Kong, China, Association for Computational Linguistics.
Nicole Mariah Sharon Belvisi, Naveed Muhammad, and Fernando Monso-Fermandez. 2020. Forensic author- ship analysis of microblogging texts using n n-grams and stylometric features.

---

Alexis Conneau and Douwe Kiela. 2018. Senteval: An evaluation toolkit for universal sentence representa- tions. arXiv preprint arXiv: 8888000549,
Aric A. Hagberg, Daniel Schult, and Pieter J. Swart. 2008. Exploring network structure, dynamics, and function using networkx. In Proceedings of the 7th Python in Science Conference, pages 11 15, Pasadena, CA USA.
Arun Das and Paul Rad. 2020. Opportunities and chal- lenges in explainable artificial intelligence (xai): A survey.
Oren Halvani, Christian Winter, and Anika Pflug. 2016. Authorship verification for different languages, gen- and topics. Digital Investigation, 16:S33-S43,
Geri L. Dimas, Renata A. Konrad, Kayse Lee Maass, and Andrew C. Trapp. 2022. Operations research and analytics to combat human trafficking: A sys- tematic review of academic literature. PLOS ONE, 17(8):e0273708.
Julien Hay, Bich-Lien Doan, Fabrice Popincau, and Ouassim Ait Elhara. 2020. Representation learning of writing style. In Proceddings of the Sixth Work- shop on Noisy User-generated Text (W-NUT 2020), pages 232-243, Online. Association for Computa- tional Linguistics.
Artur Dubrawski, Kyle Miller, Matthew Barnes, Benedikt Boecking. and Emily Kennedy. 2015. Leveraging publicly available data to discern patterns of human-trafficking activity. Journal of Human Trafficking, 1(1):65-85. 85.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pre- training with wradient-disentaneee embedding shar- ing.
Anirudh Ekambaranathan. 2018. Using stylometry to track cybercriminals in darknet forums.
Michelle Ibanez and Rich Gazan. 2016. Virtual indica- tors of sex trafficking to identify potential victims in online advertisements. In 2016 IEEF/ACM Interna- tional Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 818-824.
EUROPOL. 2020. The challenges of countering human trafficking in the digital era.
Maël Fabien, Esau Villatoro- Tello, Petr Motlick,, and Shantipriya Parida. 2020a. BertAA BERT fine- tuning for authorship attribution. In Proceedings of the 17th International Conference on Natural Lan- guage Processing (ICON), pages 127-137, Indian Institute of Technology Patna, Patna, India. NI.P As- sociation of India (NLPAI).
Michelle Ibanez and Dan Suthers. 2016. Detecting covert sex trafficking networks in virtual markets. pages 876-879.
Michelle Ibanez and Daniel D. Suthers. 2014. Detec- tion of domestic human trafficking indicators and movement trends using content available on open in- ternet sources. In 2014 47th Hawaii International Conference on System Sciences, pages 1556-1565.
Maël Fabien, Esau Villatoro-Telle Petr Motlickk, and Shantipriya Parida. 2020b. BertAA : BERT fine- tuning for authorship attribution. In Proceedings of the 17th International Conference on Natural Lan- guage Processing (ICON). pages 127-137, Indian Institute of Technology Patna, Patna, India. NLP As- sociation of India (NLPAI).
ILO. 2012. Ilo global estimate of forced labour.
Fereshteh Jafariakinabad, Sansiri Tampradab, and Kien A. Hua. 2019. Syntactic recurrent neural net- work for authorship attribution.
William Falcon and The PyTorch Lightning team, 2019. Py PyTorch Lightning.
Brian Fichtner. 2016. California department of justice report. Superior court of the state of California.
Ruoming Jin, Dong Li, Benjamin Mudrak, Jing Gao, and Zhi Liu. 2021. On estimating recommendation evaluation metrics under sampling.
Tianyu Gao, Xingcheng Yao. and Danqi Chen. 2022. Simcse: Simple contrastive learning of sentence em- beddings.
Fredrik Johansson and Tim Isbister. 2019. Foi cross- domain authorship attribution for criminal investi- gations. In Conference and Labs of the Evaluation Forum.
John Giorgi. Osvald Nitski, Bo Wang, and Gary Bader. 2021. Declutr: Deep contrastive learning for unsu- pervised textual representations.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Trunsactions on Big Data 7(3):535-547.
Rachel M Green and John W Sheppard. 2013. Com- paring frequency-and style-based features for twitter author identification. In The Twenty-Sixth Interna- tional FLAIRS Conference.
Keenan Jones, Jason R.C. Nurse, and Shujun Li. 2022.
Are you robert or roberta? deceiving online author- ship attribution models using neural text generators. Proceedings of the International AAAI Conference on Web and Social Media, 16(1):429-44) -40.
Shriya TP Gupta, Jajati Keshari Sahoo, and Rajen- dra Kumar Roul. 2019. Authorship identification using recurrent neural networks. In Proceedings of the 2019 3rd International Conference on Informa- tion System and Data Mining, ICISDM 2019, page 133-137, New York, NY, USA. Association for Com- puting Machinery.
Mayank Kejriwal and Rahul Kapoor. 2019. Network- theoretic information extraction quality assessment in the human trafficking domain. Applied Network Science, 4(1):44.
12

---

Mayank Kejriwal and Pedro Szekely. 2022. Knowledge graphs for social good: An cutity-centric search en- gine for the human trafficking domain. IEFF Trans- actions on Big Data, 83)):592-606.
generated text: Named-entin recognition for com- bating human trafficking. In Findings of the Asso- ciation for Computational Linguistics: ACI. 2022, pages 2854-2868, Dublin, Ireland. Association for Computational Linguistics.
Burcu B. Keskin, Gregory J. Bott, and Nickolas K. Free- man. 2021. Cracking sex trafficking: Data analysis, pattern recogmition, and path prediction. Production and Operations Management, 30(4):1110-1135.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos 2020.
Kristina Lugo-Graulich. Indicators of sex traffick- ing in online escort ads. https:/ http://wwwwwoooo gov/ soffilest/nij/grantte wttpentts//55553.55
Araya, Siqi Yan, and Orion Captum: A unified and generic model interpretability library for pytorch.
Kristina
and Leah F. Meyer. 2021. Law enforcement guide on indicators of sex trafficking in online escort ads. Justice Research and Statistics Association.
Daniel Kosmas, Thomas C. Sharkey, John E. Mitchell, Kayse Lee Maass, and Lauren Martin, 2022. Inter- dicting restructuring networks with applications in illicit trafficking.
Satyapriya Krishna, Tessa Han, Alex Gu, Javin Pom- bra, Shahin Jabbari, Steven Wu, and Himabindu Lakkaraju. 2022. The disagreement problem in ex- plainable machine learning: practitioner's 's perspec- tive.
Andrei Manolache, Florin Brad, Antonio Barbalau, Radu Tudor Ionescu, and Marius Popescu. 2022. Veridark: A large-scale benchmark for authorship verification on the dark web.
Ines Montani, Matthew Honnibal, Matthew Honni- bal, Sofie Van Landeghem. Adriane Boyd, Hen- ning Peters, Maxim Samsonov, Jim Geovedi. Jim Regan, György Orosz, Paul O'Leary McCann, Søren Lind Kristiansen, Duygu Altinok. Roman, Leander Fiedler, Grégory Howard, Wannaphong Phatthiyaphaiuun, Explosion Bot, Sam Bozek, Mark Amery, Yohci Tamura, Bjöm Böing, Pradeop Ku- mar Tippa, Leif Uwe Vogelsang. Ramanan Balakrish- nan, Vadim Mazaev, GregDubbin, jeannefukumaru, Jens Dahl Mollerhoj. and Avadh Patel. 2020, explo- sion/spaCy: v3. v3.0.Orc: Transformer- maasformer-bbsed pipelines. new training system, project templates, custom mod- els, improved component API, type hints & lots more. Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning. CoRR, abs/1910.09700.
Vlad Krotov and Leiser Silva. 2018. Legality and ethics of web scraping.
Ramnath Kumar, Shweta Yadav, Raminta Daniulaityte, Francois Lamy, Krishnaprasad Thirumarayan, Usha Lokala, and Amit Sheth. 2020. Edarkfind: Unsuper- vised multi-view learning for sybil account detection. In Proceedings of The Web Conference 2020, WWW '20, page 1955-1965, New York, NY. USA. Associa- tion for Computing Machinery.
Chirag Nagpal. Kyle Miller, Benedikt Boecking, and Artur Dubrawski. 2017a. An entity resolution ap- proach to isolate instances human trafficking on- line. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 77-84, Copenhagen, Den- mark. Association for Computational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations.
Meng-Chieh Lee, Catalina Vajiac, Aayushi Kulshrestha, Sacha Levy, Namyong Park, Cara Jones, Reihaneh Rabbany, and Christos Faloutsos. 2021. Infos- human-
Chirag Nagpal, Kyle Miller, Benedikt Boecking, and tur Dubrawski. 2017b. An entity resolution approach to isolate instances of human trafficking online.
hield: Generalizable information- trafficking detection. In 2021 IEEE 37th Inter- national Conference on Data Engineering (ICDE), pages 1116-1127.
Pratheeksha Nair, Yifei Li, Catalina Vajiac, Andreas M. Olligschlaeger, Meng-Chieh Lee, Namyong Park. Duen Horng Chau, Christos Faloutsos, Reihanch Rab- bany, and Chich Lee. 2022. Vispad: Visualization and pattern discovery for fighting human traffick- ing. Companion Proceedings of the Web Conference 2022.
Tianda Li, Yassir El Meshahi, Ivan Kobyzev, Ahmad Rashid, Atif Mahmud, Nithin Anchuri, Habib Haji- molahoseini, Yang Liu, and Mehdi Rezagholizadeh. 2021. A short study on compressing decoder-based language models.
Juanita Ordoñez, Rafael Rivera Soto, and Barry Y. Chen. 2020. Will longformers pan out for authorship verifi- cation? notebook for pan at clef 2020. In CLEF.
Yifei Li, Pratheeksha Nair, Kellin Pelrine, and Reihaneh Rabbany. 2022. Extracting person names from user
13

---

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang, Zachary Vetito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang- Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high- perforrerormeeee deep learning library. In Advances in Neurul Information Process- ing Systems 32. pages 8024-8035. Curran Associates, Inc.
Vageesh Saxena, Nils Rethmeier, Gijs Van Dijck, and Gerasimos Spanakis. 2023, Vendorlink: An nlp ap- proach for identifying & linking vendor migrants & potential aliases on darknet markets.
Saeideh Shahrokh Esfahani, Michael J. Cafarella, Mazi- yar Baran Pouyan, Gregory DeAngelo, Elena Eneva, and Andy E. Fano. 2019. Copeext-speccfic language modeling for human trafficking detection from online advertisements. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 1180- 1184,,14,,,111111111111111 Florence, Italy, Associa- tion for Computational Linguisties.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay. 2011. Scikit-lear: Machine learning in Python. Journal of Machine Learning Research,
Prasha Shrestha, Sebastian Sierra, Fabio González, Manuel Montes, Paolo Rosso, and Thamar Solorio. 2017. Convolutional neural networks for authorship attribution of short texts. In Proceedings of the 15th Conference of the European Chapter the Associa- tion for Computational Linguisties; Volume 2, Short Valencia, Spain. Association
Charles Pierse. 2021. Transformers Interpret.
Papers, pages for Computational Linguistics.
POLARIS. 2018. Human trafficking statistics.
Efstathios Stamatatos, 2016. Authorship verification: a review of recent advances. Research in Computing Science, 123:9-25.
POLARIS. 2020.
Polaris analysis of 2020 data from the national human trafficking hotline.
Polaris. 2023.
Myths, facts, and statistics tyttsfaacts- and- statisstttssics.. (Accessed on 10/15/2023).
Abby Styliamou, Jessica Schreier, Richard Souvenir, and Robert Pless. 2017. Traffickcam: Crowdsourced and computer vision based approaches to fighting sex trafficking. In 2017 IEEF Applied Imagery Puttern Recognition Workshop (AIPR), pages 1-8.
polaris.
Rebecca Portnoff, Danny Huang. Periwinkle Doerfter, Sadia Afroz, and Damon Mccoy, 2017. Backpage and bitcoin: Uncovering human traffickers. pages 1595-1604.
Pedro Szekely, Craig A. Knoblock, Jason Slepicka, Andrew Philpot, Amandeep Singh, Chengye Yin, Dipsy Kapoor, Prem Natarajan, Daniel Marcu, Kevin Knight, David Stallard, Subessware S. Karunamoor- thy, Rajagopal Bojanapalli, Steven Minton, Brian Amanatullah, Todd Hughes, Mike Tamayo, David Flynt, Rachel Artiss, Shih-Fu Chang. Tao Chen, Ger- ald Hiehel, and Lidia Ferreira. 2015. Building and us- ing a knowledge graph to combat human trafficking. In The Semantic Web ISWC 2015, pages 205-221, Cham. Springer International Publishing.
Sujatha Pothula and Dhavachelvan. 2011. Precision k in multilingual information retrieval. International Journal of Computer Applications, 24.
Awais Rashid, Alistair Baron, Paul Rayson, Corinne May-Chahal, Phil Greenwood, and James Walker- dine. 2013. Who am i? analyzing digital personas in cybercrime investigations. Computer, 46(4):54-61.
Julian Szymanski and Maciej Naruszewicz. 2019. Re- view on wikification methods. Al Communications, 32:1-17.
Nils Rethmeier, Vageesh Kumar Saxena, and Isabelle Augenstein. 2020. Tx-ray: Quantifying and explain- ing mmodel-knowlede transfer in (un- iuneeuueee nlp.
Xiao Hui Tai, Kyle Soska, and Nicolas Christin. 2019. Adversarial matching dark net market vendor ac- counts. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '19, page 1871-1880, New York, NY, USA. Association for Computing Machin- ery.
Dylan Rhodes. 2015. Author attribution with cnn's.
Rafael A. Rivera-Soto, Olivia Elizabeth Miano, Juanita Ordonez, Barry Y. Chen, Aleem Khan, Mareus Bishop, and Nicholas Andrews. 2021. Learning uni- versal authorship representations. In Proceedings of the 2021 Conference on Empirical Methods Nat- ural Language Processing, pages 913-919, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
Edmund Tong, Amir Zadeh, Cara Jones, and Louis- Philippe Morency. 2017. Combating human traffick- ing with multimodal deep models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1547-1556, Vancouver, Canada. Association for Computational Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter.
14

---

Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee, 2020. Authorship attribution for neural text gener- ation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 8384-8395, Online. Association for Computational Linguistics.
Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistae, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-the-ee natural language processing. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pages 38 38-45, Online, Association for Computational Linguistics.
UNDOC, 2020. Global report on trafficking in persons.
UNODC. 2023, Glotip 2asppp2022___2212221 2022__eb.pdff https://ww unodc. goodc.org/aae wwwp///owmenn///deeeo.o.oo.. aanttta--ndaaan and--aaddaaaaa anallsssssss/ /lltip//022/GLOTiPPPP 2a1PPP_2222_______ pdf. (Accessed on 10/15/2023).
C. Vajiac. D. Chau, A. Olligschlaeger. R. Macken- zie, P. Nair, M. Lee, Y. Li, N. Park, R. Rabbany, and C. Faloutsos. 2023a. Trafficvis: Visualizing organized activity and spatio-temporal patterns for detecting and labeling human trafficking. IEEE Trunsactions on Visualization & Computer Graphics, 29(01):53-66 62.
Min Yang and K.P. Chow. 2014. Authorship attribution for forensic investigation with thousands of authors. volume 428, pages 339-350.
Chen Zhao, Wei Song. Xianjun Liu, Lizhen Liu, and Xinlei Zhao. 2018. Research on authorship attribu- tion of article fragments via mns. In 2018 IEFE 9th International Conference on Software Engineering and Service Science (ICSESS), pages 159.
Catalina Vajiac, Meng-Chich Lee, Aayushi Kulshrestha, Sacha Lévy, Namyong Park, Andreas M. OI- ligschlaeger, Cara Jones, Reihaneh Rabbany, and Christos Faloutsos. 2023b. Deltashield: Informa- tion theory for human- trafficking detection. ACM Transactions on Knowledge Discovery from Data, 17:1- 27.
Rong Zheng, Yi Qin, Zan Huang, and Hsinchun Chen. 2003. Authorship analysis in cybercrime investiga- tion. In Intelligence and Security Informatics, pages 59-73, Berlin, Heidelherg. Springer Berlin Heidel- berg.
Guido Van Rossum and Fred I. Drake Jr. 1995. Python reference manual, Centrum voor Wiskunde en Infor- matica Amsterdam.
Jian Zhu and David Jurgens. 2021. Idiosyncratic but not arbitrary: Learning idiolects in online registers reveals distinctive yet consistent individual styles. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 279- 297, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. 2022. Open-set recognition: A good closed-set classifier is all you need. International Conference on Learning Representations.
izzet Bozkurt, O. Baghoglu, and Erkan Uyar. 2007. Au- thorship attribution. 2007 22nd international sympo- sium on computer and information sciences, pages 1-5.
Longshaokan Wang, Eric Laber, Yeng Saanchi, and Sherrie Caltagirone. 2019. Sex trafficking detec- tion with ordinal regression neural networks. ArXiv, abs/1908.05434.
A Appendix
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers.
A.1 Infrastructure Schedule
Split Ratio: We perform data splitting into train- ing, validation, and test datasets using a standard ratio of 0.75:0.05:0.20 for our experiments. To en- sure reproducibility, we set the seed parameter to 1111 during this split.
Anna Wegmann, Marijn Schraagen, and Dong Nguyen. 2022. Same author or just same topic? towards content-independent style representations. In Pro- ceedings of the 7th Workshop on Representation Learning for NLP, pages 249-268, Dublin, Ireland. Association for Computational Linguistics.
Training: Our network training and evaluation are conducted on a Tesla V100 GPU with 32 GB of memory. To optimize the networks, we utilize the Adam optimizer with specific parameter con- figurations. The 31 and 32 values are set to 0.9 and 0.999, respectively. Additionally, we include an L2 weight decay of 0.01 in our optimization process. To determine the optimal learning rate, we experiment with values of 0.01, 0.001, and 0.0001. After analyzing the results, we find that the best
Jessica Whitney Murray E. Jennex, Aaron C. Elkins, and Eric G. Frost. 2018. Don't it want to get caught? don't say it: The use of emojis in online human sex trafficking ads. In Hawaii International Conference on System Sciences.
Chawit Wiriyakun and Werasak Kurutach. 2021. Fea- ture selection for human trafficking detection models. In 2021 IEEE/ACIS 20th International Fall Confer- ence on Computer and Information Science (ICIS Fall), pages 131-135.
15

---

train_loss
Val
accuracy
DACLUTT stylaTab
TwO
Atylatro
0148115872 4ESTEDBERT
aryTatas stacttonts 15 diSTE1DERT
MINILM
(I KIBERT
23 MOnSEM
AIBERT
MEXtUM
AIBERT
6.
5.4
0.2
spocs
Figure 8: Training loss
Figure 9: Validation loss
Figure 10; Balanced accuracy
micro-F1
weighted-Fts
Uncllasstreer ResesTRMERRT
Unclassttter
IosEER dist
Un-Clpestter DesERTo ssecttiveeet'
DCCLUTR stalc asses1op73
becLUTe
styledre
DecuuTR styledab discitgerg
MiviLH
D K3BERT
3 MCKSLM
AlBERT
dEStilSERT
MinLUN
D AUBERT discllBERT
66
64
Figure 11: Micro-F1 score
Figure 12: Weighted-F1 score
Figure 13: macro-Fl score
Figure 14: Training loss, validation loss, and performance of trained classifiers on validation dataset.
performance is achieved when the learning rate is set to 0.001. Lastly, we employ a warm-up strategy for the initial 100 steps, followed by a linear decay.
Computational Details: Tables 4 presents de- tails about the number of trainable parameters. training time, and epoch of convergence for all the trained classifiers in our experiments. Please note that while training LM-Classifier on the author identification task takes the least time, training the language model our dataset takes 65 epochs and additional days, 7 hours, and 20 mins.
Architectures & Hyperparameters: Consider- ing the computational resources at our disposal, we initialize our transformer using pre-trained model checkpoints of DistiBBERT-base-aasee DistiRoBERTaa-aas DistilGPT2, all M--- v2, AIBERT-small, DeBERTa-v3-small, T5-small, DeCLUTR-small. and Embedding architec- tures. To perform the classification, we add se- quence classification head on top of each of these architectures. All models are trained until conver- gence, employing a batch size of 32 and a mini- batch of 4. We conducted experiments with lower batch sizes of 8, 16, and 20 but ultimately found that a batch size of 32 yielded the best results. Please note that 32 is the largest batch size we can compute without encountering memory issues during training.
Figure 14 contains other training details, such as training and validation loss. Finally, we present the balanced accuracy, micro-F1, weighted-F1, and macro-Fl scores the validation set for repro- ducibility purposes. We generate these results by exporting Wandb (Biewald, 2020) reports.
Models
Param
Time
Epoch
DistiBBERT
66M
25:44:00
63
82M
11:13:05
27
DistilGPT2
82M
25:45:00
82
all-minil
22M
21:50:04
62
ALBERT-smalI-v2
11.6
16:08:52
44
DeBERTa-v3-small
44M
22:13:50
16
T5-small
35M
19:03:35
50
All the experiments in research are imple- mented in python3.9 ( (Van Rossum and Drake Jr, 1995) using Sklearn (Pedregosa al., 2011), Py- Torch (Paszke et al., 2019), Hugging-face (Wolf et al., 2020), and Lightning2.0 (Falcon and The PoTorc Lightning team, 2019) frameworks.
DeSl.UTR-small
86M
05:53:33
26
Style-Embedding
128M
18:43:06
36
LM-Classifier
86M
04:47:10
19
Table 4: Total number of trainable parameters, training time, and convergence epoch for the trained classifiers.
16

---

Random Initialization: Due limited re- sources, we only examine the effects of different initializations on our model's performance for the established DeCLUTR-smaal benchmark. Table displays the mean and standard deviation in the model's performance against balanced accuracy, Micro-F1, Weighted-F1, and Macro-F1 scores. The results indicate minimal to no effects on these scores across different initializations.
That said, we see drift among vendor loca- tions within our dataset. Around 75 vendors in our dataset post ads between east-west, 6 between east-north, 107 between east-south, 284 between east-central, 7 between west-north, 53 between west-south, 34 between west-central, 7 between
north-south, 8 between north-central, and south- central American geographies. Our analysis fur- ther reveals that 7 vendors post their advertisements across all 41 states of the United States from which our dataset was collected.
Seed
Acc.
Micro-F1
Weighted-F1
Macro-F1
100
0.9256
0.9285
0.9286
0.8694
0.9243
0.9283
0.9286
0.8677
A.4 Datasheet
1000
0.9172
0.9205
0.9204
0.8559
1111
0.9230
0.9261
0,9259
0.8656
A.4.1 Motivation
5000
0.9197
0.9229
0,9229
0.8598
For what purpose was the dataset created? Was there a specific task in mind? Was there a spe- cific gap that needed to be filled? Please pro- vide a description. Law Enforcement Agencies (LEAs) and researchers currently use sex traffick- ing indicators to distinguish between online es- cort advertisements and Human Trafficking (HT) cases. These investigations involve connecting ads to specific individuals or trafficking networks, often by examining phone numbers or email addresses. However, existing methods for linking these es- cort ads have limitations. Therefore, we release IDTraffickers, an authorship attribution dataset, to identify and link potential HT vendors by analyzing and linking distinctive writing characteristics.
Mean
0.9219
0.9252
0.9252
0.8636
Std.
0.0030
0.0031
0,0032
0.0050
Table 5:
Influence of different initialization on DeCLUTR-small classifier's performance
A.2 Language Distribution of Dataset
Our analysis reveals that approximately 99.32% of our dataset's vocabulary is English. Following that, in descending order, we find Spanish, Chinese, Japanese, Sotho, French, Bokmal, Dutch, Tswana, Nynorsk, Swedish, Latin, Basque, Swahili, Malay, Xhosa, and Welsh making up the remaining portion. Consequently, we did not explore the use of mul- tilingual models in our research. Given that only a small fraction of our dataset's vocabulary lies outside English, we anticipate that employing mul- tilingual models would not significantly improve performance. These statistics obtained using the lingua language detection python package.
A.4.2 Composition
What do the instances that comprise the dataset represent (e-g,, documents, photos, people, coun- tries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description. The instances the dataset comprise raw text sequences obtained by merging the title and description the escort ads using the [SEP] token.
A.3 Drift in vendor writing styles
The absence of ground truth poses challenges in collecting ads from authors spanning multiple years comment on the stylometric drift. Our data col- lection specifically covers escort advertisements between December 2015 and April 2016, during which we observed no significant shifts in authors' writing styles. Moreover, given the brevity of this period, it is challenging for us to ascertain the sta- bility of escort vendors. Notably, establishing the business age for potential human trafficking ven- dors is intricate, mainly because not all such ven- dors are apprehended by law enforcement. How- ever, research indicates that more organized traf- fickers exploit victims extensively over extended timeframes (Lugo-Graan Graulich and Meyer, 2021).
How many instances are there in total (of each type, if appropriate)? What data does each in- stance consist of? "Raw" data (e.g., unprocessed text or images)or features? Is there a label or target associated with each instance? In total, collected 87,595 ads connected to 5,244 ven- dors. The dataset is released in a .csv file with "TEXT" and "VENDOR" being the two columns. The "TEXT" column contains the input sequence of string format, whereas the "VENDOR" column contains class labels in int
pandas dataframe
17

---

format.
based on these phone numbers. Each community is assigned label ID, which forms the vendor labels.
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., ge- ographic coverage)? If so, please describe how this representativeness was it is not representative of the larger set, please describe why not (e.g., cover a more diverse range of instances, because instances were with- held or unavailable). The IDTraffickers dataset comprises advertisements from the Backpage es- cort market, spanning locations across 14 states and 41 cities, and the United States. Figure 15 demonstrates the density of unique advertisements collected across American states. Initially, we col- lected total of 513,705 ads. However, filtered out ads that did not include phone numbers, as these phone numbers serve as the ground truth for our dataset. Consequently, our dataset consists of 202,439 ads that meet this criterion.
Are there recommended data splits (e.g., train- ing, developmen//vliiaaton,, testing)? If so, please provide a description of these splits, ex- plaining the rationale behind them. We per- form our data splitting into training, validation, and test datasets using a standard ratio of 0.75:0.05:0.20 for experiments.
If'
Are there any errors, sources of noise, redun- dancies in the dataset? If so, please provide a description. Our analysis in section 5.3 reveals the possibility of two or more vendor labels as- sociated with the same entity. However, in the absence of absolute ground truth, we cannot make any conclusive statements. Figure 3 demonstrates the presence of significant noise within the data, such punctuations, emojis, white spaces, and random characters deliberately inserted by escort vendors to evade automated systems.
Is the dataset self-contained, or does link to or otherwise rely on external resources (e.g., web- sites, tweets, other datasets)? If it links to or relies on external resources, a) are there guar- antees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the ex- ternal resources as they existed at the time the dataset was created); e) are there any restric- tions (e.g., licenses, fees) associated with any of the external resources that might apply to a dataset consumer? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, appropriate No. The dataset is self-contained.
Figure 15: Density of unique advertisements collected acTOSs American states
Is any information missing from individual in- stances? If so, please provide description, ex- plaining why this information is missing (e.g., because it was unavailable). This does not in- clude intentionally removed information, but might include, e-g., redacted text. No
Does the dataset contain data that might be con- sidered confidential (e.g., data that protected by legal privilege or by doctor-patient confiden- tiality, data that includes the content of individ- uals' nonpublic communications)? If so, please provide a description. While we acknowledge the potential privacy concerns associated with the information within escort advertisementss, we have taken measures mitigate these risks. Specifically, we have masked sensitive details such as phone numbers, email addresses, age information, post IDs, dates, and links within the advertisements. All the digits of phone numbers in our ads are replaced
Are relationships between individual instances made explicit (e.g., users' movie ratings, so- cial network links)? If so, please describe how these relationships are made explicit. To gen- crate ground truth labels for authorship task in our dataset, we employ the TJBatchExtractoo Nag- pal et al. (2017b) and CNN-LSTM-CRE classifier Chambers et al. (2019) to extract phone numbers from the ads. Subsequently, we utilize NetworkX Hagberg et al. (2008) to create vendor communities
18

---

by the letter "N." Similarly, we replace all the email addresses with < EMAIL ID >, post IDs with POST_ID NNNNNN where represents a digit of a number, dates with < DATES >, and links with < LINK >. Since our dataset contains only text data, all the escort images are discarded. Furthermore, as explained in preprocessing section 3, we also experiment with various entity recogni- tion techniques to try masking escort names and the posted locations mentioned in the advertisements. However, due to noise in our data, we encountered challenges accurately masking these segments, resulting in false positive entity predictions. Nev- ertheless, considering that previous research indi- cates escorts often use pseudonyms in their ad- vertisements (Carter et al., 2021; Lugo-Graulich) and no public records of these advertisements exist after the seizure of Backpage Escort Markets in 2016, we find it unlikely that anyone can exploit the personal data in our advertisements to harm these individuals.
religious beliefs, political opinions or union memberships, or locations; financial health data; biometric or genetic data; forms of govern- ment identification, such as social security num- bers; criminal history)? If so, please provide a description. Despite our masking measures, our efforts to extract escort names, ads location, ethnic- ities, and sexual orientations failed due to the noise in our data. These details are mentioned in the text description of the ads.
A.4.3 Collection Process
How was the data associated with each instance acquired? Was the data directly observable (e.g,, raw text, movie ratings), reported by sub- jects (e.g,, survey responses), or indirectly in- ferred/derived from other data (e.g., part-of- speech tags, model-based guesses for age lan- guage)? If the data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verifledd? If so, please describe how. To generate vendor labels in our dataset, we employ the TJBatchExtractor Nag- pal et al. (2017b) and CNN-LSTM- classifier Chambers et al. (2019) to extract phone numbers from the ads. Subsequently, we employ NetworkX Hagberg al. (2008) to create vendor communities based on these phone numbers. Each community is assigned a label ID, which forms the vendor labels. Given the large size of our dataset and the abundance of phone numbers in the advertise- ments, manual validation of cach extracted phone number is infeasible. However, we address this challenge by training the CNN-LSTM-CRI classi- fier using five random initializations assess the robustness of the predictions. To evaluate the qual- ity of these predictions, we compare them across all initializations. Our experiments demonstrate high performance with 99.86% Levenshtein accu- racy, 99.82% perfect accuracy, and 99. 99.50% digit accuracy (for further details, please refer (Cham- bers et al., 2019)), indicating strong robustness in the model's predictions.
Does the dataset contain data that, if viewed di- rectly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If' so, please describe why. The dataset comprises text from escort advertisements that contains sexual descrip- tions of the services.
Does the dataset identify any subpopulations (e.g., by age, gender)? so, please describe how these subpopulations are identified and provide a description of their respective distributions within the dataset. We mask all the age details of escorts in our dataset. However, the advertisements still contain description of the escort ethnicities.
Is it possible to identify individuals (i.e., one or more natural persons), either directly indi- rectly (i.e,, in combination with other data) from the dataset? If so, please describe how. While it is not impossible for individuals to find connec- tions from our dataset, we have taken extensive measures to minimize that risk by masking private identifiers in the ads. The dataset consists of ads from the Backpage escort markets collected be- tween December 2015 and April 2016. Since the seizure of the website, no public records exist for these ads.
What mechanisms or procedures were used to collect the data (e.g.. hardware apparatuses or sensors, manual human curation, software pro- grams, software APIs)? How were these mech- anisms or procedures validated? The data is provided to us from Bashpole Software, Inc..
Does the dataset contain data that might be con- sidered sensitive in any way (e.g., data that re- veals race or ethnic origins, sexual orientations,
Did you collect the data from the individuals in question directly, or obtain it via third parties or
19

---

other sources (e-g., websites)? Over what time- frame was the data collected? Does this time- frame match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created. The data collected in this research is scraped from online posted ads between December 2015 and April 2016 the Backpage Escort Markets.
link other access point to the "raw" data. No.
Is the software that was used to prepro- cess/clean/labll the data available? If so, please provide a link or other access point. No.
A.4.5 Uses
Has the dataset been used for any tasks already? If so, please provide description. This re- search uses the IDTraffickers dataset to establish au- thorship identification and verification benchmarks.
Were the individuals question notified about the data collection? If so, please describe (or show with screenshots or other information) how notice was provided, and provide a link other access point to, otherwise reproduce, the exact language of the notification itself. No.
What (other) tasks could the dataset be used for? None.
Is there anything about the composition of the dataset or the way it was collected and prepro- cessed//leaned/lbbeeed that might impact future uses? For example, is there anything that a dataset consumer might need to know to avoid that could result in unfair treatment of in- dividuals or groups (e.g., dereotyying, quality of service issues) or other risks or harms (e.g., legal risks, financial harms)? If so, please pro- vide a description. Is there anything a dataset consumer could do to mitigate these risks or harms? Despite our best efforts to mask sensi- tive information, it is important to note that the dataset still contains certain details such escort pscudonyms, posted locations, ethnicity, and sex- ual preferences. Although the likelihood of this information being used to harm individuals is low, we strongly discourage any unethical research or commercial applications that could potentially lead to the identification of escort individuals, either di- rectly or indirectly. Our intention is to prioritize the privacy and well- well-being of all individuals involved in the dataset.
Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with sereenshots or other in- formation) how consent was requested and pro- vided, and provide link or other access point to, otherwise reproduce, the exact language which the individuals consented. NA
consent was obtained, were the consenting in- dividuals provided with a mechanism to revoke their consent in the future or for certain uses? If so, please provide description, as well as link or other access point to the mechanism (iff appropriate). NA
A.4.4 Prepocessinglleeningghbbiigg
Was any preproesiing/ceannnglaeiii of the data done (e.g., discretization or bucketing, tok- enization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a descrip- tion. If not, you may skip the remaining ques- tions in this section. To prioritize privacy and minimize the potential for misuse, we implemented measures to protect sensitive information within the ad descriptions in our dataset. This involved masking various elements, such as phone numbers, email addresses, age details, post IDs, dates, and links mentioned in the ads. By applying these safe- guards, we aim to maintain the confidentiality of individuals involved while still enabling valuable research and analysis.
A.4.6 Distribution
Will the dataset be distributed third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a descrip- tion. Yes, we intend to make our dataset avail- able through the Dataverse data repository to mit- igate any potentially illegal or fraudulent use of our dataset. The access to the data will be sub- ject to specific conditions, including the require- ment for researchers to sign a non-disclosure agree- ment (NDA) and data protection agreements. These agreements will prohibit sharing the data and its use for unethical commercial purposes.
Was the "raw" data saved in addition to the pre- processed/cleanee/Jbbllle data (e.g., to support unanticipated future uses)? If so, please provide
20

---

How will the dataset will be distributed (e.g., tar- ball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)? Yes
permitted, we will acknowledge and appreciate the contributions of researchers and support their continued research endeavors.
When will the dataset be distributed? We will be releasing the dataset after the final decision from the EMNLP committce, along with the camera- ready version.
Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these re- strictions, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms, well any fees associated with these restrictions. No
A.4.7 Maintenance
Will the dataset be updated (e.g., to correct label- ing errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated dataset consumers (e.g., mailing list, GitHub)? As our research progresses, we are actively exploring NLP- based entity extraction techniques to improve the privacy of individuals in the dataset. Specifically, are investigating methods to effectively mask escort pseudonyms, posted locations, and ethnic- ity. Our goal is to develop reliable approaches that ensure the anonymity and confidentiality of indi- viduals involved. To keep the research community informed, we will share our progress and findings through research publications and provide updates to the data description on the Dataverse portal.
others want to extend/augmmen//uill on/contribute to the dataset, is there mecha- nism for them to do so? If so, please provide a description. Will these contributions be validated/verifiedd? If so, please describe how. If not, why not? Is there process for communiaating/disrrbtttigggg these contributions to dataset consumers? If so, please provide a description. We encourage researchers to actively contribute to and expand our dataset by extending, augmenting, and building upon it. However, in order to prioritize the safety and well-being of the individuals included in the dataset, have made the decision to restrict the sharing rights of the dataset. Nevertheless, warmly welcome fellow researchers collaborate with us on further advancements related to the dataset. While sharing the dataset itself not
21


Reviews:
[{"review_id": "eAFTCv6Vsg", "reviewer": "Reviewer_fd99", "summary": "", "questions": "Q-A: What do you mean by \"closed-set/open-set verification/classification/identification\"? Please clarify where you used each.\nQ-B: Is Figure 2 (A) the number of tokens or sentences? Based on the numbers, I guess it is tokens, but the label (SENT-LEN) is misleading.\nQ-C: Apparently, Backpage has been seized since 2018! But other websites like Craigslist (as you mentioned) are still active, and I assume your proposed approach could be beneficial in identifying HT on these other websites too. Can you give more examples, and briefly discuss how your approach can be suited for a few other similar prominent websites?\nQ-D: It appears from the Appendix that you have not collected any media/images from these advertisements. But, have you performed any manual analysis to see if there are duplicates that could be used for improving the authorship attribution task? (e.g., if the ads have used similar/same images, they might be from the same vendor or fake). At least you may want to briefly discuss this as a future research direction.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback and for acknowledging the importance of our work. We encourage the reviewer to find answers to their concerns below, and given an opportunity, we would like to integrate our responses to the comments in the camera-ready version of the manuscript. \n\n**R1:** We acknowledge the reviewer's valid concern regarding establishing definitive ground truth for escort advertisements (ads) associated with human trafficking (HT). This difficulty is inherent to measuring illegal activities: It is impossible to find undetected crime. Even data based on arrest records does not include undetected ads. Because we wanted to ensure that we use data that included HT ads, we analyzed Backpage escort website ads, as prior investigations have revealed instances of HT ads on the Backpage escort website ([1]). Furthermore, it is important to clarify that our primary objective is not to pinpoint HT ads directly. Instead, and as indicated in lines 656-665 of the manuscript, we aim to offer a tool that supports law enforcement in prioritizing their resources effectively by finding potential HT ads.\n\n**QA:** The closed-set authorship identification setting utilizes a classification task to analyze the writing styles of input ads and assign them to one of the predefined vendor classes ([2, 3]). Essentially, the objective is to classify textual ads into predetermined vendor labels (say vendor 1 - vendor 100) available during the training. Figure 1 (ii), Section 4.1, and Section 5.1 demonstrate this task's setup and the results. In contrast, the open-set authorship verification scenario utilizes a ranking task to enables a similarity analysis between vendor ads from established and _potential future vendors_ (say vendor 501). Importantly, this includes vendors that were not encountered during the training phase. Figure 1 (iii), Section 4.2, and Section 5.2 demonstrate the setup and the results associated with this task. \n\n**QB:** We thank the reviewer for pointing this out and apologize for the confusion. The values on the y-axis of Figure 2 (A) represent the number of tokens per ad. We plan to correct the label on the x-axis in the camera-ready version to avoid confusion. \n\n**QC:** The reason for not focusing on other active websites is related to the unavailability of comparable HT information in the data on those websites. \n\nRegarding using our approach for other websites, our methodology employs text ads to identify and establish connections among vendors associated with online escort ads. While the fundamental principles of our research can be theoretically extended to any active online escort website, the generalization capability of our approaches depends upon variations in writing styles, the use of chat-GPT-like automated systems, the presence of different languages, and data noise. In such instances, we recommend our readers improve the trained model through further fine-tuning, leveraging transfer learning techniques from the IDTraffickers dataset, and applying these to their specific data context.\n\n**QD:** We concur with the reviewer's observation regarding our dataset's potential use of images as soft labels. This aspect holds promise and is currently a focal point of our research efforts. We intend to present our findings in upcoming research. That said, we plan to incorporate this aspect into the discussion section of the finalized camera-ready version.\n\n**T1, T2, T4:** We agree with the reviewer's comment. We will ensure to rectify it in our camera-ready version. \n\n**T3:** We agree with the reviewer's comment here. We plan to include the literature for the authorship verification task in our camera-ready version. \n\n**References**\n\n1. BACKPAGE.COM'S KNOWING FACILITATION OF ONLINE SEX TRAFFICKING - https://www.govinfo.gov/content/pkg/CHRG-115shrg24401/html/CHRG-115shrg24401.htm\n2. Open Set Text Classification using Convolutional Neural Networks - https://aclanthology.org/W17-7557.pdf\n3. Open-Set Recognition: a Good Closed-Set Classifier is All You Need? - https://arxiv.org/abs/2110.06207", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A1: Human trafficking is an important and sensitive matter, especially in the escort market and adult services industry. Identifying such advertisements and vendors can help law enforcement identify potential human and sex trafficking incidents faster. \nA2: The authors collected a relatively large corpus of escort advertisements from the Backpage website before it was seized. The corpus contains over 87k ads from 14 different states.\nA3: The authors evaluate their proposed authorship attribution with other authorship attribution tasks as well as through an authorship verification of the present dataset\n", "reasons_to_reject": "R1: While this work aims to propose an approach based on authorship attribution for the identification of potential human trafficking vendors, in the end, it only helps identify ads posted by different vendors. This is because the authors do not have ground truth for vendors with the real risk of human trafficking and cannot claim their performance in detecting such vendors. However, I assume this is not unusual, and gaining access to such information would be difficult, if possible at all, for every vendor. And still, the proposed approach is domain-specific and takes into account the unique characteristics of an escort market website which is slightly different from any other authorship attribution task and could help in the detection of potentially suspicious vendors and advertisements.", "questions_to_authors": "Q-A: What do you mean by \"closed-set/open-set verification/classification/identification\"? Please clarify where you used each.\nQ-B: Is Figure 2 (A) the number of tokens or sentences? Based on the numbers, I guess it is tokens, but the label (SENT-LEN) is misleading.\nQ-C: Apparently, Backpage has been seized since 2018! But other websites like Craigslist (as you mentioned) are still active, and I assume your proposed approach could be beneficial in identifying HT on these other websites too. Can you give more examples, and briefly discuss how your approach can be suited for a few other similar prominent websites?\nQ-D: It appears from the Appendix that you have not collected any media/images from these advertisements. But, have you performed any manual analysis to see if there are duplicates that could be used for improving the authorship attribution task? (e.g., if the ads have used similar/same images, they might be from the same vendor or fake). At least you may want to briefly discuss this as a future research direction.\n", "paper_topic_and_main_contributions": "This paper presents an approach based on authorship attribution to identify potential human trafficking vendors on an online escort market currently seized (Backpage) and achieve a macro-F1 score of 0.87. Then they perform an authorship verification based on the extracted style representation with a mean r-presicion score of 0.89. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "T1: \"authorship techniques\" and \"authorship approaches\" have been used in the paper. But they are not always correct. The authorship approaches are not discussed in this paper. Instead, this paper discusses authorship \"attribution/verification\" techniques/approaches and maybe authorship-based approaches for detecting HT.\nT2: Since the reader reads each page from top to bottom, it would look/feel more natural if the flow of figures (Figure 1, for ex.) is from top to bottom and the starting point is clearer (especially in Figure 1 (i)).\nT3: The related work section has focused on authorship attribution, but one of the major tasks in this paper is authorship verification. That also needs to be discussed with enough emphasis in this section, even though some of these references and methods might be shared among attribution and verification. \nT4: You should place the footnote marks after the punctuation (line 201, for ex.)\n"}}, {"review_id": "nmecLxUrPY", "reviewer": "Reviewer_buUC", "summary": "", "questions": "* Q1) Line 211: What languages are found in the US-only filtered dataset (besides English)?  \n\n* Q2) Line 367-369: Surprised to see only 1% improvement (expected better). Wondering if multilingual models (such as mBERT, Roberta-xlmr etc) made any difference. \n\n* What is your take on generalization of styles to authors that maybe many years apart. Are HT vendors short lived-in their business, or do they have a career that lasts over a decade? \n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback and for acknowledging the uniqueness of our work. We encourage the reviewer to find answers to their concerns below, and given an opportunity, we would like to integrate our responses to the comments in the camera-ready version of the manuscript. \n\n**Response to the rejection concern:** We acknowledge the reviewer's concern regarding our dataset's age and regret not addressing it in our limitations section. We intend to address this oversight in the final version of our manuscript. Despite the presence of several active online escort platforms like Craiglist, YesBackpage, and Classifiedads, we focus our research on Backpage ads spanning December 2015 to April 2016. This choice stems from documented human trafficking (HT) activities on the Backpage website ([1]). We opted not to incorporate other active websites due to the absence of HT information in data from other escort websites. Please note that our research does not aim to track or assist currently ongoing human trafficking investigations. Instead, we aim to showcase the potential of authorship attribution techniques to aid law enforcement agencies in resource prioritization. Given this research objective, the age of the dataset holds smaller significance.\n\n**Q1:** Our analysis shows that around 99.32 % of the vocabulary in our dataset is from the English language. Complete details about the presence of all the languages are as follows:\n\n| **Language** | **Frequency** |\n|----------|----------------------|\n| ENGLISH  | 99.3200 %            |\n| SPANISH  | 0.4378 %             |\n| CHINESE  | 0.1382 %             |\n| JAPANESE | 0.0207 %             |\n| SOTHO    | 0.0196 %             |\n| FRENCH   | 0.0127 %             |\n| BOKMAL   | 0.0103 %             |\n| DUTCH    | 0.0058 %             |\n| TSWANA   | 0.0058 %             |\n| NYNORSK  | 0.0046 %             |\n| SWEDISH  | 0.0046 %             |\n| LATIN    | 0.0035 %             |\n| BASQUE   | 0.0034 %             |\n| SWAHILI  | 0.0023 %             |\n| MALAY    | 0.0023 %             |\n| XHOSA    | 0.0023 %             |\n| WELSH    | 0.0023 %             |\n\nThese statistics are obtained using the lingua language detection (https://github.com/pemistahl/lingua-py) Python library. \n\n**Q2:** As indicated above, the vast majority of vocabulary in our dataset (over 99%) belongs to the English language. Consequently, we did not explore the use of multilingual models in our research. Given that only a small fraction of our dataset's vocabulary lies outside English, we anticipate employing multilingual models would not significantly improve performance. \n\nAs detailed in lines 365-366, the language model trained on our dataset achieves a test perplexity of 6.22. However, this relatively high perplexity is expected due to noise within our dataset. Escort vendors intentionally introduce random noise into their advertisements to evade detection by automated systems. Consequently, the trained language model faces challenges in predicting whether the next token in the sequence is a word, a symbol, an emoji, or a series of whitespaces. We hypothesize that the minor <1% improvement in the performance of the trained classifier can be attributed, in part, to this random noise effect. This discussion can be found in lines 366-369 of our manuscript.\n\n**Q3:** Due to the absence of ground truth, collecting ads from authors spanning multiple years presents challenges. Determining the business age for potential human trafficking vendors is difficult, especially considering that law enforcement can not apprehend all such vendors. However, it is established that more organized traffickers exploit victims extensively over extended timeframes ([2]).\n\nIn essence, different people write differently, and since the authorship task relies on identifying individual writing styles and perpetrators of human trafficking spans all racial, ethnic, and gender demographics ([2, 3]); generalizing such approaches to vendors' existing years apart is not straightforward. Our methodology employs text ads to identify and establish connections among vendors associated with online escort ads. While the fundamental principles of our research can be theoretically extended to any active online escort website, the generalization capability of our approaches depends upon variations in writing styles, the use of chat-GPT-like automated systems, the presence of different languages, and data noise. In such instances, we recommend our readers improve the trained model through further fine-tuning, leveraging transfer learning techniques from the IDTraffickers dataset, and applying these to their specific data context.\n\n**Reproducibility:** We plan to attach the GitHub repository to our code in our camera-ready version for better reproducibility.\n\n**References:**\n\n1. BACKPAGE.COM'S KNOWING FACILITATION OF ONLINE SEX TRAFFICKING - https://www.govinfo.gov/content/pkg/CHRG-115shrg24401/html/CHRG-115shrg24401.htm\n2. Global Report on Trafficking in Persons 2022 - https://www.unodc.org/documents/data-and-analysis/glotip/2022/GLOTiP_2022_web.pdf\n3. Myths, Facts, and Statistics - https://polarisproject.org/myths-facts-and-statistics/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper presents a unique (perhaps rare) resource to the NLP/CL community. \n* Presents a baseline model, thus accelerating a future development that revise and improve quality of authorship verification models.", "reasons_to_reject": "Dataset comes from 2015/12 - 2016/04, perhaps a bit outdated regarding what has happened on the Internet. \n ", "questions_to_authors": "* Q1) Line 211: What languages are found in the US-only filtered dataset (besides English)?  \n\n* Q2) Line 367-369: Surprised to see only 1% improvement (expected better). Wondering if multilingual models (such as mBERT, Roberta-xlmr etc) made any difference. \n\n* What is your take on generalization of styles to authors that maybe many years apart. Are HT vendors short lived-in their business, or do they have a career that lasts over a decade? \n", "paper_topic_and_main_contributions": "Human trafficking (HT) is a societal problem and has online presense in the form of escort ads on certain websites. While the law enforcement agencies are working to tackle the problem, they need tools, and tools and models that need data. \nThis works presents a dataset and plans to release dataset, \n\nIn addition, they also present baseline models to identify(ID) the vendors via authorship verification. They have two tasks: classification and ranking task, where the latter is more generalizable. \nThe authors attempt to mask some direct clues (such as phone numbers, email ID, etc ) that directly indicate author ID, and instead expect the models to identify cluster of ads that are from the same author. \n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pcq916qKia", "reviewer": "Reviewer_iA59", "summary": "", "questions": "In this case, I would like the authors to comment on the points listed in reasons to reject.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback and for acknowledging our work as relevant, challenging, exciting, refreshing, and cutting-edge. We encourage the reviewer to find answers to their concerns below, and given an opportunity, we would like to integrate our responses to the comments in the camera-ready version of the manuscript. \n\n**A: Drift** The absence of ground truth poses challenges in collecting ads from authors spanning multiple years to comment on the stylometric drift. Our data collection specifically covers escort advertisements between December 2015 and April 2016, during which we observed no significant shifts in authors' writing styles. Moreover, given the brevity of this period, it's challenging for us to ascertain the stability of escort vendors. Notably, establishing the business age for potential human trafficking vendors is intricate, mainly because not all such vendors are apprehended by law enforcement. However, research indicates that more organized traffickers exploit victims extensively over extended timeframes ([2]). \n\nIndeed, we see a drift among vendor locations within our dataset. The extent of this drift across different American geographies is outlined in the following table:\n\n| Geographic Pair | Number of Common Vendors |\n|-----------------|-------------------------|\n| East - West     | 75                      |\n| East - North    | 6                       |\n| East - South    | 107                     |\n| East - Central  | 284                     |\n| West - North    | 7                       |\n| West - South    | 53                      |\n| West - Central  | 34                      |\n| North - South   | 7                       |\n| North - Central | 8                       |\n| South - Central | 207                     |\n\n_Our analysis further reveals that 7 vendors post their advertisements across all 41 states of the United States from which our dataset was collected._\n\nFinally, our methodology employs text ads to identify and establish connections among vendors associated with online escort ads. While our study's principles can be theoretically generalized to any online active escort website, the practical implications may vary based on certain factors, including disparities in writing styles, use of chat-GPT-like automated systems, the presence of different languages, and data noise. Additionally, gathering ads from authors spanning considerable time intervals is challenging. In essence, different people write differently, and since the authorship task relies on identifying individual writing styles and perpetrators of human trafficking, it spans all racial, ethnic, and gender demographics ([2, 3]); generalizing such approaches to vendors' existing years apart is not straightforward. In such instances, we recommend our readers improve the trained model through further fine-tuning, leveraging transfer learning techniques from the IDTraffickers dataset, and applying these to their specific data context. \n\n**B: Sensitivity** As outlined in lines 235-249, 626-655, and detailed in Appendix section A.2, we have implemented extensive measures to safeguard individual identities. These precautions include masking personal information, including phone numbers, email addresses, age details, post IDs, dates, and links. This masking is executed to prevent any possibility of reverse-engineering, thus mitigating the potential for misuse of our dataset.\n\nUnlike other encryption techniques, our masking process eliminates personal information. For instance, an email address like xzy@gmail.com is transformed into <EMAIL_ID_23>, a phone number such as +00 00 0000 000 becomes +NN NN NNNN NNN, and a link like www.google.com is rendered as <LINK> within our dataset. However, due to substantial noise in our dataset, we could not remove the first names of escorts utilized in the advertisements. Nevertheless, considering prior research indicating that escorts frequently use pseudonyms in their ads ([4, 5]), combined with the absence of public records for these ads post the 2018 seizure of Backpage Escort Markets, we assess it as improbable that individuals could exploit the personal data in our ads to cause harm.\n\nFurthermore, given the legitimate interest of our research, our masking methodology aligns with the spirit of the General Data Protection Regulation (GDPR) article 6, the lawfulness of processing ([6]), which acknowledges the importance of balancing privacy concerns with societal benefits. In this context, our research contributes to the betterment of society by facilitating studies aimed at addressing critical issues such as human trafficking and enhancing law enforcement efforts.\n\nFinally, we intend to grant access to the dataset via the Dataverse data repository portal. However, this access is subject to stringent restrictions. Researchers seeking access must undergo an application process, necessitating approvals and assurances. These assurances encompass non-disclosure and data protection agreements, ensuring responsible and secure use and release of the data.\n\n**C: Correlations** We thank the reviewer for pointing this out. Our analysis indicates that our trained classifier employs both mechanisms to distinguish between various vendors' writing styles effectively. Example A illustrates a scenario where our trained model has acquired a proxy function of a particular vendor's writing style. On the other hand, Example B demonstrates an instance where the model has developed the ability to differentiate between vendors by leveraging contextual cues. Please note that both examples are drawn from the true positive predictions within the test dataset.\n\n**_Example A:_** The examples below illustrate the model's inclination to formulate predictions using proxy functions instead of contextual cues. Our explanations clarify that the model assigns greater attribution to tokens presented in uppercase format. This leads us to believe that our trained model associates sequences of consecutive uppercase letters as a proxy function for representing advertisements from Vendor 5801.\n \n**_A1._** COME?? AND ENJOY THE SWEET TOUCH VERY BEAUTIFUL ??SENSUAL LATINAS GIRLS SUPER AMICABLE GOOD TREATE VERY SWEET NICE?? LADIES GOOD MASAJE ??SOF HAND GOOD SERVICE NO RUSH?? WE?? WAITING FOR?? YOU COME PAPI SENSUAL ??HOT NENAS BELLAS ??LATINAS WE PRIORATY?? IS FOR YOU EXCELENTE ??SERVICE ??FOR MORE INF COME WE LOCATION NNNN FLORENCE AVE TELL NNN.NNN.NNNN.& NNN.NNN.NNNN Posters age: <AGE> • Location: BELL HUNTINGTON PARK TELL NNN.NNN.NNNN ?, City of Long Beach, Downtown, Long Beach, Los Angeles, San Gabriel [SEP] <LINK> [SEP] SENSUAL curves EXÓTICAS?? Beautiful Ladies Sexy pretty ??GIRLS WAITING FOR YOU ??sweet Nice Touch?? - Los Angeles escorts - <LINK>\n\n**_A2._** MASSAGE TERAPEUTA SOF TOUCH ALL BODY RELAX SEXY CURVES EXOTIC SENSUAL HOT NENAS BELLAS LATINAS GIRLS SUPER AMICABLE AND SWEET COME BABY WE WAITING FOR YOU GOOD SERVICE NO RUSH WE PRIORITY IS FOR YOUR BEST TREATED WELCOME PAPI ENJOY THE SWEET TOUCH PRETTY GIRLS NICE PLACE AND PRIVATE ROOM BEAUTIFUL LADIES FOR MORE INF COME WE LOCATION NNNN FLORENCE AVE TELL (NNN)NNN.NNNN.& (NNN)NNN.NNNN Posters age: <AGE> • Location: BELL HUNTINGTON PARK TELL NNN.NNN.NNNN?, Downtown, Inglewood / Hawthorne [SEP] <LINK> [SEP] ??SWEET pretty ?? Latinas good ??Services ?? No rushed ?? N SEXY??Staff??Sof ??TOUCH ??good MASAJE?? - Los Angeles escorts - <LINK>\n\n**_A3._** ??PATITE ??SENSUAL??PRETTY?? NENAS LATINAS ??GIRLS COME ??AND ENJOY ????THE?? SWEET ??GOOD ??MASAJE ??EVERYTHING?? YOU NEED?? WE HAVE ????PRETTY ??LADIES ??GOOD ??SERVICE ??NO?? RUSH ??VERY AMICABLE ??GIRLS?? WELCOME ??PAPI WE WAITING ??FOR ??YOU THE PLACE IS?? VERY ??CLEAN ??AND?? PRIVATE ROOM ??FOR MORE ??INF COME ??WE LOCATION NNNN FLORENCE AVE ?? TEL ??NNN.NNN.NNNN & ??NNN.NNN.NNNN Posters age: <AGE> • Location: Downtown, Los Angeles [SEP] <LINK> [SEP] ?? Come On ??over Enjoy??SEDUCTIVE Beautiful ??Latinas?? waiting for ??You COME for good ?? MASAJE?? - Los Angeles escorts - <LINK>\n\n**_Example B:_** The examples below illustrate the model's inclination to establish predictions relying on contextual cues rather than solely depending on a proxy function. While we lack a definitive means of confirming the model's focus on the advertisement context, our explanations highlight pronounced inconsistencies in word attributions for similar tokens across these advertisements. Consequently, our assessment leads us to believe that these predictions are more likely a result of the contextual cues present within the advertisements of Vendor 390.\n\n**_B1._** Yes you read it right lesbians....but we love to show off and spoil a hard working man!!!we are Bbw so please do expect a thin girl. Just be polite and respectful and you will see two girls set ablaze!!! TEXT-NNN-NNN-NNNN S.S.-𝑁𝑁,𝐻.𝐻.−��,�.�.−NNN, Hr.-$NNN... Remember rates for time no illegal acts!!! Text now we are hosting in Lancaster east of downtown Lancaster!!! NNN-NNN-NNNN Posters age: <AGE> • Location: Harrisburg [SEP] <LINK> [SEP] Lesbians looking for a nice guy! - Lancaster escorts - <LINK>\n\n**_B2._** IN CALLS IN LANCASTER ONLY!!!Lancasters luscious lesbian BBWs are looking for a nice ,polite,and generous, gentleman to show a good time !!!?? If this sounds like a good plan to you TEXT US QUICK!!!?? Please?? no women, no couples,no drugs/alcohol, no explicit/inappropriate talk, or multiple men.?? Please be respectful. TEXT-NNN-NNN-NNNN S.S.-??NN?? H.H.-??NNN?? Hr.-??NNN?? Bonus= If you read to the end text the word \"candy\" to us to get a special surprise!!!?? Posters age: <AGE> • Location: Lancaster [SEP] <LINK> [SEP] Lancasters luscious lesbians!!! - Harrisburg escorts - <LINK>\n\n**_B3._** These are N BBWs You wanna make time for? A very real couple for N years? Sexy??,funny??,intelligent ??,and we are N Lesbians know how to show a guy a good time? We are in Lancaster close to Outlets. In calls only? ??? Please no women,no couples,no drugs/alcohol,no explicit comments,and no multiple guys???!!! PLEASE BE POLITE AND RESPECTFUL!!! ??NN-S.S.?? ??NNN-H.H.?? ??NNN-Hr.?? PLEASE TEXT ONLY-NNN-NNN-NNNN?? Posters age: <AGE> • Location: Lancaster Lincoln highway [SEP] <LINK> [SEP] Too hot for games?! Come join N\n\n**Typos Grammar Style And Presentation Improvements::** We agree with the reviewer's comment. We will ensure to rectify it in our camera-ready version. \n\n**References** \n\n1. BACKPAGE.COM'S KNOWING FACILITATION OF ONLINE SEX TRAFFICKING - https://www.govinfo.gov/content/pkg/CHRG-115shrg24401/html/CHRG-115shrg24401.htm \n2. Global Report on Trafficking in Persons 2022 - https://www.unodc.org/documents/data-and-analysis/glotip/2022/GLOTiP_2022_web.pdf \n3. Myths, Facts, and Statistics - https://polarisproject.org/myths-facts-and-statistics/ \n4. Comparing manual and computational approaches to theme identification in online forums: A case study of a sex work special interest community - https://www.sciencedirect.com/science/article/pii/S2590260121000229\n5. Indicators of Sex Trafficking in Online Escort Ads - https://www.ojp.gov/pdffiles1/nij/grants/305453.pdf\n6. Article 6 GDPR. Lawfulness of processing - https://gdpr-text.com/read/article-6/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Reasons to accept listed below:\n\nA: The paper is highly relevant and discusses a challenging real-world issue: human trafficking. To address the challenge, the authors propose a new and exciting dataset IDTraffickers that is large enough to fine-tune LLMs. In this way, we can leverage the power of LLMs and vector stores to address a societal issue at scale.\n\nB: The reported statistics are relevant and useful to help demonstrate the quality of the dataset. While I still have concerns about the data itself, I thought the statistics were necessary and near-sufficient.\n\nC: The design of the two tasks and the classification and recall methods to address them -- fine-tuning LLMs and applying vector stores for retriever setups -- is refreshing and cutting-edge.", "reasons_to_reject": "I can identify a few weaknesses listed below:\n\nA: Drift: Is there stability in the vendors, themes, locations and style of the scraped data points across time, and how much are the trained models susceptible to such drastic changes.\n\nB: Sensitivity: There are several places in the main text -- including an extended discussion in the Appendix -- about data privacy and security. How reliable are these PII-filtering techniques in this dataset? [See flag for Ethical concerns]\n\nC: Correlations between label (vendor) and text structure: I'd like to see a few examples of text *as they correlate to the vendors*. I am curious whether the trained LLM models are learning a proxy function of the vendor and not really using the entire context. The classification performance looks a little too promising...\n", "questions_to_authors": "In this case, I would like the authors to comment on the points listed in reasons to reject.", "paper_topic_and_main_contributions": "The paper contributes a novel and sizable dataset IDTraffickers to combat human trafficking -- by the methods, I'd say focused in the US -- by highlighting a key challenge: conventional methods require a very hands-on approach on a continuous stream of large-scale data, making it expensive. With LLM/LLM-with-memory models trained on the dataset, a first pass through the large amounts of data can be delegated to the model, and human intervention can be applied as a more fine-grained analysis. \n\nAuthors demonstrate relevant data statistics (frequency histograms and POS distributions) including one creative figure captioned \"wikifiability\", indicating that the scraped dataset is substantial, diverse and grounded. And suggest two tasks -- Authorship Identification and Authorship Verification. Tasks are addressed in distinct and rational setups: Identification is conducted via a standard LLM + Classifier setup; in Verification, the authors choose to use a vector store -- somewhat serving as a recommendation engine.\n\nAll-in-all, the combination of dataset and initial model performance baselines appear to suggest that the contribution paves a novel way of addressing human trafficking problems when considering the datasets in real-time and in bulk.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None come to mind."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I might have missed a few. Kindly note:\n\nA: line 80 - *none* analyzed\n\nB: lines 219, 220, 222 - (citations not in brackets)\n\nC. line 93 - United State: \"s\" missing\n\nIn general, a speling and grammar check might be useful.\n\nD: line 1 - \"violating their fundamental human right\" - sounds a little obvious.", "Justification_for_Ethical_Concerns": "Given the dataset IDTraffickers is about Human Trafficking, there needs to be a concerted effort to ensure that Personally Identifiable Information (PII or  P2I) is purged to protect victim identity. The paper describes in adequate detail the filtering steps conducted *by the authors* (lines 235 to 249) but an external review might be useful and necessary.\n\n"}}]

Author rebuttal:

Meta-review: The study introduced a dataset aimed at potential human trafficking operations, comprising 87,595 text ads and 5,244 vendor labels. This addresses a significant social issue, and reviewers agree that the dataset will be a valuable asset to the research community.
Various important points were brought up by the reviewers that should be considered to enhance the paper. For instance, Reviewer buUC highlighted i) the time frame of the dataset, ii) multilingual models - providing some justification through experimental results would be more beneficial than mere speculation.

Reviewer iA59 pointed out ethical concerns, which the authors addressed in the rebuttal. Such discussion should be elaborated upon in the paper for better clarity.
Decision: Accept-Main