SWEET - Weakly Supervised Person Name Extraction for Fighting Human Trafficking
Authors: Javin Liu
Hao Yu
Vidya Sujaya
Pratheeksha Nair
Kellin Pelrine
Reihaneh Rabbany
Venue: EMNLP 2023 Findings (2023)
Keywords: Information Extraction
Large Language Models
Generation
NLP Applications
Resources and Evaluation

Abstract: In this work, we propose a weak supervision pipeline SWEET: Supervise Weakly for Entity Extraction to fight Trafficking for extracting person names from noisy escort advertisements. Our method combines the simplicity of rule-matching (through antirules, i.e., negated rules) and the generalizability of large language models fine-tuned on benchmark, domain-specific and synthetic datasets, treating them as weak labels.
One of the major challenges in this domain is limited labeled data. SWEET addresses this by obtaining multiple weak labels through labeling functions and effectively aggregating them. SWEET outperforms the previous supervised SOTA method for this task by 9% F1 score on domain data and better generalizes to common benchmark datasets. Furthermore, we also release HTGEN, a synthetically generated dataset of escort advertisements (built using ChatGPT) to facilitate further research within the community.

SWEET: Weakly Supervised Person Name Extraction for Fighting Human Trafficking
Javin Liu"
Hao Yu"
University of Southern California Mila Quebec AI Institute
Vidya Sujaya"
Pratheeksha Nair
Kellin Pelrine
Reihanch Rabbany
McGill University
Mila Quebee Al Institute
Abstract
et al., 2017; Li al., 2022: Kejriwal al., 2018: Kejriwal and Kapoor, 2019). However, this is very challenging because the text is often noisy, ungram- matical, and obscured. Information related to the names of individuals, services, locations, etc, often have letters replaced by symbols or emojis. The writing style constantly evolves and is made ad- versarial avoid detection by online filters and moderators.
In this work, we propose weak supervision pipeline SWEET: Supervise Weakly for Entity Extraction to fight Trafficking for extracting person names from noisy escort advertisements. Our method combines the simplicity of rule- matching (through antirules, i.e., negated rules) and the generalizability of large language mod- els fine-tuned on henchmark, domain-specific and synthetic datasets, treating them weak labels. One of the major challenges in this do- main is limited labeled data. SWEET addresses this by obtaining multiple weak labels through labeling functions and effectively aggregating them. SWEET outperforms
Based on discussions with domain experts, in- cluding criminologists, law enforcement, and other groups fighting trafficking, was determined that extracting advertised names from the ads is partic- ularly important. Traffickers tend to control, on average, four to six victims and often post adver- tisements on their behalf (Thorn, 2015). Hence, multiple people being advertised in the same or similar ads is strong indicator of trafficking (Lee et al., 2021). Other entities such locations, phone numbers and email addresses could also be informa- tive. However, most websites have explicit fields for these entities which makes them more straight- forward to extract. The same not true for person names which are usually embedded inside the text of the ads and are much more complex to extract. Traditional extraction methods, such the embedding- based Flair (Akbik et al., 201 8a) and Spacy NLP models, do not do well this type of data because they are sensitive noise and tend decrease in performance as more noise is injected into the data (Li et al., 2022). NEAT (Li et al., 2022) stands as the current SOTA name extractor in this domain and uses an ad-hoe combination of a name dictionary, a rule dictionary, and a BERT-based dis- ambiguation layer. Moreover, NEAT relies a sample of manually labeled data provide optimal results. This is challenge because the relevant datasets for this task are typically curated in-house and not shared publicly due to their sensitive nature. Similar constraints also apply to methods such as crowd-sourcing.
previous su- pervised SOTA method for this task by 9% F1 score on domain data and better generalizes to common benchmark datasets. Furthermore, we also release HT GEN, synthetically generated dataset of escort advertisements (built using ChatGPT) to facilitate further research within the community.
1 Introduction
Over 6.3 million people worldwide are victims forced sexual exploitation or human trafficking (HT) on any given day (International Labour Orga- nization), and the majority of the victims have been advertised online (Polaris. 2021; MINOR, 2015) through escort websites (Rhodes, 2016). HT is an organized crime and traffickers tend to advertise multiple victims simultaneously. Hence, finding connections between online escort advertisements hints towards them being posted by the same indi- vidual and strongly indicates organized activity. Several methods have previously taken the ap- proach of uncovering connections between ads by looking for repeated phrases, phone numbers, lo- cations, prices, service types, etc., in the text (Lee al., 2021; Tong et al., 2021; Rabbany et al., 2018). Hence, efficient entity extractors must extract accu- rate and relevant information from ad text (Nagpal
Theeee authors contrihuted equally to this rescarch.

---

To address these challenges, we propose weakly supervised method SWEET (Supervise Weakly for Entity Extraction to fight Trafficking)', that efficiently extracts person names from escort ads and independent of training labels. We define label-indeeendence in the scope of our work as not requiring domain or task-specific labels. SWEET uses novel combination of fine-tuned language models (elucidated in Section 3), antirules, and an HMM-based method of aggregating them. Our contributions are:
Entity extraction from escort ads This is an important task that involves identifying and extract- ing named entities from given ad. Rule-based models typically rely on regular expressions, hand- crafted rules, and gazetteer-based approaches. For person names, NEAT (Li et al., 2022) is the SOTA method that combines gazetteer (person name dic- tionary) with a regular expression extractor (rule dictionary) and RoBERTa (Liu al., 2019) based disambiguation layer called HT-bert. Such rule- based systems were shown(Li et al., 2022) have a better performance over statistical models (De- vlin et al., 2018; 8; Peters et al., 2018; Liu et al., 2019) on the noisy escort ads dataset. However, on benchmark NER datasets, NEAT performs much worse. This is likely result of its rule-based com- ponents designed to capture patterns in escort ads. Rule-based approaches can also be complemen- tary to statistical-based approaches. In other do- mains, (Ratinov and Roth, 2009) argued that the NER task relies highly on external knowledge and shows that a statistical model combined with a rule- based gazetteer match makes netter-performinn hybrid model.
We study the problem of name extraction from noisy text through the novel lens of weak supervision and propose new method.
Novel:
Effective:
The proposed method SWEET is very effective and outperforms the previous SOTA in person name extraction on domain data by 9% Fl score.
Label-efficient: SWEET leverages weak labels that are easy and cheap to obtain. This is very useful in several real-world applications, particu- larly in the HT domain, where obtaining labels is expensive and challenging due to the sensitive nature of the data.
Weak supervision Since we consider multiple sources of weak labels for our task, we rely on a popular weak-supervision framework. Skweak, for label aggregation and learning. Skweak (Li- son et al., 2021) is Python-based based weak supervi- sion framework made specifically for NER tasks. It works together with the Spacy library (Honni- bal et al., 2020), allowing users to create labeling functions (LFss) that label an input text token as a type of entity and facilitates downstream process- ing. Skweak provides a variety of different LF types: heuristics, gazetteers and document-level functions allowing us to easily combine weak sig- nals from our proposed combination of rule based systems and LLMs. These make Skweak better choice compared to other weak label aggregators such as Snorkel (Ratner al., 2016). The labels obtained from these LFs are later aggregated using Hidden Markov Model (HMM).
Improving reproducibility: We also introduce HTGEN, a synthetically generated dataset of es- cort ads which, unlike the real-world datasets used in papers in this domain, can be published for further research and helps with reproducibil- ity.
2 Background
In this section, we cover the background informa- tion and related works necessary understand this paper's ' context better.
Named Entity Recognition Named Entity Recognition (NER) is the task of identifying el- ements in the text that correspond to high-level categories like PERSON, ORGANIZATION, LO- CATION, DATES, and other concrete concepts that can be explicitly named. The NER task aims to ex- tract all occurrences of such entities. In this paper, the focus is on human names because they are most associated with an HT victim and names are the most common entity type to appear in escort ads.
The HMM's states and observations correspond to the true labels and LF outputs respectively. Skweak relies on majority vote strategy (where each LF is voter) get a predicted label. This is then used calculate the HMM's initial tran- sition and emission probabilities, which are then updated until convergence using the Baum- Welch algorithm. To account for possible dependencies
Our code is available at https//gthaa..omm/Compccctttii NILR

---

between LÍs, Skweak tempers the probability den- sity (weight) of an LF depending on its redundancy. It decreases the weight of an LF that produces a label of type 'PERSON", for example, it shows high recall with other LFss that also produce 'PER- SON' labels (high overlap in predictions). Post ag- gregation, separate classifier model can be trained on the HMM's aggregated results and used directly other datasets without requiring the LFs or the HMM
2. Fine-tuned Models: We fine-tune base versions of RoBERTa (Liu et al., 2019) and DeBER- Tav3 (He et al., 2021) for the task of Named Entity Recognition on six different datasets, namely: the train splits of CoNLL22003 (Tjong Kim Sang and De Meulder, 2003), WNUT2017 (Derczynski et al., 2017), Few-NERD (Ding et al., 2021) (Only Level 1), WikiNER (English) (Nothman al., 2012), domain dataset la- belled by ChatGPT called HTUNSUP, and a domain dataset generated and labeled by Chat- GPT called HTGEN. The details and labeling method of HTUNSUP are provided Section 4.6 and Appendix A, respectively. In total, 12 fine-tuned models (one per dataset and model type) act LFs in SWEET. Any word pre- dicted by fine-tuned model, that matches a white-space-separace word in the ad text, gets annotated as a "PERSON NAME" according that LF.
Large Language Models Recent works (Gilardi al., 2023; Huang et al., 2023a) have shown that ChatGPT can be used as a data annotator and ad- dresses ethical concerns regarding using human labelers on datasets that contain sensitive informa- tion. We experimented with using ChatGPT to extract person names from ads and used this la- beled set to tun- BERT-based models as LFs in our framework.
We also experimented with using ChatGPT for generating escort ads create an additional do- main dataset. It has been shown that LLMs like ChatGPT exhibit stochastic behaviours (Bender et al., 2021) and are susceptible to biases from the real world internet data it was trained on. Recently, several works (Li et al., 2023) (Liu et 2023) have demonstrated effective jailbreak techniques that bypass the content filters that are imposed on ChatGPT to limit the level of bias and toxic lan- guage output by ChatGPT, (Carlini et al., 2021) could also extract training data directly from GPT LLMs.
Using the Skweak framework, we fit an HMM on the annotated train set and apply the fitted model on the annotations to obtain final set of aggregated labels. The HMM's initial parameters are obtained using a majority voter that treats each LF as voter of equal weight to get a predicted label.
We also experimented with using the three com- ponents of previous SOTA NEAT (Li et al., 2022) as LFs but ultimately found it did not boost perfor- mance (see NEAT (Weakly Supervised) in Table 3). Specifically, we considered:
1. Context Rules:
extract person names based on part-of-speech (POS) tags and common phrases that include names. The rules consist of regular expressions that work capture the most com- mon contexts where names are seen. (eg. I'm NNP, My name is NNP, You can call me NNP).
3 Methodology
In this paper, we introduce a hybrid weakly su- pervised Skweak-hased model that uses both rule- based approaches (such as "antirules" detailed be- low) and statistical-brat based approaches (such as De- BERTa and RoBERTa models trained on different NER datasets) as labeling functions and combine them into a single, more effective predictor. Our methodology pipeline is in Figure 1.
2. Name Dictionary: common names with con- fidence scores tuned based on frequency in in- domain data. These names are used in gazetteer matching, where the names are used as the gazetteer and the pipeline, given word in a sentence, will check if it is a word in this dictio- narylgazetteer.
The proposed weak supervision pipeline in SWEET consists of 2 main types of L.s::
1. Antirules: Rules that determine entities as nor person-names. The top X% most frequent words of the dataset (X E {10, 20, 30, 40, 50)) are an- resulting in 5 antir-
3. Word Disambiguation: filters results of the weighted output of rule and dictionary extrac- tor with the help of DeBERTaV3 model that was fine-tuned for the task of masked language modeling on a domain dataset (Li et al., 2022).
notated as "NOT_NAME ule LFs total. The antirules help counteract possible noise.

---

Labeling Functions (LF)
skweak
HMM
1. Fine-tune LLMs
Create LFs
Multiple Fine-tuning Datasets
Fine-Tuned
3. Annotate using LFs
5. Aggregate
LLMs
Annotated data
Labeled data
Frequency Antiruuess
Figure I: SWEET Architecture
During the training of the DeBERTaV3 model, individual words are masked and the task for the model to predict these masked words. A word's context is passed to the fill mask, and the number of correct predictions (the number of predictions by the fill mask that is in the name dictionary) is divided by the total number of predictions by the fill mask get confidence score for the disambiguation.
Similar to CoNLL2003, we use only the B-PER and I-PER classes for evaluation.
4.3 Broad Twitter Corpus (BTC)
The Broad Twitter Corpus (BTC) (Derczynski al., 2016) dataset large, diverse, and high-quality annotated corpus, created for the development and evaluation of NER in social media. BTC includes tweets from different regions, time periods, and types of Twitter users. It was annotated by combi- nation of NLP experts and crowd workers, ensuring both high quality and crowd recall.
4 Datasets
To fine-tune SWEET LFs, we used 4 open-source datasets, namely the training sets of CoNLL2003, WNUT2017, FewNERD-LI, and WikiNER-cn and 2 escort ad datasets, namely HTUNSUP and HTGEN. To evaluate SWEET, used the test of CoNLL2003, WNUT2017, BTC, Tweebank and an escort ad dataset called HTNAME. We also generated a domain dataset HTGEN whose train set was used for fine-tuning LFs and test set was used for evaluating SWEET. We provide the dataset statistics in Table 1, and the descriptions below.
4.4 Tweebank
Tweebank(Jiang et al., 2022) was developed to ad- dress the challenges of analyzing social media data, specifically Twitter messages through NER and syntactic parsing. The researchers created new English NER corpus called Tweebank-NER based on Tweebank and annotated it using Amazon Me- chanical Turk.
4.5 HTNAME
4.1 CoNLL,2003
This is a modified version of the HTlk dataset from (Li et al., 2022) consisting of 995 escort advertisements in English and French, where we rectified three main types of labeling errors.
CoNLL2003(Tjong Kim Sang and De Meulder, 2003) is very popular baseline for evaluating the performance of different NLP systems and algo- rithms. Our experiments use CoNLL2003 data from the HuggingFace which is in English. For evaluation, we only consider the name annotations (i.e, B-PER and I-PER) and ignore other entity classes.
1. Parsing issues in cases where the first name and last name were joined as a single string (FirstNameLastNamee) were not correctly anno- tated as "FirstName. LastName".
2. Apostrophe issue cases where an extra s is added to the name. Examples like 'Mizz Mer- cede's parlour' was wrongly annotated as 'Mer- cedes' (the should not be included as the name).
4.2 WNUT2017
WNUT2017(Derrzynski al., 2017) consists of user-generated text and contains many examples of informal language, abbreviations, misspellings, and other noisy characteristics. Due to this, models tend to have lower recall values on WNUT2017.
3. Blank labels significant number of examples were missing true labels. We manually added the missing labels and reintroduce this dataset as HTNAME.

---

Datasct
Train Size
Test Size
Total Size
Batch Size
4.7 HTGEN: Generated escort ads dataset
CoNLL2003
14041
3453
17494
128
WNUT2017
3394
1287
4681
128
We explore ChatGPT's ability to generate escort ads with the motivation of developing shareable domain specific dataset for furthering research in this field".
FowNERD-LI
131767
18824
150591
32
WikiNER-en
129907
14435
144342
128
BTC
20022
Tweehank
1201
HTNAME
995
995
HTUNSUP
6160
6160
32
We used role playing to convince ChatGPT (API with GPT-3.5-turbb-002
HTGEN
9424
818
10242
32
that it is a researcher studying patterns in escort ads. This bypasses the filters meant to reduce inappropriate outputs, mis- use, and generation of text found in such ads. We followed up with a description of the entities in the ads and some of the patterns unique the HT domain. (Bonifacio, 2023) had shown that provid- ing the context before every question often helped improve the output quality. We also provided a sam- ple conversation between user (who provides the prompt) and ChatGPT (providing the required gen- crated response) from three examples taken from our private HTNAME dataset. We scrambled phone numbers and other sensitive data for privacy rea- sons. In this manner, generated 10,000 syn- thetic escort ads called HTGEN and used them fine-tune a BERT model. At the time of writing, it costs $0.002 per 1000 tokens, and for 10,000 ads, costs approximately $4.40 USD. This is signifi- cantly cheaper than hiring manual labelers.
Table Statistics of the datasets used. The grayed-our numbers indicate that the test sets are not used in our experiments.
4.6 HTUNSUP
HTUNSUP was gathered by us from private es- cort websites and has around 6000 ads. Multiple recent works have shown that ChatGPT's strong performance on text related tasks makes it strong candidate for automatic labeling of certain types of data (Huang et al., 2023b; Mei et al., 2023; Zhu et al., 2023). Additionally, due to its outstanding performance on HTNAME (Table 2), we use it to generate pseudo labels for HTUNSUP, which we treat as ground-truth.
Weak Labeling of HTUNSUP We used the Chat- GPT API (gpt-3.5-turbo) extract person-name entities from ads. The average token count includ- ing the prompt, and response was 263. Due to content modcration and dialogue optimization, on encountering improper or explicit text (as in the case of escort ads), ChatGPT responds incoherently and for such cases, we return "no entity detected"..
We provide examples of ads from HTNAME and HTGEN in appendix D but note that some contain explicit language. The ads generated by ChatGPT are pretty similar to those in HTNAME. and ChatGPT was successfully able to replicate the patterns and noise of the HT domain.
The prompt used was: "I want you to act as natural and no-bias labeler, extract human names and location address and social media link or tag in the format 'Names: AnLocations: AnSocial: If multiple entities exist, separate them by If no entity exists, say N. Your words should extract from the given text, can't 't add/modify any other words, should be as short as possible, remember don't 't include the phone number. For one name, it should be less than 3 words."
5 Experimental Setup
5.1 Baselines
We compare SWEET with eight baselines, includ- ing off-the-shelf libraries, BERT-based models, domain speciif NER methods and a simple major- ity vote for aggregating weak labels as opposed to Skweak, ordered by their appearance in our result tables.
After post-processing and removing certain rare. high-frequency wrong predictions manually, obtain HTUNSUP with high-quality pseudo labels and use it for fine-tuning BERT models.
Spacy² - A popular open-source Python li- brary for natural language processing tasks. The Spacy NER package uses statistical mod- els, neural network architectures and deep learning techniques trained on large amounts of data.
F1
Precision
Recall
0.90
0.89
0.91
"The exact prompts used and the code for the dataset gen- eration can provided upon request. "https: pattps://space //spacy. wat.lc.../uuage//eeo inggistiicicc featurest named-entitin
Table 2: Performance ChatGPT for name extraction on HTNAME

---

TwitterNER (Mishra and Diesner, 2016) semi-supervised approach that combines text cluster information and word embeddings and uses a CRF for NER
and ELMo achieve higher scores than SWEET, but SWEET the next best performing model. On Tweebank, TwitterNER, designed specifically for tweets, is the best performing method. Please note that on re-evaluating NEAT on HTNAMF, we found an increase in the strict FI score from the pre- vious results (0.76 from Table 2 in Li et al. (2022) becomes 0.78), and use this stronger performance as the baseline, as reported in Table 4.
LUKE (Yamada et al., 2020) - transformer based model that learns contextualized embed- dings, fine-tuned on TACRED dataset (Zhang et al., 2017)
EL (Peters al., 2018) model that learns contextualized word representations, tuned for NER
Comparison of SWEET with NEAT:
When com-
pared with NEAT on the domain datasets, SWEET increases Fl score and recall by 9% and 18% spectively whilst maintaining precision. Further- more, SWEET significantly surpasses NEAT on CoNLLL2003, WNUT2017, BTC and Tweebank. This may be attributed to signals from the fine- tuned DeBERTa and RoBERTa LFs as well as false positive reduction due to the antirules.
Flair (Akbik et al., 2018b) trained charac- ter language model that learns contextualized string embeddings, tuned for NER
NEAT (Li et al., 2022) previous SOTA for person name extraction from escort ads variation of NEAT where each individual component is treated as an LF in our weak supervision pipeline
NEAT (Weakly Supervised)
Performance of SWEET v.s. individual LFs:
Majority Vote - a simple strategy of choosing the label based on consensus of majority of the weak labels, competitor to Skweak.
Table 5, we report the performance of individual labeling functions. When compared to the per- formance of SWEET in Table 4, we can see that SWEET successfully combines different fine-tuned models resulting in an aggregated model that out- performs these individual models, those fine- tuned on domain data (HTUNSUP and HTGEN). SWEET'S recall and precision scores being higher than all individual fine-tuned LFs (except for pre- cision on HTN AMEE which is on-par with HTUN- SUP), shows the success of weak supervision in using weaker sources (LFs) to produce stronger final model. The weak supervision methodology however can impact the performance significantly as we can see in Table and Table 4, when compar- ing simple majority vote performance with the proposed SWEET. The majority vote baseline still performs relatively well, and shows strong recall, however, SWEET performs significantly better in 5 out of 6 datasets.
5.2 Evaluation Settings
We compare the performance of SWEET with its variants and the baselines using the average Fl clas- sification score 5-folds of the test set. For each run, fold acted the test set while the remaining were used fit the HMM. Since we only consider the 'PERSON" class, we adopt the following evalu- ation metrics to better our use case.
Word-level matching: both ground truth and pre- diction split into words (separated by space) and each word is treated as separate entity. Lowercase: both the ground truth and predictions are converted to lowercase.
Exact Match: since the entity matching is on the word level, we consider strict match as a True Positive sample.
5.3 Results
Table shows the results of ab- lation experiments of SWEET, focused on the HT datasets. Firstly, removing antirules decrease preci- sion, showing that they are helpful in informing the model on what is not name. Meanwhile, we see mixed effect of domain LFs in the model. In row 3, we observe the highest precision and F1 scores on HTNNam with the removal of in-domain LI's. On HTGen, we instead see significant decrease, but note that is still higher than other baselines Table 4. In the last rows, we observe that using only
A Ablation Study
Tables and summarize the results our ex- periments for the general and HT domain datasets respectively.
Overall performance of SWEET: SWEET ob- tains the highest F1 score and recall on 4 out of datasets, and beats NEAT all datasets. On the HT datasets, SWEET performs the best. On general noisy datasets, WNUT2017 and BTC, SWEET sim- ilarly performs the best. On CoNLL2003, Flair

---

Method
CONLLAW03
WNUT3
BTC
Tweebank
sgaCy (Homniell 2020)
24
.18
TwitsnNiii (Misles and Dioserr, 2016)
A8 205
.01
61.4 X9
14 10
5-.4-10
12.44 B5
36 14
.26 244
.874 84
87-
-51- 285
TUKE (Nmmod et dl, 2000)
31 -1
18
1100
19
55
/6
1
92
31 1. 101
137 01
26.1 06
35
11 (Pein 2013)
56
.95
39
15
.55
.7.1
23.2
Fair (Alblk at 2019)
58 202
.97.4 2.2
1.0.2: 02
600 15
.39 18
2.15
.3 13.42000
26 225
21 03
1024 107
766 299
62.2
NEAT (Origins)) et 2022) 2022)
17 00
43.
07
+
22: 06
47 044
16
11t 02
Afc 201
100 = 00
124 + 055
42 04
17+
NEAT (Mially Supersisial
42.4
07
42.4 04
174
Mogenty voie
16
14.1
95
65
33 16
06
05
100
16
SWEET -Domosn Zeat
36
06
78 1.
30
02
69
101
00
82
06
6.1 035
86
20
001
55
06
45.4
106
85.1
79
98
38
02
Table 3: Word-level strict match results on 5-fold open source test sets: Fl score, Precision, Recall of SWEET and
baselines. NEAT (Weakly Supervised) refers to the use of NEAT (Li ct al., 2022) components as labeling functions skweak.
Method
HTNEME
HTGEN
F
Prec
Rec
Prec
spaCy (Homnibal 2020)
+ .33
.1
.50 + .03
TwitteNNER (Mishra and Diesner, 2016)
.66
52.t
.70 + .03
LUKE (Yamada et al., 2020)
63 .03
85
.11
84 = 02
56
ELMo (Peters et al., 2018)
4 ,02
.66
.66
.61 + 06
Hlair (Akbik al., 2019)
.02
.73
322
.83 + 04
NEAT (Original) (Li 2022)
.78 04
83
74
.63 1 02
79
NEAT (Weukly Supervised)
.79 02
80
.77 02
77 1
664 1 02
78
Mujority
.73 + .02
.99 +
95 + .01
74 4 02
.65 + 03
85
SWEET Domain Date
88 4 01
85 =
92 01
.75
02
71 4 03
78
SWEET
.7 4 001
88
92 + 01
02
.76 +
03
84 +
Table 4: Word-level strict match results on 5-fold HT test sets: F1 score, Precision, Recall of SWEET and baselines.
NEAT (Weakly Supervised) refers to the use of NEAT (Li et al., 2022) components as labeling functions in skweak.
Fine tuning Dalaset
Precision
Recall
Discussion
HTUSSUP
57 4 03
31 = 02
622 : 02
CGNLL2003
47 +
67 + 03
.69 + 00
6.1
Effect of using ChatGPT
F
57+
WikiNER-en
52+ 01
48 + 42
.4 + 02
Although ChatGPT is the best performing model on HTNAME (Table 2), there are concerns regard- ing its use on escort ads. First, there is some fi- nancial cost associated with employing ChatGPT on large datasets of escort ads which limits its ac- cessibility whereas method is open- source and completely free to use. Second, the stability of the OpenAT APT output is a concern as discussed in (Chen et al., 2023). While this does not hurt our method much where one only needs to get good performance a single time (to provide weak labels on a fixed dataset), and one could consider even the more powerful GPT-4, it means it would be unreli- able to build a system for real-world applications based on ChatGPT alone. Finally, depending on the method used to access ChatGPT, there can be privacy concerns with this sensitive data. The in- tended use our method is on large scale real-time ads with sensitive information, and using ChatGPT, is not feasible due to cost, stability and privacy issues mentioned above.
WNUT2011
20.+ 02
722
HTUNSUP
82+ 02
84 = 03
83 03
HTGEN
72:4 02
= 02
+:
RoBERTI-basc
CoNLL2003
.72 02
68 - 03
Fow-NERD-LI
.88 18:1.00 02..0022
- 02
02
WikiNER-en
.49 10:10000
1 03
.56 1:
WNUT2017
.68 1. 02
73
11 003
.66 1. 03
Table 5: Fine-tuned model LFs on HTNAME evaluated on 5-fold test sets. All these LFs have a lower per- formance compared to the aggregated model SWEET, reported in Table 4.
domain LFs yields similar Fl scores with SWEET, but compromises recall for precision. We note that without using any domain data (second to last row of Table 4), SWEET able to achieve the same Fl score as a setup that uses only domain data.
We also observe that removing HTUNSUP LFs results in an Fl score decrease only on HTNAME. Moreover, these LFs do as well on their own (last four rows in Table 6), although they are still lower than SWEET on HTNAME, showing that the ag- gregation of LFs fine-tuned on diverse datasets is a key component of SWEET.
Moreover, although ChatGPT is valuable in- gredient in SWEET, our ablations directly show that it is not sufficient. In the target domain our full system achieves 87% Fl (Table 4). Simply apply-

---

Method
HTNAME
HTGEN
F1
Prec
Rec
F1
Prec
Rec
AIL
HTGEN LFS
88 01
85 + .01
.22 + .01
.81 = .02
.76 + .03
.84 + .03
LFs
HTUNSUP LFs
86 022
.22 = 02
91 = .02
.2 = .02
.77 = .03
84 - .03
LFs
HTGEN LFS- HTUNSUP LFs (Domain Data)
88 01
.85
.01
.75 = 02
.71
.78 .03
LFs
Antirules
85 = 01
.79
.01
.8 - 02
.75
.03
A Autirdess HTGEN LFs
85 f .01
.79
93 .01
.81 1 .02
.75
.03
LI
Antirules
HT UNSUP LFs
83 1: .02
.76 1: 001
92 001
81 1: .02
.76 1: .03
84 1: .03
AIL Antirules- HTGEN LFs HTUNSUP LFs
82 = 022
.76 + .02
.92 .02
.81 + .01
.76 + .03
.84 + .03
Only HTGEN L.ss
76 .01
.02
.81 +
.79
Only HTGEN LFs Antirales
86 = 001
.82
.92 .01
.75 = .04
.05
Only HTUNSUP LFfs
87 = 01
.88 + 001
88 : 001
81 = 03
.77 + .04
81 = .04
Only HTUNSUP LFs + Anntirales
87 = 011
.92 = 02
85 .02
.76 = 00
.83 - .05
.66 + .05
Only JTGGEN LPs HTUNSUP LFs
85 1 01
.85
022
87 02
82 03
.78 003
81 .03
Only HTGEN LFs HTUNSUP LFs Antirules
86 022
.86
.2
.79
Table 6: Ablation results of SWEET. AAntiruess indicates removal of the antirule LFs component of SWEET.
HTGEN LFs or HTÜNSUP LFs indicates L.Ss fine-tuned on HTGEN or HTUNSUP respectively.
ing language models trained on ChatGPT-laheled data achieves 82% at best (Table 5), significantly lower. Conversely, approach without using any ChatGPT-labeled data maintains 88% F1 (Table "SWEET- DotainData").
SWEET maintains or improves this performance on removing any LFs trained on relevant domain data. SWEET does not require any human annotators and labeling which is a very important improvement over previous methods because allows it be applied easily to real-world datasets. can also be easily adapted to other domains by designing spe- cific new labeling functions. We also release a new public escort ad dataset HTGEN that uses ChatGPT for both data generation and labeling, which will improve reproducibility and benchmarking in this domain where most data cannot be shared publicly, as it contains personal information.
6.2 Label Efficiency
Our pipeline eliminates the need for human label- ers as it is independent of training labels for the task at hand, making use of existing open-source la- beled data for fine-tuning language models instead. This is advantageous over other label-dependee NER models, especially in the HT domain where data labeling is costly and time-consuming ing label independent is also beneficial in terms of ethics and privacy as the HT domain contains many examples of private information like names, locations, social media tags, and phone numbers. These types of data would no longer need to be read by a crowdsource worker or human labeler in order to generate labels for training. More im- portantly, our method is also independent of LF's trained on the same data as the domain. For e.g, SWEET excluding LFs trained on HTGEN and HTUNSUP (SWEET DamainData) performs as well or even better than SWEET.
Be-
Limitations
Despite SWEET having achieved SOTA results HTNAME, several limitations need to be consid- ered while interpreting the findings.
1. Dataset Size and Diversity: The HTNAME database used in this research is modified ver- sion of HTlk from Li et al. (2022). The dataset contains limited number of data (around 1000 ads). 5-fold evaluation is deployed to accom- modate for this small size. While generating HTGEN, although efforts were made to ensure its representativeness and diversity, HTGen's 's data may still not be 100% representative of real data. This may limit the generalizability of the models trained on it to other domains or datascts.
7 Conclusion
In this paper, we introduced SWEET, a weak su- pervision pipeline that extracts person names from escort ads without the need for manual labeling. The experimental evaluations show that SWEET achieves SOTA results on the HTNAME dataset with an Fl score of 0.87, outperforming the previ- ous SOTA by 9% while also generalizing better to CoNLL2003 and WNUT2017 datasets. Moreover,
2. Explainability and transparency: HT is high-stakes application domain and trans- parency, explainability, and accuracy are all cqually important. SWEET has the ability to aggregate multiple models together to get a final

---

prediction. However, it uses HMM for aggre- gation which is more complex and less explain- able than the ad-hoc way deployed by Li et al. (2022). Moreover, the output of the fine-tur one-tuned LFs is also less interpretable. SWEET sacrifices some explainability for increased performance. However, thanks to its modularity, the HMM aggregator can be swapped out for a more ex- plainable method.
Research Ethics Board Office at the authors' uni- versity for using this type of data. We have also studied the current best practices for the project through a commissioned Responsible AT Institute evaluation, one of whose recommendation was to focus on have biweekly consultations with human trafficking survivors and have been mindful of not reproducing biases in the design of the algorithm. No personal attributes such as age, physical descriptors, eth- nicity, etc were used. In addition, we have also reviewed the current law and policy implications through a comprehensive legal risk assessment and mitigation memorandum from a law firm. As the laws and policies in this domain are catching up with the technologies, we also plan to conduct re- search in close collaboration with domain experts into the most ethical approach Al in this do- main. Lastly, for transparency and accountability, all of the algorithms being developed will made available online. Data scraping scripts and our datasets (except those synthetically generated) will not be shared publicly to maintain confidentiality and anonymity.
design. To this end, we
3. Cost: We were able to improve performance by aggregating BERT models fine-tuned on a va- riety NER datasets. This time-consuming and costly. We used industrial -grade GPUs (RTX8000, V100 and A100) to fine-tune the BERT models.
4. Privacy: The HT domain contains private in- formation like person's 's name, address, social media tag, email, and phone number. Although the HTNAME and HTUNSUP datasets consists of ads scraped from public websites, these web- sites may not be fully regulated, and thus, data coming from these ads may violate the privacy of individuals being advertised in those ads. We had to refrain from releasing these datasets be- cause of privacy and ethical concerns. Thus, directly replicating our results HTNAME and HTUNSUP is not possible. To help counteract this and improve reproducibility, we released HTGEN.
Acknowledgements
This research is partially funded by the Canada CIFAR AI Chairs Program and Samsung-Mile Re- search Grant.
5. Lack of variety: We have shown that the HT- GEN dataset generated by ChatGPT contains realistic-sounding ads. However, the ads tend to be more repetitive than our datasets (HTUNSUP, HTNAMe) with real data scraped from the web. Changing the temperature settings and other pa- rameters showed promise at boosting the varia- tion in the generations but the model was more inclined to produce nonsensical sentences. Fu- ture work should aim at increasing variety in the generated ads and keep noise withing desired range. With more variety of data in the gen- erations, it would also be possible decrease the size of HTGEN by reducing the number of redundant examples.
References
Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR: An casy-to-use framework for state-of-the- art NLP. In NAACL 2019, 2019 Anmual Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations) pages 54-59.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018a. Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638-1649, Santa Fe, New Mexico, USA. Associa- tion for Computational Linguistics.
Ethics Statement
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018b. Contextual string embeddings for sequence labeling. In COLING 2018, 27th International Con- ference on Computational Linguisties, pages 1638- 1649.
Rigorous ethical considerations were taken during the process of this research. The data we use is publicly available, and due to the nature of the ad- vertisements, there are no reasonable expectations for privacy. However, due to the sensitivity of the data, Ethics Approval has been obtained from the
Emily M Bender, Timnit Gebru, Angelina McMillan-
Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models

---

be too big?. In Proceedings of the 2021 ACM confer- ence on fairness, accountability, and transparency, pages 610-623.
Fan Huang, Haewoon Kwak, and Jisum An. 2023b. Is (ChatGPT) better than human annotators? potential and limitations of [ChaiGPT) in explaining implicit hate speech. In Companion Proceedings of the ACM Web Conference 2023. ACM.
Stephen Bonifacio. 2023. Creating chatgpt based chatbot using "in-context learning" method.
Ben Hutchinson, Vinodkumar Prabhakaran, Emily Den- ton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020. Social biases in NI.P models as barriers for persons with disabilities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5491-5501, Online, Association for Computational Linguistics.
Nicholas Carlini, Florian Tramer. Eric Wallace, Matthew Jagielski, Ariel Vosss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, UI- far Erlingsson, al. 2021. Extracting training data from large language models. In USENIX Security Symposium, volume 6.
Lingjao Chen, Matei Zaharia, and James Zou, 2023. How is chatgpt's behavior changing over time? arXiv preprint ar30 .90009.
International Labour Organization. 2022. Global estimates of modern slavery: forced labour and forced marriage. https:/ https://ww... olg. -ate://---dddddoomm---iieeeeeeee httpum/ett//pbllicattoo///ceeeee
Leon Derczynski, Kalina Bontcheva, and Ian Roberts. 2016. Broad Twitter corpus: A diverse named entity recognition resource. In Proceedings of COLING 2016, the 26th International Conference on Compu- national Linguistics: Technical Papers, pages I 1169- 1179, Osaka, Japan. The COLING 2016 Organizing Committee.
pdff.
Hang Jiang, Yining Hua, Doug Beeferman, and Deb Roy. 2022. Annotating the tweebank corpus on named entity recognition and building NLP models for social media analysis. CoRR, abs/2201 1.0281..
Mayank Kejriwal and Rahul Kapoor. 2019. Network- theoretic information extraction quality assessment in the human trafficking domain. Applied Network Science, 4(1):1-26.
Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Linsopatham. 2017. Results of the WNUT2017 shared task on novel and emerging entity recogni- tion. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140-147, Copenhagen, Denmark. Association for Computational Linguis- tics.
Mayank Kejriwal, Pedro Szekely, and Craig Knoblock. 2018. Investigative knowledge discovery for com- bating illicit activities. IEEE Intelligent Systems. 33(1):53-66 63.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1rXiv:1810.04800.
Keita Kurita, Nidhi Vyas. Ayush Pareek, Alan W Black. and Yulia Tsvetkov. 2019. Measuring bias in con- textualized word representations. In Proceedings of the First Workshop Gender Bias in Natural Lan- guage Processing, pages 166-172, Florence, Italy. Association for Computational Linguistics.
Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. 2021. Few-NERD; A few-shot named entity recognition dataset. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 1998-32,3, Online. Association for Computational Linguistics.
Meng-Chies Lee, Catalina Vajiac. Aayushi Kulshrestha, Sacha Levy, Namyong Park, Cara Jones, Reihaneh Rabbany, and Christos Faloutsos. 2021. Infos- hield: Generalizable information-theoreti human- trafficking detection. In 2021 IEEE 37th Inter- national Conference on Data Engineering (ICDE). pages 1116-11221 IEEE
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-workers for text- annotation tasks. arXiv preprint 2rx.v:2303.15056,
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško. Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina SoMMillan-Mjjo Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matus- sière, Lysandre Debut, Stas Bekman, Pierric Cis- tac. Thibault Gochringer, Victor Mustar. François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Process- ing: System Demonstrations, pages 175-184, Online
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre- training with ing. arXiv preprint arXiv:2111.0954.
embedding shar-
Matthew Honnibal, Ines Montani, Sofie Van Lan- deghem, Adriane Boyd, et al. 2020. spacy: Industrial- strength natural language processing in python.
Fan Huang, Haewoon Kwak, and Jisun An. 2023a. Is chatgpt better than human annotators? potential and limitations of chatgpt explaining implicit hate speech. arXiv preprint arXiv:2302.07736.

---

and Punta Cana, Dominican Republic, Association for Computational Linguistics.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistiex: Human Lunguage Tech- nologies, Volume I (Long Papers), pages 2227-2237, New Orleans. Louisiana. Association for Computa- tional Linguistics.
Haoran Li. Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. 2023. Multi-step jailbreaking privacy attacks on chatgpt.
Yifei Li, Pratheeksha Nair, Kellin Pelrine, and Reihaneh Rabbany. 2022. Extracting person names from user generated text: Named-entit recognition for combat- ing human trafficking. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2854-2868.
Polaris. 2021. 2020 u.s national human trafficking hot- line statistics. https: https://shottur..atto
Reihanch Rabbany, David Bayani, and Artur Dubrawski. 2018. Active search of connections for case building and combating human trafficking. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2120- 2129.
Pierre Lison, Jeremy Barnes, and Aliaksandr Hubin. 2021. skweak: Weak supervision made easy for NLP. In Proceedings of the 59th Annual Meeting of the As- sociation for Computational Linguistics and the 1 1thh International Joint Conference Natural Language Processing: System Demonstrations. Association for Computational Linguistics.
Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the thirteenth conference on compu- tational natural language learning (CoNLL-2009), pages 147-155.
Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. 2023. Jailbreaking chatgpt via prompt engineering: An empirical study.
Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. 2016. Data pro- gramming: Creating large training sets, quickly. Ad- vances in neural information processing systems, 29.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 190n.1197.1. 1692.
Leanne Maree Rhodes. 2016. Human trafficking as cybercrime. AGORA INTERNATIONAL OF ADMIN- ISTRATION SCIENCES, 4:23-29.
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint arXiv:1711.05101.
Thorn. 2015. Report on the use tech- nology to recruit, groom and sell domes- tic minor trafficking victims. https: 1/2715111qpwey226mkc11 wwwpnkkwwwwooooeeweieeeii hetdna-ssl. http://c/o//occoeeo//eeeeeeeee/e IiSSurvivor_Survey pdf.
Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D. Plumbley, Yuexian Zou, and Wenwu Wang. 2023. Wavcaps: chatgpt-assiste weakly-labelled audio captioning dataset for audio-language multimodal research.
Erik F. Tjong Kim Sang and Fien De Meulder, 2003. Introduction to the CoNLL-2003 shared task: named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142- 147.
S. D. MINOR, 2015. A report on the use of technol- ogy to recruit, groom and sell domestic minor sex trafficking victims.
Shubhanshu Mishra and Jana Diesner. 2016. Semi- supervised named entity recognition in noisy-text. In Proceedings of the 2nd Workshop on Noisy User- generated Text (WNUT), pages 203-212.
Yiqi Tong, Yidong Chen, and Xiaodong Shi. 2021. A multi-task approach for improving biomedical named entity recognition by incorporating multi- muutt--raauulriity information. In Findings the Association for Com- putational Linguisticx: ACL-IICNLP 2021, pages 4804 4813.
Chirag Nagpal, Kyle Miller, Benedikt Boecking, and Ar- tur Dubrawski. 2017. An entity resolution approach to isolate instances of human trafficking online. In Proceedings of the 3rd Workshop on Noisy User- generated Text, pages 77-84, Copenhagen, Denmark. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: Sfate-of-ther of-the-art natural language processing. Proceedingy of the 2020 Conference on Empirical
Joel Nothman, Nicky Ringland, Will Radford, Tara Mur- phy, and James R. Curran, 2012. Learning multilin- gual named entity recognition from Wikipedia. Arti- ficial Intelligence, 194:15.

---

Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online. Association for Computational I Linguistics.
Individual Labeling Functions We provide the performance of individual labeling functions in this section. Table 7 shows the performance of LFs used in the NEAT (Weakly Supervised) experiment in Table 3. Table and Table 9 shows the results of in- dividual fine-tuned models used as LFs in SWEET.
Tkuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextualized entity representations with entity- aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 6442 6454, On- line. Association for Computational Linguistics.
Model
F1
Precisson
Rocall
Label Ladel-Indep NEAT Rules
43 1 00
88 00
28 1.
Label-ladep NEAT Dictionary
83
Label-hodep NEAT Disambiguation
79 02
80 02
77
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 35-45, Copenhagen, Denmark. Association for Com- putational Linguisties.
Table 7: NEAT-based LFs scores on HTNAME evalu- ated on f-fold test sets.
Model
Dataset
F1
Precision
Rocall
UITUNSUP
.57 1..11
82 L. .14
.57 1 .1
CONLL.2008
.02
.99
DeBERTaV3 buse
Few NERD-LI
92.1.0
.95 1 001
.91 1 011
Yiming Zhu, Peixian Zhang, Ehsan-UI Haq, Pan Hui, and Gareth Tyson. 2023. Can chaigpt reproduce human-generated labels? study of social computing tasks.
WikINER-cn
96
002
02
WNUT2017
.58 1 04
.56 L .04
.65 04
ITUUSUPP
.74..06 06
.84 .07
.60 1 07
CoNLI
.98
.02
.99
RoBERTa base
Few NERD-LI
.92
00
1 02
WikNNER-cn
.93 + 03
94 + 085
.97 +
WNUT2017
.02
Reader's discretion: Appendix D contains ex- amples of escort ads that contain vulgar and explicit language. If you prefer to avoid explicit language or find such content uncomfortable, we advise you stop reading before reaching appendix D.
Table Fine-tuned model LFs of SWEET on CoNLL2003 evaluated 5-fold test sets.
Model
Dalusel
Precision
Recall
HTUNSUP
A1 11:21
50 50.100 11
52 1 .14
CoNLL2003
.72 .04
71 11:100 08
.73
A Labeling Functions
DeRERTaV
Rew-NERD-I.
56
82 02
WikiNER-en em
.6 1.11
88 1.13
.22 13
WNUT2017
62
+15+.15
Fine-tuned Models For fine-tuning the Token- wrapper from Hug-
ITUNSUP
.3 53.1.07
1.13
.63 L
CaNi 2000
.71 .6
.67 + 08
.71 + 08
Classification models, gingface (Wolf et al., 2020) AutoModelForToken Classification class to compile different backbones, here RoBERTa and DeBERTaV3. We adopt the train-test split of CONLL2003 and WNUT2017 from the Huggingface datasets (Lhoest et al., 2021) for the convenience of reproducing the re- sults ("wnut_l1? _7" for WNUT2017, "conl12003" for CoNLL2003). For FewNERD-LI and WikiNER- en, we directly downloaded the official stored file and parsed it the same format as Dataset loaded from the Datasets Library for running the experi- ments.
RoDERRa-base
NERD-LI
WikiNER-en
46 + 10
66 63.009 09
.5 09
WNUT2017
.67 .09
881
1 .14
.48 1 1.
Table 9: Fine-tuned model LFs of SWEET on
WNUT2017 evaluated on 5-fold test sets.
B ChatGPT and the HT Domain
ChatGPT is an advanced language model devel- oped by OpenAl and built upon the GPT architec- ture. Although the training data that is used to train ChatGPT is not made public, it is built upon the GPT (Generative Pre-trained Transformer). These models were trained on Common Crawl, a dataset that includes wide range of web content and ad- related content. By providing essential background and convineing ChatGPT that it is researcher that studies human trafficking, we were able to guide to generate a list of patterns generally seen in escort ads (Figure 2).
In the training stage, the learning rate is relative to batch size, which is 2 x 10 x soatch_size// 128). We train for 5 epochs with AdamW optimizer (Loshchilov and Hutter, 2017) on one A100 (40GB) GPU,
Detailed information about both the size of the training datasets and the used training batch size are listed in Table 1.

---

(ees for sale come and pickk your favoribe bouqued loday" - orlering them up
symbol potential vietims taise
before
Sexy Cougar 41 Hello there, my name is Gina. I'm a sexy and mature brunette with curves in all the right places. Let show you what real woman experienced and know how to please man. Incalls only. Call or text me at 905-XXX-XXXXX to book an appointment.
"Ouncall girt aval otle now come exporience the plecsure of our tstented oscorts" this ad uses the tern "outoal" to suggest that the trafficced individuals sell be brught to the cuavomer nither then thes customer going them.
do. I'm
"Meeet our lovely ladees Lisa, Kate,, and waiting for call this ad ricknames instead names,
tactic used
instickers
"Grifriend experierns a- call now for a good time with the hotteet girts town" this ad employs misspelings avade fihers as well as intimaling that these gills w offering kind of refilionaiip experience in urder supeel la vulnerabile idvriduals.
Figure 3: Derogatory or Offensive Language
SeXy BlAcK MiLFi Have you tried Sexy, spicy Black woman? Look no further, I'm the perfect combination Sexiness, Beauty, Intelligance, and personality. I'm Angell, years old and I'm the perfect woman for you. I have smooth chocolate skin, an smazing body and know how to please. I'm completely independent discrest, always ready to satisty your needs. provide all safe services, no rushed sessions, and easy parking. Call or text me now at XXXXXXXXXXXXXX AA Trades
By fnndng and analyzing simiar ads to these, researchers will be able devalop better undorstanding of patterns and language human traffickers uttimetlly enabling law enforcement to more effectively sombat this heinous activity.
Figure 2: ChatGPT's Understanding of HT Domain
C Problems with LLMs
Large language models, such as ChatGPT have garnered recognition from researchers for their ver- satility on a wide range of NLP tasks while ex- hibiting commendable performance compared to other SOTA models. Nevertheless, it is imperative acknowledge that these models have also faced significant scrutiny and critical appraisals. In this context, we mainly highlight the issue of Encoding Bias.
Figure 4: Usage of Vulgar and Sexually Ex- plicit Words
Young SPANISH BEAUTY IN TOWN XoXo Outcalls 24/7 Hil am a young, sexy, and beautiful Spanish girl ready to provide you with a mesmerizing experience. offer outcalls 24/7, and I am always discreet and professional. provide an unforgettable experience and satisfy all your fantasies. Call or text me to book an appointment. No black men. Xoxo
Due to the large size of the datasets used to train LLMs, it is very difficult to check and verify every part of the training dataset for potential bias. Thus, it common for large language models to exhibit various types of bias as stated in (Bender al., 2021: Hutchinson et al., 2020; Kurita et al., 2019). When building HTGEN, we observed that the gen- erated text had a propensity to include derogatory language (Figure 3), and sexually explicit words (Figure 4). It also generated ads that have a nega- tive sentiment towards the black community (Fig- ure 5). Of all the examples generated in HTGEN, black people are denied services more than any other ethnic group. No mention of the words found in Figures 3 and had occurred in any prompting processes. This ability to generate realistic sound- ing ads suggests that there is a good chance it had exposure to escort ads during its training process.
Figure 5: Racism in HTGEN
Title: Michelle's $80 CARCALL Special!!! Ø
Description: LOWBALLERS WILL BE BLOCKED!!! $80 $130 F-S $160 BB-FS 123-456-789
Figure 6: Example ad From HTNAME
Title: GisellaGorgeous Puerto Rican e @ Description: My name is Gisella sexy little pelite thing with curves that don't stop & am excited to be your grown up bestie and come over for a private playdate. I love to make new friends! Ø D Let's get together and see what kind of trouble we can stir up! @ @ @ (916)888-7777 no extra photos don't ask for any. am not a cheap commodity S incalls only.
D Examples of escort ads
Figure 7: Example ad From HTGEN


Reviews:
[{"review_id": "ahBPGXmUrC", "reviewer": "Reviewer_5iZ8", "summary": "", "questions": "A. Though there are few works on weakly supervised NER for Noisy text, there are many for weakly supervised NER. Those are more ideal benchmarks than methods like spacy NER and flair? Was there are any specific reason behind selection of these over weakly supervised NER for effective benchmarking.\nB. The ratio of data from HT's (unsup + Gen) to HTName is very low, with HTName having significantly lesser samples. What was the reason behind not looking at improving the test set? For example, synthetic test set with human evaluation would be more helpful in moving the area forward? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their remarks and hope to answer their questions below:\n\n**Q1. The paper argues as weak supervision for NER from noisy Human trafficking post, but the benchmark is very weak, infact uses only 995 samples from the domain. Meanwhile the authors focus on showing generalizability against CoNLL, WNUT which begs to question if the paper focus is on NER generalization or NER for Human trafficking post. Besides the total number of named entities etc are completely unknow for HTName which is used for testing. Moreover some of the results in Table 5, show that simple LF’s with source domain data to perform onpar with SWEET, which again is another area of concern. Moreover weakly supervised NER framework is compared with regular NER methods for justification (other than NEAT).** \n\n\nDue to the explicit and sensitive nature of escort ads (which is the main source of data in HT detection studies), there are no publicly available datasets particularly for the task of extracting person names from these ads. The only known private dataset is HTName from previous work [1] which is small and contains only 995 samples. Table 1 from [1] provides insights on the number of names, etc in HTName and we will include this information in our camera-ready version, if accepted. \n\nThe previous SOTA for the HT domain, NEAT, had poor performance on CoNLL and WNUT. So in our work, we show that our method outperforms it on both the domain-dataset (HTName) as well as general NER datasets (CoNLL and WNUT). To clarify, our motivation for including experiments on these general datasets is not for NER generalization but rather to show that SWEET is a very flexible pipeline that can be adjusted based on the task at hand. LFs can be easily swapped out and exchanged with other LFs.  \n\nOn table 5, it is true that the simple LFs with source domain data perform on par with SWEET on HTName, however they do not generalize well on other datasets and given the small size of HTName and low costs of adding more labeling functions, one can expect SWEET to perform better in the real-world. \n\nWe will revise our writing in the paper to better reflect these points.\n\n**Q2.Though there are few works on weakly supervised NER for Noisy text, there are many for weakly supervised NER. Those are more ideal benchmarks than methods like spacy NER and flair? Was there any specific reason behind selection of these over weakly supervised NER for effective benchmarking.**\n\nWithin the weak supervision paradigm, our work falls under programmatic weak supervision where instead of 1 weak label, each data point is associated with multiple weak labels which are aggregated using the HMM. In this way, it is inherently different from weakly supervised methods that deal with a single weak label or label noise. To compare with the weakly supervised NER baselines, we would need to pick one of the LFs to act as the “weak label” which makes it a slightly different problem setup than the one we focus on. Our experiments aim to show that aggregating multiple weak labels provides better results compared to using pre-trained methods like Flair, Spacy or other transformer-based models. \n\n\n**Q3. The ratio of data from HT's (unsup + Gen) to HTName is very low, with HTName having significantly lesser samples. What was the reason behind not looking at improving the test set? For example, synthetic test set with human evaluation would be more helpful in moving the area forward? why create synthetic data when there are existing SoTA?** \n\nHTName is a relatively small dataset consisting of 1000 ads used in the paper of one of our benchmarks [1] (referred to as HT1k in that paper). Since then we introduced HTUnsup which is 6 times the size of the HTName]. HTName was labeled using Mechanical Turk and there are recent works that show that these workers are increasingly relying on ChatGPT [5] and other recent works that discuss how ChatGPT does better than the human labellers for annotations [6]. \n\nAside from that, the data naturally contains explicit language and sensitive information like email, phone number, and address and thus using manual labellers is not ideal. This difficulty in labeling this data is one of the main motivating factors for a weakly supervised solution. Moreover, synthetic labeling is significantly more cost effective. In view of these considerations, a synthetic dataset with synthetic labellers allows for more privacy and allows us to scale up more rapidly and at a cheaper cost (as mentioned in section 4.5).\n\n\n**Q4. How does the proposed idea fair against https://dl.acm.org/doi/abs/10.1145/3502223.3502228 and other similar works, especially in terms of novelty. Because reading sections 3 shows the LF's used are fairly heuristical and common. Meanwhile the 6 models used are meant for regular NER. If there are models that are specific to noisy text it is more better to cross compare on the results. Often because of the fact that the noisy data results heavily depend on how the noise is modeled.**\n\nThe work cited above uses a similar approach to weak labels as us where a fine-tuned BERT model is used to generate pseudo labels. In our work, however, we have focused on leveraging simple heuristics and predictions from several such fine-tuned language models and aggregating them to make an ensemble. We consider this problem setting to be different from one where a single weak or noisy label is handled and hence did not compare with such methods in our experiments. The cited work is also semi-supervised and is dependent on a small set of labeled samples whereas our method doesn't rely on any manual labels. To provide better comparison with methods for noisy NER, we are extending our work to include more datasets used in noisy NER methods, so that SWEET’s performance can be indirectly compared with them. \n\n**Q5. how is the variance of noise in HK dataset, otherwise one can argue that LF's are more overfit for the data in hand. Besides many of these posts exhibit syntactic patterns. So in the explanation of datasets used for evaluation and building LF's such details need to be precisely presented to ground the arguments in section 5. Besides the test set of HTName is very small, I wonder if the results are debatable?**\n\nWe will improve the writing in the revised version to include more details relevant for LF building.\nTo clarify, the noisy nature of the data is emphasized to demonstrate the challenges related to its annotation and why simple rule-based approaches and gazetteers alone may fail. None of the LF models were trained on any ground-truth labels in the HT domain and the HMM was trained on only the outputs of the LFs. Hence, overfitting is unlikely. During the training process, HTName remains an unseen test set and the evaluation results provided in Table 3 are the mean and standard deviation over 5-folds. \n\nWe acknowledge the remaining comments on specific lines and table results and will include update the paper to reflect them. \n\n**References:**\n\n[1] Li, Yifei, et al. \"Extracting Person Names from User Generated Text: Named-Entity Recognition for Combating Human Trafficking.\" Findings of the Association for Computational Linguistics: ACL 2022. 2022.\n\n[2] Ratner, Alexander J., et al. \"Snorkel: Fast training set generation for information extraction.\" Proceedings of the 2017 ACM international conference on management of data. 2017.\n\n[3] Pierre Lison, Jeremy Barnes and Aliaksandr Hubin (2021), \"skweak: Weak Supervision Made Easy for NLP\", ACL 2021 (System demonstrations).\n\n[4] Zhou, Jianlong, et al. \"Ethical ChatGPT: Concerns, challenges, and commandments.\" arXiv preprint arXiv:2305.10646 (2023).\n\n[5] Veselovsky, Veniamin, Manoel Horta Ribeiro, and Robert West. \"Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks.\" arXiv preprint arXiv:2306.07899 (2023).\n\n[6] Fabrizio Gilardi, Meysam Alizadeh, & Maël Kubli (2023). ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30), e2305016120.\n\n[7] U.N. (2021). U.N News. Traffickers abusing online technology, UN crime prevention agency warns.\n\n[8] Chen, Lingjiao, Matei Zaharia, and James Zou. \"How is ChatGPT's behavior changing over time?.\" arXiv preprint arXiv:2307.09009 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "In addition to contributions mentioned above, following are few reasons for acceptance\n\n- The paper is well drafted and easy to follow\n- The literature selected and area of focus though narrow has large value in HT post mining and NLP community.\n- Moreover the authors do introduce new synthetic dataset, which was effectively analyzed in this work.\n- The methods selected are well established making the study suitable for fostering further research in this areas.\n- The empirical results are exhaustive enough to justify the results.\n", "reasons_to_reject": "The paper argues as weak supervision for NER from noisy Human trafficking post, but the benchmark is very weak, infact uses only 995 samples from the domain. Meanwhile the authors focus on showing generalizability against CoNNLL, WNUT which begs to question if the paper focus is on NER generalization or NER for Human trafficking post. Besides the total number of named entities etc are completely unknown for HTName which is used for testing. Moreover some of the results in Table 5, show that simple LF's with source domain data to perform onpar with SWEET, which again is another area of concern. Moreover weakly supervised NER framework is compared with regular NER methods for justification (other than NEAT). \n\n\nFollowing the key points to be improved for the paper\n\n- Lines 005 - 007 regarding combining rule matching with generalization for weak labels is unclear. Rather you can mention the specific labelling function capability of the sweet architecture here in brief. \n- Line 017 - 018 why create synthetic data when there are existing SoTA? Please add some details on the same to keep the work interesting to audience. For example lack of public sets (if it is the reason?)\n- Line 090-095, can you characterize efficiency quantitatively? For example, you can say something like it takes 50% less time to build model with proposed approach (if something like this is indeed possible)\n- Line 082-084: How does the proposed idea fair against https://dl.acm.org/doi/abs/10.1145/3502223.3502228 and other similar works, especially in terms of novelty. Because reading sections 3 shows the LF's used are fairly heuristical and common. Meanwhile the 6 models used are meant for regular NER. If there are models that are specific to noisy text it is more better to cross compare on the results. Often because of the fact that the noisy data results heavily depend on how the noise is modeled.\n- Line 265 - Please mention on the rationale behind selection such diverse sets, especially non noisy dataset like CONNLL\n- Line 391 - Till here most of the sections are clearly written, however there is one question which is unanswered. The definition of noise which is being addressed. Specifically how is the variance of noise in HK dataset, otherwise one can argue that LF's are more overfit for the data in hand. Besides many of these posts exhibit syntactic patterns. So in the explanation of datasets used for evaluation and building LF's such details need to be precisely presented to ground the arguments in section 5.\nBesides the test set of HTName is very small, I wonder if the results are  debatable? \n- Line 468 - What is the purpose of evaluation on CoNLL 2003, especially the main argument of the entire paper is on noisy text. Meanwhile from results we can see that Antirules are acting like filters leading to drop in results, which supports my earlier argument on line 082. Please reconsider this section to accomodate such details.\nMeanwhile Table 5 and 6 are hard to follow and correlate with earlier sections. For example, please define  SWEET Include like roberta + deberta with many datasets + frequency rules. Where both LMs are trained on various XYZ dataset (From section 4). Then add that - HTGEN LF's include models of roberta deberta with HTGen data only. for comparison. Not the others. Because comparing SWEET which has many LF's + other components with only LF's of specific type needs to be clearly highlighted with rationale, else it would reflect as benefit of other components. For example, The takeaway from Table 5, is to use HTGen LF's + Antirules but not full sweet system? Because in Table 6, we can see removing these the results are not as high.\nLine 509 - Entire section has no value add, rather please merge this with limitation sections to justify benefit of work.\n\n\n", "questions_to_authors": "A. Though there are few works on weakly supervised NER for Noisy text, there are many for weakly supervised NER. Those are more ideal benchmarks than methods like spacy NER and flair? Was there are any specific reason behind selection of these over weakly supervised NER for effective benchmarking.\nB. The ratio of data from HT's (unsup + Gen) to HTName is very low, with HTName having significantly lesser samples. What was the reason behind not looking at improving the test set? For example, synthetic test set with human evaluation would be more helpful in moving the area forward? ", "paper_topic_and_main_contributions": "The paper proposes an adaptation of weakly supervised system of noisy text NER. Specifically it develops labeling functions and heuristic rules for NER from human trafficking posts. The paper is well drafted with relevant details for anyone with little knowledge of weakly supervised NLP. The paper besides provides extensive evaluation of the pipeline showing various benefits and drawbacks of the components. Meanwhile the limitation section is the key highlight, where the authors effectively justify and accept the weakness of the said pipeline which is rarely seen in research papers.  Overall a solid empirical paper!!", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Please see comments above."}, "extra_text": {}}, {"review_id": "XANkeALIEF", "reviewer": "Reviewer_pn8R", "summary": "", "questions": "No", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their remarks and hope to answer their questions below:\n\n**Q1. The current setting, which involves using the model (fine-tuned roberta/deberta) trained on HTUNSUP and HTGEN to indirectly teach another model (HMM), should not be referred to as weak supervision. IF you have access to the train sets of HTUNSUP and HTGEN, it would be more performant to directly train a model on them or compare your HMM method with other baselines that fully utilize these training. Essentially, your method is no different from these fully-supervised baselines since both have access to the full training set but you use it in an indirect way deliberately.** \n\nOur focus in this paper is name extraction from escort ads for combatting HT. For HTName and HTGEN datasets, we do not use true labels for training the labeling functions (fine-tuned models). We will improve the writeup in the paper to better clarify that we do not have access to in-domain true labels. This is why we call it weakly supervised training. The labels used to fine-tune roberta/deberta on HTUNSUP and HTGEN are weak labels obtained from ChatGPT. The performance of only using those trained models for our task is reported in Table 4 (e.g. 0.67 for deberta on HTUnsup), which is significantly lower than what we achieve with SWEET (0.87). NEAT, the previously state-of-the-art model in this domain, does depend on a training set of HT data directly from the main target dataset HTName. We compare against the previously best model in such a setting, and show our approach gives better performance, even in this “unfair” comparison setting where NEAT gets to use in-distribution labeled training data and our method does not.\n\nIn table 5, second row, we report the performance of SWEET without having access to any domain data, i.e, performance on HTName without having the labeling functions that were fine-tuned with ChatGPT-labeled HT data. Similarly the results for CoNLL and WNUT are noted after removing the labeling functions that were fine tuned on the training split of the CoNLL and WNUT datasets respectively. We can see that SWEET without domain data still performs as good (or even better in case of WNUT), compared to when it has the target domain labeling functions (SWEET in table 3). \n\n**Q2. Admittedly, human trafficking is an important problem but I don’t see how extracting person names can help largely help. I personally think NER on noisy text is a pretty general and important task so I would love to see this method be applied to general domains(in the correct setting).** \n\nThis paper presents an approach for NER for HT, which is a part of our bigger project on finding HT in escort ads by looking for organized activity which advertises multiple people. A majority of traffickers have multiple victims, and hence one major indication of HT is the presence of multiple people being advertised within the same ad or similar groups of ads[1, 7]. If an ad or a group of connected ads advertise multiple names, it is likely that the activity is related to trafficking. This is considered along with other posting patterns (geographical movements, temporal burstiness, etc.). Extracting the advertised names from these ads is critical to scalably flag such suspicious ads for further investigation. We agree that NER on noisy text in general is an important task but it is not our current focus. The focus of this work is to show how LFs trained on non-domain data can be better leveraged to efficiently extract names from noisy escort ads. \n\n**Q3. If the motivation is to address the NER problems in noisy texts, then maybe performing NER on truly noisy texts is a better idea instead of testing the methods on CoNLL2003 and WNUT 2017. Testing a model that has been indirectly trained using a training set is not considered generalization.** \n\nThe main goal of this paper is improving NER performance for the HT text which contains data that is sensitive and hard to manually label. We report the performance on CoNLL and WNUT to show our proposed method, unlike NEAT which is the previous state-of-the-art for NER on HT data, generalizes well to these commonly used benchmark datasets. This, along with the HTGen, helps with reproducibility of the results. We are extending our work to include more datasets used in noisy NER methods\n\nWe will improve the writing to better reflect these points in the revised version. \n\n**References:** \n\n[1] Li, Yifei, et al. \"Extracting Person Names from User Generated Text: Named-Entity Recognition for Combating Human Trafficking.\" Findings of the Association for Computational Linguistics: ACL 2022. 2022.\n\n[2] Ratner, Alexander J., et al. \"Snorkel: Fast training set generation for information extraction.\" Proceedings of the 2017 ACM international conference on management of data. 2017.\n\n[3] Pierre Lison, Jeremy Barnes and Aliaksandr Hubin (2021), \"skweak: Weak Supervision Made Easy for NLP\", ACL 2021 (System demonstrations).\n\n[4] Zhou, Jianlong, et al. \"Ethical ChatGPT: Concerns, challenges, and commandments.\" arXiv preprint arXiv:2305.10646 (2023).\n\n[5] Veselovsky, Veniamin, Manoel Horta Ribeiro, and Robert West. \"Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks.\" arXiv preprint arXiv:2306.07899 (2023).\n\n[6] Fabrizio Gilardi, Meysam Alizadeh, & Maël Kubli (2023). ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30), e2305016120.\n\n[7] U.N. (2021). U.N News. Traffickers abusing online technology, UN crime prevention agency warns.\n\n[8] Chen, Lingjiao, Matei Zaharia, and James Zou. \"How is ChatGPT's behavior changing over time?.\" arXiv preprint arXiv:2307.09009 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work has some novelty: transferring the knowledge from exisiting NER models/datasets for noisy text NER tasks.\n\n2. The results seem good.", "reasons_to_reject": "1. The current setting, which involves using the model (fine-tuned roberta/deberta) trained on HTUNSUP and HTGEN to indirectly teach another model (HMM), should not be referred to as weak supervision. If you have access to the train sets of HTUNSUP and HTGEN, it would be more performant to directly train a model on them or compare your HMM method with other baselines that fully utilize these training sets. Essentially, your method is no different from these fully-supervised baselines since both have access to the full training set but  you use it in an indirect way deliberately.\n\n2. The writing can be largely improved and I can't follow well.\n\n3. Admittedly, human trafficking is an important problem but I don’t see how extracting person names can help largely help. I personally think NER on noisy text is a pretty general and important task so I would love to see this method be applied to general domains(in the correct setting).\n\n4. If the motivation is to address the NER problems in noisy texts, then maybe performing NER on truly noisy texts is a better idea instead of testing the methods on CoNLL2003 and WNUT2017. Testing a model that has been indirectly trained using a training set is not considered generalization.", "questions_to_authors": "No", "paper_topic_and_main_contributions": "To address the human trafficking problem, this paper proposes a weakly supervised method to extract person names from noisy texts. Specifically, utilize heuristic rules to filter out false positive entities, and employ models tuned on existing NER datasets for silver data generation. This silver data will then be used for training on human trafficking data at a later stage.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No", "Typos_Grammar_Style_and_Presentation_Improvements": "No"}, "extra_text": {}}, {"review_id": "evjnBWNaZC", "reviewer": "Reviewer_6zce", "summary": "", "questions": "1.Since ChatGPT is capable of achieving optimal results on tasks and can effectively explain prediction outcomes, could this influence the further research value of the task?\n\n2.In the paper, the authors mentioned using the Skweak framework for weakly supervised learning, but they did not elaborate on why they chose the Skweak framework. I would like to know the advantages of Skweak compared to other weakly supervised frameworks and how it contributes to improving the performance of the SWEET method.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their remarks and hope to answer their questions below:\n\n**Q1. The methods mentioned in the paper mainly involve aggregating existing approaches, lacking originality, and there is no clear explanation of why these methods can address data noise issues in Person Name Extraction.**\n\nThe noisy nature of the data demonstrates the challenges related to annotation and explains why the existing approaches underperform. In our real-world HT domain, where it is hard to access labeled data and the text is noisy and unstructured, we show that treating the labels from existing approaches and aggregating them in a weak supervision framework can achieve superior and robust performance, as our experiments confirm this. \n\nIn particular, we provide an interesting and novel ensemble approach of combining multiple classifiers that leverages 6 different datasets (ConLL, WNUT2017, Few-NERD-LI, WikiNER-en, HTUnsup, and HTGen) while providing both improved results and the ability to work in a transfer-learning setup. \n\nBy leveraging multiple signals our model can easily adapt to different types of data which contributes to the high performance on both the HT domain data and general settings. Moreover, we present a way to generate noisy data that matches the domain and is labeled automatically with ChatGPT, to achieve better performance on the in-domain HTName dataset. We will highlight the novelty better in the revised version of the paper. \n\n\n**Q2. The paper only tested ChatGPT on the HTName dataset and did not evaluate it on other datasets. However, ChatGPT’s performance on the HTName dataset outperformed the proposed method in the paper, indicating that the proposed method is not state-of-the-art (SOTA).** \n\nAlthough ChatGPT is the best performing model, there are concerns regarding its use on escort ads. First, there is some financial **cost** associated with employing ChatGPT on large datasets of escort ads which limits its accessibility whereas our method is open-source and completely free to use. Second, the **stability** of the OpenAI API output is a concern as discussed in [8]. We have found that even the performance of the 0613 versioned GPT-3.5 endpoint can vary and result in performance worse than BERT and our method. While this does not hurt our method much where one only needs to get good performance a single time (to provide weak labels on a fixed dataset), and one could consider even the more powerful GPT-4, it means it would be unreliable to build a system for real-world applications based on ChatGPT. Finally, depending on the method used to access ChatGPT, there can be **privacy** concerns (for example, it can be used in HIPAA compliant way - i.e., to the standard of US health data privacy regulation - by accessing it through Azure, but that requires approval, and the standard OpenAI endpoint is not HIPAA compliant) with this sensitive data. The intended use of our method is on large scale real-time ads with sensitive information, and using ChatGPT, is not feasible due to cost, stability and privacy issues mentioned above. We appreciate the reviewer’s comment and will edit our discussion section to better include these points in the revised version.\n\n**Q3. The paper used relatively few of the latest baselines, and compared to these baselines, the improvements brought by the proposed method in the paper are limited.** \n\nWe respectfully disagree and highlight that SWEET gains a 9% improvement on F1-score over baselines on the main domain dataset - HTName. Our method beats NEAT [1], the previous SOTA in this domain and hence we excluded some of the baselines from [1]. Additionally, the performance of individual fine-tuned LMs which are used as LFs are provided in Table 4 which also act as recent baselines where SWEET’s F1-score (0.87) is higher than the best performing LF (0.82). \n\n**Q4. Table 3 is missing the evaluation results of SWEET on the HTGen dataset.**\n\nSWEET leverages LFs which includes roberta and deberta models fine-tuned on both general NER datasets and domain-specific HTGen. Hence, evaluating our method on HTGen would not be a fair comparison given our method has seen all the HTGEN data. However, in Table 6, we provide the results of *SWEET - HTGen LFs* on HTGen (0.97 F1) which performs on par with NEAT in Table 3 but with higher precision. In the next version of the paper, we plan to add a small test set for HTGen, which is not used for fine-tuning the labeling functions in SWEET and have that entry filled in Table 3, to reduce confusion. Thank you for your feedback.\n\n**Q5.Since ChatGPT is capable of achieving optimal results on tasks and can effectively explain prediction outcomes, could this influence the further research value of the task?**  \n\nWe thank the reviewer for bringing up this interesting discussion point. \n\nWith the advent of ChatGPT, we believe that this question may be raised for several NLP tasks. For the task at hand, some of the concerns regarding the usage of ChatGPT have been addressed in our response to the above question 2. Moreover, for the given task, we believe that a model like ChatGPT is an overkill and we propose leveraging smaller LLMs with much fewer parameters (and training cost and computational requirements) and show that it provides competitive results. NER results are also easily explainable as opposed to more complicated tasks which may benefit from the explanations that ChatGPT is able to provide, for example, misinformation detection. \nLastly, there are studies [4] investigating ethical concerns and challenges of using ChatGPT which need to be further understood before relying on it completely for solving similar real-world problems.\n\n**Q6. In the paper, the authors mentioned using the Skweak framework for weakly supervised learning, but they did not elaborate on why they chose the Skweak framework. I would like to know the advantages of Skweak compared to other weakly supervised frameworks and how it contributes to improving the performance of the SWEET method.**\n\nSnorkel[2] and Skweak [3] are the two most popular frameworks for programmatic weak supervision. Skweak (2021) is a newer toolkit than Snorkel (2017) and was specifically designed for NLP tasks. Thus it is more fitting for our HT NER task. Skweak was also built using SpaCy, which facilitates downstream processing. Skweak’s support for Spacy also made it easier to convert the NEAT components to labeling functions for the *NEAT (Weakly Supervised)* experiment mentioned in Table 3. Moreover, Skweak provides a variety of different labeling function types: heuristics, Gazetteers and Document-level functions which allow us to combine weak signals from both rule based systems and LLMs. We will include this explanation in the revised version of the paper. \n\n**References:**\n\n[1] Li, Yifei, et al. \"Extracting Person Names from User Generated Text: Named-Entity Recognition for Combating Human Trafficking.\" Findings of the Association for Computational Linguistics: ACL 2022. 2022.\n\n[2] Ratner, Alexander J., et al. \"Snorkel: Fast training set generation for information extraction.\" Proceedings of the 2017 ACM international conference on management of data. 2017.\n\n[3] Pierre Lison, Jeremy Barnes and Aliaksandr Hubin (2021), \"skweak: Weak Supervision Made Easy for NLP\", ACL 2021 (System demonstrations).\n\n[4] Zhou, Jianlong, et al. \"Ethical ChatGPT: Concerns, challenges, and commandments.\" arXiv preprint arXiv:2305.10646 (2023).\n\n[5] Veselovsky, Veniamin, Manoel Horta Ribeiro, and Robert West. \"Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks.\" arXiv preprint arXiv:2306.07899 (2023).\n\n[6] Fabrizio Gilardi, Meysam Alizadeh, & Maël Kubli (2023). ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30), e2305016120.\n\n[7] U.N. (2021). U.N News. Traffickers abusing online technology, UN crime prevention agency warns.\n\n[8] Chen, Lingjiao, Matei Zaharia, and James Zou. \"How is ChatGPT's behavior changing over time?.\" arXiv preprint arXiv:2307.09009 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Introducing a novel solution: SWEET, as a weak supervision method, offers a fresh approach to the task of extracting person names from escort ads. Its efficiency and effectiveness surpass existing methods.\n\n2. Providing a shareable dataset: The development of the HTGEN dataset offers a collection of synthetic escort ad data that can be shared with the research community.\n", "reasons_to_reject": "1. The methods mentioned in the paper mainly involve aggregating existing approaches, lacking originality, and there is no clear explanation of why these methods can address data noise issues in Person Name Extraction.\n\n2. The paper only tested ChatGPT on the HTName dataset and did not evaluate it on other datasets. However, ChatGPT's performance on the HTName dataset outperformed the proposed method in the paper, indicating that the proposed method is not state-of-the-art (SOTA).\n\n3. The paper used relatively few of the latest baselines, and compared to these baselines, the improvements brought by the proposed method in the paper are limited.\n\n4. Table 3 is missing the evaluation results of SWEET on the HTGen dataset.", "questions_to_authors": "1.Since ChatGPT is capable of achieving optimal results on tasks and can effectively explain prediction outcomes, could this influence the further research value of the task?\n\n2.In the paper, the authors mentioned using the Skweak framework for weakly supervised learning, but they did not elaborate on why they chose the Skweak framework. I would like to know the advantages of Skweak compared to other weakly supervised frameworks and how it contributes to improving the performance of the SWEET method.", "paper_topic_and_main_contributions": "The topic of this paper is to extract person name entities from escort advertisements using weak supervision methods. The paper introduces the SWEET method to address the problem and with up to a 9% increase in F1 score compared to the current SOTA method on benchmark datasets.To facilitate further research in this domain, the paper also introduces the HTGEN dataset, which is generated by having ChatGPT synthesize synthetic escort ads. Unlike real-world datasets, HTGEN is shareable, contributing to research reproducibility and sharing.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a new approach for extracting person names from the noisy texts. It used heuristic rules and labeling functions the enhance the labeling results. The experimental results show that the F1 scores are increased by 9% compared to the SOTA methods. The paper is well-organized and easy to follow, except for the writing. The baselines and settings are not very suitable. For example, only texting the performance with ChatGPT on the HTName. Conducting more experiments will make the paper more convincing.
Decision: Accept-Findings