Toward a Critical Toponymy Framework for Named Entity Recognition: A Case Study of Airbnb in New York City
Authors: Mikael Brunila
Jack LaViolette
Sky CH-Wang
Priyanka Verma
Clara Féré
Grant McKenzie
Venue: EMNLP 2023 Main (2023)
Keywords: critical toponymy
named entity recognition
geographic information science
gentrification
new york city
airbnb
place
TL;DR: We introduce a NER model to extract toponyms for critical urbanism and the study of platforms like Airbnb.

Abstract: Critical toponymy examines the dynamics of power, capital, and resistance through place names and the sites to which they refer. Studies here have traditionally focused on the semantic content of toponyms and the top-down institutional processes that produce them. However, they have generally ignored the ways in which toponyms are used by ordinary people in everyday discourse, as well as the other strategies of geospatial description that accompany and contextualize toponymic reference. Here, we develop computational methods to measure how cultural and economic capital shape the ways in which people refer to places, through a novel annotated dataset of 47,440 New York City Airbnb listings from the 2010s. Building on this dataset, we introduce a new named entity recognition (NER) model able to identify important discourse categories integral to the characterization of place. Our findings point toward new directions for critical toponymy and to a range of previously understudied linguistic signals relevant to research on neighborhood status, housing and tourism markets, and gentrification.

Toward a Critical Toponymy Framework for Named Entity Recognition:
A Case Study of Airbnb in New York City
Mikael Brunila Jack La Sky CH-Wang
Priyanka Verma' Clara Féré' Grant Mckenzie
'Platial Analysis Lab, Department of Geography, McGill University
Urban Planning and Governance Lab, School of Urban Planning, McGill University
$Department of Sociology, Columbia University
A Department of Computer Science, Columbia University mikael. brunb w la@gmail com
Abstract
systems of power that link the two (Rose-Redwood et al., 2010).
Critical toponymy examines the dynamics of power, capital, and resistance through place names and the sites to which they refer. Studies here have traditionally focused on the semantic content of toponyms and the top-down institu- tional processes that produce them. However, they have generally ignored the ways in which toponyms are used by ordinary people in ev- eryday discourse, as well as the other strate- gies of geospatial description that accompany and contextualize toponymic reference. Here, we develop computational methods to measure how cultural and economic capital shape the ways in which people refer to places, through a novel annotated dataset of 17.440 New York City Airbnb listings from the 2010s. Building on this dataset, we introduce new named en- tity recognition (NER) model able to identify important discourse categories integral the characterization of place. Our findings point to- ward new directions for critical toponymy and a range of previously understudied linguistic signals relevant to research on neighborhood status, housing and tourism markets, and gen- trification.
We apply the critical toponymic perspective large dataset of annotated Airbnb listings in New York City, paying particular attention the socio- spatial circulation of neighborhood names, Airbnb hosts need to communicate aspects of the location of their listings to potential renters. Nearly all hosts rhetorically situate their units in set of spa- tial identities and relations, but their ways of doing so are diverse. Some use conventional neighbor- hood names (perhaps including nearby neighbor- hoods well), others describe proximity to nearby landmarks, while still others simply describe ac- cessibility to types of institutions and businesses such as hospitals, police stations, and restaurants. Spatial variation in these linguistic strategies of emplacement is the main object of our analysis. In other words, we ask: What can we learn about urban dynamics from the ways in which residents of different neighborhoods describe their property locations to prospective renters?
Whereas conventional toponymic analysis is usually limited to the semantic content of place names, we expand our focus include wide range of linguistic features that reflect spatial re- lationships (e.g., expressions of proximity, em- wtwpettvviitylllonsiiii formal
1 Introduction
Places are not only bounded regions and the ma- terial forms they contain, but are vested with and shaped by symbolic associations (e.g., Bell, 1997; Gieryn, 2000: Tuan, 1977). The names we use for places- toposstsoooossmm-m-s play a key role in the "pro- duction of space" (Lefebvre, 1991), stabilizing the social reality of both place associations and place boundaries in ways that typically reflect power dy- namics. The fact that New York City real estate developers have tried to rebrand parts of Harlem "SoHa" (Davidson and Fagundes, 2019), for example, suggests that the area's previous names were seen undesirable the tenants they hope to attract. Critical toponymy refers the field of research that takes as its focus this relationship be- tween places, their names, and the practices and
beddedness, and toponyms for neighborhoods, streets, landmarks, businesses, and so on. To extract references to place and spatial relations from unstructured listing descriptions, we train a custom named entity recog- nition (NER) model on novel, dataset to identify a range of toponyms and spatial relationships. We analyze these linguistic features alongside a range of sociodemographhic variables measured at the neighborhood level, demonstrating multiple associations between toponymic practices and neighborhood status.
In doing so, we offer a number of contribu- tions. First, we expand the methodological and

---

conceptual scope of critical toponymy. Nearly all toponymic studies have focused on the central- ized naming practices of elites such as mayors or large real estate developers, a "top-down" approach (Bigon, 2020, 3). By contrast, our dataset reflects the "bbtttomup"" toponymic practices of a larger and more diverse set of social actors at scale im- possible without modern computational techniques. In addition, our NER model allows expand the range of place-descriptive resources from the semantics of toponyms in isolation to more sub- tle, variegated, and relational linguistic strategies, while still retaining an orienting emphasis on neigh- borhood names, boundaries, and their relationship to neighborhood status and change. Thus, we are able to investigate not only what urban areas are called, but also in which socio-spatial contexts they are invoked by name in the first place.
not require, specialized models such as ours. Our training data and models are publicly available on Github.
Prior work
2.1 Critical toponymy studies and Airbub
Whereas early studies of toponymy were oriented toward the enumeration, etymology, and taxonomy of place names-wiide early practitioners likening the toponymist to "botanical collector" (Wright, 1929, 140) the "critical turn" (Rosc-Redwood Redwood et al., 2010; Medway and Warnaby, 2014) since the 1980s has shifted attention toward place nam- ing practices and their relation to social and po- litical life: ethnic tensions, regime changes, col- lective memory, commercialization, and so on. In imperial contexts, for example, colonizers often re- name(d) territories, cities, and streets to reflect their own ethnolinguistic background and political ideals (Carter, 2013; Wanjiru-Mwita and Giraut, 2020), while the re-imposition of indigenous toponyms is an early form of action by many post-liberation groups (Mamvura et al., 2018; Njoh, 2017; Wanjiru and Matsubara, 2017). In specifically urban con- texts, critical toponymy has examined street, neigh- borhood, and landmark names as they relate to poli- tics and diplomacy (Rusu, 2019; Sysiö et al., 2023), to the corporatization of public spaces (Light and Young, 2015) and to shifting neighborhood status hierarchies and gentrification (Masuda and Book- man, 2018; Madden, 2018).
Secondly, we contribute to the growing litera- ture on Airbab and housing dynamics. Others have shown the ways in which Airbnb and short-term rentals (STRs) more broadly accelerate gentrifica- tion, widen the "rent gap" (Smith, 1987), and re- move housing from long-term rental markets (Bar- ron et al., 2020; Ayouba et al., 2020; Horn and Merante, 2017; Wachsmuth and Weisler, 2018). Nevertheless, little attention has been paid to the linguistic strategies that mediate economic trans- actions between hosts and renters. By uncover- ing linguistic signals related to gentrification and neighborhood status, we offer findings that can be extended to other residential contexts as well - in particular where new avenues for "technologically and culturally driven" gentrification (Wachsmuth and Weisler, 2018) are being opened up by the growing sector of "platform real estate" (Fields and Rogers, 2021).
What nearly all of these studies have in common is focus on the ways in which powerfully situated actors make decisions about "official" place names: mayors renaming streets honor local heroes, cor- porations renaming sports stadiums, or real estate developers rebranding low-status neighborhoods in their efforts to attract residents to new apartment compounds. While such actors exert tremendous influence on the toponymic landscape, toponymic adoption by the urban population is neither guaran- teed (e.g., Hui, 2019) nor well studied (Light and Young, 2017).
Finally, offer two methodological contri- butions. First, we introduce a new schema for geospatial NER labeling and corresponding human-annotated dataset with accompanying mod- els. While this data is particular to New York City, our model is able to generalize beyond the anno- tated entities in the training data (see Appendix B). Second, we propose set of new lightweight but accurate methods for toponymy resolution and geospatial dependency parsing. To anticipate our results, our models considerably outperform off- the-shelf NER models on our data and task (see Table 1). This suggests that NER applications to social science at the local level will benefit from,
A major contribution here, therefore, is to ex- amine toponymic reference among thousands of Airbnb hosts and customers whose ways of inscrib- ing urban space do not necessarily conform to those of developers or city planners. While GIS scholars have studied Airbnb at length for example, its pat- terns of expansion (Gutiérrez et al., 2017) and role
htttps: thub. www.///wwwww..////c/c.ooo /airpmbb///aaabbbbbaaaaaabrbbeel /aiarbbbbbbabbbbeeeeee place-ner

---

Upper East Side
Roosovelt Island
Mott Baven
Fore Hills
Flushing
Eluilurat
Harlem
Long Isaad City
RegoPark
goodd
Fortham
City Bland
Throgt Neck
Woodhidde
Exst Harken
Moringalde Heights
Janiiic
Demars Bharwway
Mestwth
Sunnjaide
Astoria
War Henginn Heghts
Upper West Side
Times Square
Chelsea
Midtown
Rodd
West Village
Gromency
Murray
Flattron Distritt
Greenwich Village
Kitchen
Shurrmment
Browfovlll
Greempoint
East Village
Naho
Climas
Lower East Side
Sast
Soho
Willia
burg
Bushwick
Little Italy
CrownBeiiht
Groens
Baroigh Bank
Chinatown
York
Budiurd Slopvesant
Hern
Cide Conter
Downtown
Brooklyn
Prasipee
Lallers Gärdens
Tribeca
Vinegar HIR
Dumbe
Sooes
Eass Flatsoa
Bay Bedge
Canársie
Financ al District
Slope
FlelBush
Battery Park City
Wordace Terress
Kenwiggonn
Brighton
Brooklya Heights Cobbie Hill
Gowerus
Coney Jaland
Boorum Mill
Rocka Rockawlly Beach
Carroll Gerdens
Columbbi
RedBBloo
Figure 1: The network of cross neissbeeghboooood mentions in Airbub listings reflects various geographic, cultural economic relationships. Here, a weighted directed edge exists between neighborhoods A and equal to the number of listings located weights tend to reflect prestige and desirability, whereas modularity classes tend to reflect geographic relations. Image filtered to the giant component and nodes with degree at least 10.
that mention
its description. Colors represent modularity classes. Asymmetries edge
in gentrification (Wachsmuth and Weisler, 2018) nobody has of yet combined geospatial amalysis of Airbnbs with toponymic analysis. In so doing, we heed continued calls to situate embedding- based ap- proaches to language analysis alongside or within other data structures reflecting forms of, for exam- plc. spatiotemporal and cocio-interactional varia- tion (Bender and Koller, 2020; Brunila and La lette, 2022). While NLP methods have previously been used study Airbnb, vious work that extracts spatial entities and their relations from texts on the platform.
unstructured text and determining their geospatial referents process called "geo-parsing" (Jones and Purves, 2008, proeeett numerous prac- tical challenges. The case of New York neighbor- hoods illustrates this well. For starters, there no official set neighborhood names and boundaries in New York. While the NYC Department of Plan- ning offers its own map, it cautions that "neighbor- hood names are not officially designated," =2 while dataset maintained by the nonprofit BetaNYC notes that neighborhood "boundaries may over- lap, some neighborhoods may function as a micro- neighborhood within another neighborhood, or a larger district which can made up of multiple neighborhoods."5
unaware of pre-
Our approach contributes to greater understand- ing of how factors such gentrification mediate the extent which cultural signifiers adopted on the ground. Neighborhood names are partic- ularly suited this goal, as their boundaries are more ambiguous and fast-changing than other sorts toponyms. Further, Airbnb data is particularly apt for the critical toponymy perspective, given its simultaneous embeddedness in economic interac- tions and geospatial structure.
Another complication comes from the fact that even if there were a ground-truth dataset of neigh- borhood names and boundaries, they still need to be mapped observed tokens in the Airbnb data. Neighborhood names in general, and not least in New York, are often informal and vernacular, in- cluding truncations and abbreviations (such as FiDi for the Financial District), as well as multilingual
2.2 Topynymy and NLP
htttps: http::://wwwwwmt www..ncc.......o www.c............e www.gggo///t/eeeeeeeeggg ci ty- neighborhodds page
Beyond the theoretical considerations of critical to- ponymy, the tasks of extracting place names from
htttps: h //tata//aaaaaaee beta. hyttaanac/aaatstt
sediacities-myc-neiihe

---

Model
F1 on Toponyms
maintained by the data science non-profit BetaNYC due to its high granularity. For brevity, we refer to these as "canonical neighborhoods." though we recognize that other neighborhood boundaries ex- ist.
DistiRRoBERTI-CR
0.9256
spaCy (LOC+OPE+FAC+ORG)
0.6175
Stanza (LOC+GPEAFAC+ORG)
0.5107
Table The DistiRRoBERTa-RFF model fine-tuned on geospatial NER tags outperforms tfffth models at identifying locations in our test set. To align label schemes, all tags from our model identified as toponyms (tags starting with TN) were recoded to LOC. For the large spaCy model (en core sore_web____ [g) and the Stanford Stanza model, all tags identified LOC, GPE, FAC, and ORG were recoded as LOC.
3.2 Gentrification index
We adopt the "small area index of gentrification" dataset published as part of Johnson et al. (2022). The authors use changes in sociodemographhic vari- ables associated with gentrification from 2000 to 2016 measured at the Census tract level as the ba- sis of their index, which is derived via PCA and Bayesian spatial smoothing. We average these tract- level measures to the neighborhood level, as the obfuscation added to Airbnb coordinates in the form of jitter precludes tract-level analysis.
names (such as El Barrio for East Harlem) (Hu et al., 2019). In relatively informal written corpora such as Airbnb data, neighborhoods are frequently misspelled or abbreviated in non-standard ways (such as wh for Williamsburg).
4 Methods
Thus, from ponymic analysis requires two steps: toponymy detection and toponymy resolution (Wang al.. 2020; Jones and Purves, 2008). First, toponyms have to detected among the set of words com- prising a text, a variant the general task of named entity recognition. For specialized contexts such as ours, off-the-shhet NER models frequently fail to identify tokens of interest for the reasons men- tioned above (again, see Table 1). Secondly, ambi- guities introduced by the fact that different names can refer the same place, and that the same name can refer to different places, are not necessarily identifiable from textual context alone and must be resolved. There is no gold-standard approach to toponymy resolution. Here, we synthesize multiple approaches as described in Section 4.4.
engineering perspective, to-
4.1 Annotation
To train our model, we developed a 21 21-category entity taxonomy and hand-labeled these categories across roughly 2,700 listings and reviews. 16 of these categories are toponymic, i.e., apply only when place is being referenced by name. These in- clude categories for entities such as neighborhoods, streets, transit stations, parks, and businesses. Four of the remaining five categories refer to non-named, generic references to types of institutions whose presence nevertheless meaningfully characterizes urban space. The final category refers not to places or things per se, but to expressions of proximity, distance, and adjacency. By including these final five non-toponymic cate- gories in our model, we expand the range of de- scriptive strategies people use to construct linguis- tic cartographies that our model is able address. Appendix defines all 21 categories and provides examples.
relations:
3 Data
3.1
Airbnb data & neighborhood shapefiles
Our primary dataset contains the 47,440 Airbnb listings that were active in New York City in August 2019, acquired from the nonprofit Inside Airbnb." Each listing is associated with its coordinates (with jitter drawn from skewed normal distribution with a mean of roughly 200m added), as well as several other variables not relevant here.
The schema was developed with initial reference to a set of categories suggested by (Cadorel et al., 2021), and was iteratively expanded and modified from their original 4 labels in Cadorel et al. to 14 labels in some of our preliminary work (Brunila et al., 2023), and finally to the 21 labels used here. All annotation was performed using Prodigy fol- lowing an initial training session where annotators collaboratively annotated a randomly chosen set of samples. This first round of annotation identi- fied points of ambiguity and disagreement. The
As mentioned, numerous neighborhood shape- files exist for New York. Here we opt for the 264 neighoigboodoood "NYC Neighborhoods" 5
hntpp://insideeiibbbbbb
chttps://data.. http///aaaaaaaaaa..eeeeeeeaeaaet http//aaaa..ttt.../dtattaaaaaeaa //t//ttsett////////
"https://prodi.gy:

---

Willansiagg
Bookyy
ADe
FROON
PROPN
PROPN
PROEN
COONT
ADV
PROPN
COONT
PROPS
PROPN
Figure 2: Visualization of the dependency parser. Sentences searched for dependencies between tokens, merging tokens belonging to the same NER label. The resulting directed graph is filtered for instances where spatio-lemporal entities refer toponyms.
last round of annotation, on which the final model was trained, involved re-labeling all roughly 2,700 training examples and was performed by the two lead coders only.
reflecting these relations Spatio-Temporal Entities (STEs). STEs such as "5 minutes from" or "walk- ing distance" were tagged along with toponyms and other spatial entities. However, move beyond bag-of-words relationship between tags, also parse dependencies between STEs and toponyms. First, we label our corpus using the NER model described above. Next, we split each document into sentences and parse for dependencies between tokens using spaCy's transition-bass parser. If a token also has NER label, it is merged with any token belonging the same IOB-chunk, inheriting all dependencies from its individual kens. All tokens merged into entities and the re- maining non-entity tokens effectively form a di- rected graph, which is filtered for any nodes that are labeled "STE"; all dependencies that point to these nodes are removed.
Inter-annotator agreement was examined across 107 listings (a random stratified sample achieve a sufficiently representative spatial distribution). Both lead coders separately annotated each doc- ument, and then each unique span tagged by ei- ther author was extracted (N=1,554). Treating one coder's is tags as "true" and the other's as "predicted" yielded a weighted Fl score of 0.822 across all label categories. This measure of agreement is arguably conservative insofar it requires exact span matches, e.g., "The MoMA" and "MoMA" would be treated as divergent predictions in the calculation of the Fl score.
dependency
4.2 Detecting spatial language
What remains is set of weakly connected sub- graphs that each have at most one STE node and n nodes with other labels, including toponyms. If any of these nodes is a toponym, the STE must refer to it, yielding final set of individual STE nodes and their dependent toponyms. This process is illustrated in Figure 2, where the sentence is ini- tially one graph, that is then split into subgraphs at "Walking distance to" and "very close to",, both with several dependent toponyms.
We evaluated three fine-tuned models on Distil- RoBERTa embeddings (Sanh et al., 2019; Liu et al., 2019) of our Airbnb text data: 1) linear classi- fication, 2) a Conditional Random Fields (CRF) model (Lafferty et al., 2001) and 3) and CRF- BiLSTM model (Huang al., 2015). Additionally, we run few-shot in-context learning experiment prompting ChatGPT to see how our custom mod- els compare in performance to a larger LLM with- out fine-tuning. Models were given training data that had additionally been processed using 1OB- chunking (Ramshaw and Marcus, 1999). With the DistilRoBERTT-CRRE performing the best (with an F1-score of 0.814 the validation set and 0.812 the test set), we report all following downstream results with it (for a full comparison of models and details, see Appendix A).
4.4 Resolving spatial language
To match spans that were tagged neighborhoods outside of the canonical set to the selfsame, we develop a lightweight method for toponymy reso- lution. First, out of all neighborhood toponyms identified by our model but outside the canonical set, we keep those that are dependents of an STE (see subsection 4.3 and Figure 2). if and only if that STE is generally synonymous with "in," in- cluding expressions such "in the heart of" (see
4.3 Finding spatial dependencies
Listings frequently discuss distance and travel be- tween places. These relations are essential to a full picture of toponymic reference. We call tokens
/https://sppcy.tt io/api /

---

Appendix for the full list). Second, the loca- tions of the listings mentioning these toponyms are then used as input for Kernel Density Estima- tion (KDE) model that filters out locations more than two standard deviations from the mean of the distribution. Thirdly, the remaining listings yield a convex hull for the span of each unique, "non- canonical" toponym. Finally, the nearest cen- troids of canonical neighborhood hulls are selected for closer analysis. Out of the n nearest canoni- cal toponyms, we next examine which are: (a) the k nearest neighbors in terms of cosine similarity using both word2vec (Mikolov et al., 2013) and fastText (Bojanowski et al., 2017) models trained on the listing and review texts, and (b) the m near- est neighbors in terms of Jaro-Winkler (Jaro, 1989; Winkler, 1990) distance, i.e. in terms of spelling similarity. Then, each neighborhood toponym out- side the canonical set is assigned the canonical toponym that scores best on these ensemble crite- ria. To validate the findings of this paper, we finally also went over this list and corrected it manually (for further details and Fl -coors,,, see Appendix C)
listing's location by neighborhood name, but they claim to be in neighborhood other than what our geometries assume. This observation is sugges- tive of the idea of "vague cognitive regions" in geographical research: the variable ways that peo- plc categorize and break up geographic space, both cognitively and in discourse (Gao et al., 2017; Mon- tello al., 2014).
To investigate how cultural factors shape these cognitive regions, regardless of their relationship to canonical neighborhood boundaries, we proceed to a second analysis in which we use the spatio- temporal entity class of our NER model to induce toponymic spans.
5.2 Toponymic span
We analyze toponymic span by asking two questions: to what extent does the geospatial span of claims to neighborhood membership and proximity-egg. located in Midlown, close to Greenpoint-dee differ from canonical neighborhood boundaries, and how does this vary as a function of neighborhood status?
This analysis makes use of two NER categories: spatio-temporal entities (STEs) and neighborhood toponyms. Using dependency parsing (Figure 2). we identify when a neighborhood toponym oc- curs as the syntactic child of any STE that indi- cate membership: in the heart of. central to, etc. (see Appendix F). This allows us to plot the ge- ographic span of claims to neighborhood mem- bership against the coordinates of corresponding Airbnb units.
5 A Analysi
5.1
Toponymic self-reference
We begin with a simple demonstration that vali- dates relationship between neighborhood names and urban geospatial structure. To do so, we ask how frequently Airbnb listings in different neigh- borhoods mention their neighborhood by name, using toponymy resolution to capture misspellings and alternate usages. such a relationship exists, we would expect listings in more central or other- wise desirable locations to invoke neighborhood names more frequently than those in less desir- able areas. Figure 3 (a) plots this relationship, and indeed we see that this is generally the case: neigh- borhoods at the fringes of the city toponymically self-reference much less frequently than central neighborhoods. In other words, we begin to see that urban dynamies such as centrality and periph- ery are inscribed in "bottom-up" toponymic prac- tices at scale. Furthermore, from Figure I we can see that listings also reference neighborhoods out- side of their own location, generating a toponymic hierarchi of sorts.
To calculate a neighborhood's toponymic span, we take the convex hull of the points corresponding to membership claims, again employing KDE to remove any listings two standard deviations away from the distribution mean (for KDE details, see Appendix C). We take the area of the convex hull of this resulting set of coordinates, which can be com- pared to the area of the canonical neighborhood, as can be seen in Figure (b).
Figure visualizes this process in more detail for two neighborhoods: Williamsburg and Bedford- Stuyvesant (Bed-Stty"). The two neighborhoods are similar in many ways: both are quite large, con- tain high rates of Airbnbs, and are adjacent to one another. Both are centrally located to North Brook- lyn and relatively well connected the city by subway. While each has been the site of gentrifica- tion, attracting many young "transplants" moving to New York, Williamsburg is widely cited as one
These measures entail a major shortcoming, how- ever: it assumes that the canonical neighborhoods we use align with the way Airbnb users imagine urban space. host might in fact refer to their

---

0.00 0.00
0.00
0.00
40.5
0:20 0.53
0.00 0:20
40.9
0.00
3.80
22.29
3.80
0.51. 8:70
22.24
$4.35
2020
136.74
40.5
40.6
(0) Parcentage 196) listings mention their neighborhood by name
Neighborhood extent relative to actual size.
Figure 3: (a) Percentage of Airbnbs in each canonical neighhorhood that refer to their neighborhood by name. The highest numbers are concentrated in particular parts Brooklyn and Manhattan. The neighborhoods on Staten Island (in yellow, lower left corner) that have very high ratio, also have only between the extent of neighborhood hull and its actual area.
listings. (b) The ratio
of the most intense sites of gentrification in the city for two decades (see, e.g., Curran, 2007) and has been dubbed "the original hipster breeding ground" 21st-cenyury Brooklyn (Schiermer, 2014, 170). In this sense, Bed-Stuy lags behind, still attracting young transplants but with 25% lower median rents (Johnson et al., 2022, data appendix) and gener- ally lacking the desirability and fame (or notoriety) of Williamsburg.
area is much higher for Williamsburg (6.929) than for Bed-Stuy (2.027), showing how local prestige can "stretch" an area's collective cognitive region. While the comparison of these two neighbor- hoods serves as useful demonstration, Fig- ure 5 (panel 3) shows that the Pearson correla- tion between toponymic span and gentrification- operationalized with mate-off-the-ar small-area index of gentrification (Johnson et al., 2022) persists across the dataset (r .35)9. The rightmost panel of Figure displays the negative correlation between gentrification and toponymic reference other than neighborhood names. In other words, neighborhoods which have yet to gentrify invoke toponyms to situate their listings, but they are more likely invoke other signposts- transit stations. businesses, airports, for example to communi- cate the location of their units.
These differences in prestige are revealed through practices of toponymic inscription at scale. The left column of Figure 4 displays all mentions of each neighborhood, without filtering for mem- bership claims ("located in"") or Kernel Density outliers. As we can see, Airbnb hosts across four city boroughs (excepting Staten Island) situate their locations with reference to Williamsburg, regard- less of how they do so. By contrast, references to Bed-Stuy are much more locally concentrated, suggesting the lesser prestige attached to the neigh- borhood name. After filtering for usages that occur in the context of membership claim and removing outliers with Mahalanobis filtering, the spans for both shrink considerably. Nevertheless, the ratio of the filtered span area to the underlying canonical
Discussion
We suggest that critical toponymy an under- explored theoretical framework with fruitful ap- plications in applied natural language processing. Mining text data to discover place names is an old task in data science (e.g., Twaroch et al., 2008). Re- cent innovations in embedding- based models have greatly improved our ability to infer toponyms in
*Johnson et al. (2022) uses different canonical boundaries than we do. In their data, we define "Bed-Stuy" as all tracts in what they call Bedford and Stuyvesant Heights, and "Williams- burg" as all tracts in Williamsburg and North Side-South Side.
"We got similar outcomes using I Local Outlier Factor, Ma- halanobis distance, and Isolation Forest to filter span outliers.

---

Conorical Wit
40.9
Using tist mont on WB
Convet tulll mentions
Comere
40.8
60.6
40.5
-74.2
-74.1
-74.0
-73.9
-73.8
-73.7
-74.2
-70.1
-74.0
-73.9
-73.0
-73.7
(a) listings that mention Williamsuur (WB)
Outlier fittred listings that claim be Williamsburgg (WB)
Canorical Its
Liiing mant mentoor
sorvex
40.8
40.7
40.5
-74.2
-74.1
-74.0
-73.9
-73.8
-73.7
-74.2 -74.1
-74.0
-73.9
-33.8
-73.7
listings Stuy (BS)
listings
Stuy (85)
Figure 4: Visualizing toponymic span. The left column compares the unfiltered spans, defined the complete complex hull area of all neighhorhood mentions, of two adjacent neighborhoods: Williamshurg (a) and Bed-Stuy (c). Williamsburg's s considerably larger total span suggests its perceived relevance to much of the city, whereas mentions of Bed-Stuy are more locally constrained. The right column depicts the convex hulls (Barber et al., 1996) after filtering mentions entailing claims of membership (e.g.. located in) and after filtering with Mahalanobis distance to remove outliers. Again, Williamsburg's 's (b) greater ratio of its filtered span area its canonical neighborhood area suggests its greater status as neighborhood compared to Bed-Stuy (d).
unstructured text data. Thus far, such work in the context of toponymy has generally been explored as an engineering challenge to improved upon with increasingly sophisticated methods (e.g., Car- doso et al., 2022: Davari et al., 2020; Fize et al., 2021: Tao et al., 2022). While such efforts are in- valuable to engineers and social scientists alike, few have extended these novel upproaches to concrete questions of social scientific intrigue.
to investigate the relationship between bottom-up toponymic practices and neighborhood status the context the Airbnb market. We demonstrate multiple ways in which the toponymic language reflects a variety of urban geospatial and sociocul- tural dynamics. Not all hosts locate their units in reference to their residential neighborhood. With some exceptions- require much more fine-grained discussion of the idiosyncrasies of New York City neighborhoods than appropriate here-more peripheral neigh-
exploration of which would
Here we offer preliminary steps in this direction, introducing novel data and a bespoke NER model

---

Correlations between gentrification index, Airbnb listings, and toponymic usage
- 0.44
0.21
0.35
p -0.55
Gentrificaion
1000 2000 Number of listings
3000
4000
-1
0.0 2.5
5,0 7.5
10.0
-2
0
2
Self-reference ratio
Neighborhood name span
Average non-n.hood toponyms
Figure 5: From left to right, using Pearson correlation: 1) The number of Airbabs in neighborhood positively corre- lates with the neighborhood's gentrification index, defined Johnson et al. (2022). The proportion Airbnbs that mention their neighborhood by name is also positively correlated with gentrification, is the neighborhoods toponymic span (3). 4) On the other hand, toponymic references categories other than neighborhood is negatively correlatted with gentification, suggesting that hosts turn other toponymic resources when their neighhorhoods lack cultural capital in the eyes of Airbnb consumers. Unlike in the natural sciences, Pearson correlation values in the +0.2-0.5 0.5 range are regularly reported as substantively meaningful associations (e.g., Cohen, 1988).
borhoods are less likely to use neighborhood names to situate their locations, suggesting how urban dynamics of center and periphery come to be pressed in toponymic reference scale,
is trained to identify: neighborhood toponyms and spatio-temporal entities. Future work should in- vestigate toponymic practices of different kinds. Larger text corpora, spanning longer periods of time, could reveal how these dynamics-for exam- ple, the relationship between toponym span and gentrification play out diachronically. Many con- ventional indicators of gentrification process are measured with Census data that may lag behind the on-the-ground experience and economic effects of gentrification. If such linguistic signals could be shown to capture gentrification dynamics be- fore they fully manifest in conventional sociodemo- graphic data, the importance of toponymic analysis in urban contexts would become all the more ap- parent.
Furthermore, we identify a relationship between gentrification dynamics and what we call toponymy spar: the ratio between the area within which peo- ple claim membership in neighborhood, and the area of its canonical boundaries. Given the demon- strated association between Airbnbs and gentrifica- tion (suggested by our data as well; see Figure leftmost panel), this points to the possibility of a circular process. As a neighborhood gentrifies and acquires desirability among class of largely white, middle- and upper-middle class, young profession- als, Airbnb hosts and guests at the geographic fringes of those neighborhoods become more likely locate themselves within it. Identifying them- selves with new desirable toponyms might proceed to attract guests with greater efficacy, increasing the 'rent gap" introduced by Airbnbs (Wachsmuth and Weisler, 2018) () and perhaps accelerating the gentri- fication processes set in motion by STRs. While such causal dynamics would be difficult to prop- crly model, and certainly do not do here, our findings in combination with prior research on the effect of Airbnbs suggest that toponymic practices might not merely reflect ongoing urban change, but play a more active role therein.
Finally, researchers in human-computer interac- tion could expand the scope of this research pro- gram. Qualitative analyses could add consider- able depth our understanding of how tourists, commuters, and prospective residents mobilize toponymic knowledge in the process of housing search practices, whether for short- or long-term rentals and home-buying. On the supply side, the same could be done for Airbnb hosts. Given the increasing professionalization of Airbnb hosting (Bosma, 2022; Dogru et al., 2020), the ways in which such practices vary across highly profession- alized (and sometimes corporatized) Airbub hosts compared with hosts who simply rent their spare rooms or apartments when they are away could merit attention as well.
7
Future work
Due to space limitations, here we primarily fo- cus on only two of the categories our model

---

Ethical Considerations
References
Data. Our data comes from Inside Airbnb, which describes itself as "a mission driven project that provides data and advocacy about Airbnb's 's impact on residential communities." Only public snap- shots from Airbnb are collected and analyzed, and obfuscation is present, to a certain extent, for lo- cation information on listings. Despite the public nature of our data, it is unreasonable to assume that users explicitly consent for their data to be col- lected and analyzed in this way. As is with most computational social science research conducted at scale, it is infeasible to obtain explicit user consent for large-scale datasets such as ours (Buchanan, 2017). Here, we believe that the benefits of our work in illuminating the ways which neighbor- hood dynamics are inscribed in toponymic practice outweigh its potential harms.
Kassoum Ayouba, Marie-Laure Breuillé, Camille Gri- vault, and Julie Le Gallo, 2020. Does Airbnb disrupt the private rental market? An empirical analysis for french cities. International Regional Science Review, 43(1 (3(1-2):76-044
C Bradford Barber, David P Dobkin, and Hannu Huh- danpaa. 1996. The quickhull algorithm for convex hulls. ACM Transactions on Mathematical Software (TOMS), 2294)3469-483
Kyle Barron, Edward Kung, and Davide Proserpio. 2020. The Effect of Home- Saaa-nh on House Prices and Rents: Evidence from Airbnb. Marketing Science, 40(1):23-47, Publisher: INFORMS.
Michael Mayerfeld Bell. 1997. The ghosts of place. Theory and society, 2666):81-8866.
Emily M Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understand- ing in the age data. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 5185-5198.
Limitations
We interpret this paper as a proof-of-concee that relates theoretical perspectives from human geogra- phy and critical toponymy to NLP modeling, point- ing towards numerous avenues for future research. In addition to the future work suggested above, our technical solutions could be improved in sev- eral ways. While our NER models achieved strong Fl-scores, larger annotated dataset would likely improve performance. Second, our results are spe- cific to New York, and further work is required to determine to what extent they generalize other cities. Third, we filtered out reviews and descrip- tions that were not estimated to be in English with a 0.95 or higher probability. Results might differ in sociologically substantial ways with a multilin- ugual approach. Fourth, the data we use is unevenly distributed across space, an issue that future work could address by incorporating spatial smoothing techniques.
Liora Bigon. 2020. Towards creating global urban toponymy: A comment. Urban Science, 4(4):75.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching Word Vectors with Subword Information. Transactions of the Associa- rion for Computational Linguistics, 5:1 5:135-146.
Jelke R Bosma. 2022. Platformed professionalization:
Labor, assets, and earning livelihood through airbub. Enviroument and Planning A: Economy and Space. 54(4):595-610.
Mikael Brunila and Jack VaVillette 2022. What com- pany do words keep? Revisiting the distributional se- mantics of J.R. Firth & Zellig Harris. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 4403-4417, Seattle, United States. Association for Computational Linguistics.
Mikael Brunila, Priyanka Verma, and Grant McKenzie. 2023. When Everything Is "Nearby": How Airbub Listings in New York City Exaggerate Proximity. In 12th International Conference on Geographic In- formation Science (GIScience 2023). volume 277 of Leibniz International Proceedings in Informat- ics (LIPIcs), pages 16:1-16:8, Dagstuhl, Germany. Schloss Dagstuhl Leibniz-Zentrum für Informatik. ISSN: 1868-8969.
Acknowledgements
Mikael Brunila has received funding from the Kone Foundation. Sky CH-Wang is supported by a Na- tional Science Foundation Graduate Research Fel- lowship under Grant No. DGE-2236197. We would like to thank the "NLP for Social Science" workshop participants from McGill and Columbia for their thoughtful feedback on preliminary re- sults.
Elizabeth Buchanan. 2017. Considering the ethics of big data research: A case of Twitter and ISIS/SIIL PloS One, 12(12):0187155
Lucie Cadorel, Alicia Blanchi, and Andrea G. B. Tet- tamanzi. 2021. Geospatial Knowledge in Housing Advertisements: Capturing and Extracting Spatial
htattp:/iinssddairbbb..oom

---

Information from Text. In Proceedings of the 11th on Knowledge Capture Conference, K-CAP '21, pages 41-48, New York, NY, USA. Association for Com- puting Machinery.
Keren Horn and Mark Merante. 2017. home sharing driving up rents? evidence from Airbnb in Boston. Journal of Housing Economics, 38: 14-24.
Yingjie Hu, Huina Mao, and Grant McKenzie, 2019. A natural language processing and geospatial clus- tering framework for harvesting local place names from geotagged housing advertisements. Interna- tional Journal of Geographical Information Science, 33(4):714-738.
Ana Bárbara Cardoso. Bruno Martins, and Jacinto Es- tima. 2022. A novel deep learning approach us- ing contextual embeddings for toponym resolution. ISPRS International Journal of Geo aeo-Innfrmattin, 11(1):28.
Paul Carter. 2013. The road to Botany Bay: An ex- ploration of landscape and history. University of Minnesota Press.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirec- tional LSTM-CRF Models for Sequence Tagging. McXiv:150.00191 [es].
Jacob Cohen. 1988. Statistical power analysis for the behavional sciences, 2nd ed edition. I. Erlbaum Associates, Hillsdale, N.J. OCLC: ocn844923079.
Dennis Lai Hang Hui. 2019. Geopolitics of toponymic inscription in Taiwan: Toponymic hegemony, poli- ticking and resistance. Geopolitics, 24(4):916 943.
Winifred Curran. 2007. 'from the frying pan the oven': Gentrification and the experience of indus- trial displacement in williamshurg, brooklyn. Urban Studies, 44(8):1427-144 1440.
Matthew A. Jaro. 1989. Advances in Record-Linkage Methodology as Applied to Matching the 1985 Cen- sus of Tampa, Florida. Journal of the American Sta- ristical Association, 84(406):414 420.
MohammadReza Davari, Leila Kosseim, and Tien Bui. 2020. Timbert: Toponym identifier for the medical domain based on bert. In Proceedings of the 28th International Conference on Computational Linguis- tics, pages 662-668.
Glen D Johnson, Melissa Checker, Scott Larson, and Hanish Kodali. 2022. A small area index of gentrifi- cation. applied to New York City. International Jour- nal of Geographical Information Science, 36(1):137- 157.
Nestor M Davidson and David Fagundes, 2019, Law and neighborhood names. Vanderbilt Law Review, 72:757-824.
Christopher B Jones and Ross S Purves. 2008. Geo- graphical information retrieval. International Jour- nal of Geographical Information Science, 22(3):219- 228.
Tarik Dogru, Makarand Mody, Courtney Suess, Nathan Line. and Mark Bonn. 2020. Airbnb 2.0: Is it a sharing economy platform a lodging corporation? Tourism Management, 78: 104049.
JD. Lafferty, A. McCallum, and F. N. Pereira. 2001. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. Pro- ceedings of the Eighteenth International Conference Machine Learning, ICML '01, pages 282-289,, San Francisco, CA, USA. Morgan Kaufmann Pub- lishers Inc.
Desiree Fields and Dallas Rogers. 2021. Towards a Critical Housing Studies Research Agenda on Platform Real Estate. Housing, Theory and So- ciety, 38(1):72-94. Publisher: Routledge _eprint: https//ddbaag//11111100444400000 10..1.00000103100000000000111110
Henri Lefebvre. 1991. The Production of Space. Ox- ford Blackwell, Oxford, UK. Translated by Donald Nicholson-Smithh
Jacques Fize, Ludovic Moncla, and Bruno Martins. 2021. Deep learning for toponym resolution: Geocoding based on pairs of toponyms. ISPRS Inter- national Journal of Geo- Ine-onnoonnnoon, 10(12):118.
Duncan Light and Craig Young. 2015. Toponymy commodity: Exploring the economic dimensions of urban place names. International Journal of Urban and Regional Research, 39(3):435
Song Gao, Krzysztof Janowiez, Daniel Montello, Yingjie Hu, Jiue-An Yang, Grant McKenzie, Yit- ing Ju, Li Gong, Benjamin Adams, and Bo Yan. 2017. A data-synthesis -drivn method for detect- ing and extracting vague cognitive regions. Interna- tional Journal of Geographical Information Science, 31(6): 1245-1271.
Duncan Light and Craig Young. 2017. The politics of toponymic continuity: The limits of change and the ongoing lives of street names. In The political life of urban streetscapes, pages 185-201. Routledge.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretrain- ing Approach. ArXiv: 1907.1 1692 [cs].
Thomas F Gieryn. 2000. A space for place in sociology. Annual Review of Sociology, 26(1) (646346964
Javier Gutiérrez, Juan Carlos tavo Romanillos, and María Henar Ealas-Olmedt 2017. The eruption of Airbnb in tourist cities: Com- paring spatial patterns of hotels and pecr-to-peer ac- commodation Barcelona. Tourism Management, 62:278-291.
Gus-
David J. Madden, 2018 Pushed off the map: Toponymy and the politics of place in New York City. Urban Studies, 55(8):1599- 1614. Publisher: Urban Studies Journal Limited.

---

Zvinashe Mamvura, Itai Muwati, and Davie Mutasa. 2018. 'Toponymic commemoration not for one sex': the gender politics of place renaming in Harare. African Identities, (6(4):4994443.
Liufeng Tao, Zhong Xie, Dexin Xu, Kai Ma. Qinjun Qiu, Shengyong Pan, and Bo Huang. 2022. Geo- graphic named entity recognition by employing natu- ral language processing and improved bert model. ISPRS International Journal of Geo-Information, 11(12):598.
Jeffrey R Masuda and Sonia Bookman. 2018. Neigh- bourhood branding and the right to the city. Progress in Human Geography, 42(2):165-1)
Yi-Fu Tuan. 1977. Space and place: experience. University Minnesota Press.
The perspective of
Dominic Medway and Gary Warnaby, 2014. What's in a Name? Place Branding and Toponymic Commod- ification. Environment and Planning A: Economy and Space, 46(1):153-167. Publisher: SAGE Publi- cations Ltd.
Florian A Twaroch. Philip D Smart, and Christopher B Jones, 2008. Mining the web to detect place names. In Proceedings of the 5th Workshop on Geographic Information Retrieval, pages 43 44.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed Representa- tions Words and Phrases and their Compositional- ity. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.
David Wachsmuth and Alexander Weisler. 2018. Airbnb and the rent gap: Gentrification through the sharing economy. Environment and planning A: economy and space, 50(6):1147-1170.
Jimin Wang, Yingjie Hu, and Kenneth Joseph. 2020. NeuroTPR: A neuro-net toponym recognition model for extracting locations from social media messages. Transactions in GIS, 24(3):719-
Daniel R Montello, Alinda Friedman, and Daniel W Phillips. 2014. Vague cognitive regions in geogra- phy and geographic information science. Interna- tional Journal of Geographical Information Science, (8(9):1802-188) 1820.
Melissa Wangui Wanjiru and Kosuke Matsubara. 2017. Street toponymy and the decolonisation of the ur- ban landscape in post-colonial Nairobi. Journal of Cultural Geography, 34(1):1-233
Ambe Njoh. 2017. Toponymic inscription as an instru- of power in Africa: The case of colonial and post-colonial Dakar and Nairobi. Journal of Asian and African Studies, 52(8):1174-1199
Melissa Wanjiru- Mwita and Frédéric Giraut. 2020. To- ponymy, pioneership, and the politics of ethnic hier- archies in the spatial organization of British colonial Nairobi. Urban Science, 4(1):6.
L. A. Ramshaw and M. P. Marcus. 1999. Text Chunk- ing Using Transformation- Based Learning. In Susan Armstrong. Kenneth Church, Pierre Isabelle, Sandra Manzi, Evelyne Tzoukermann, and David Yarowsky, editors, Natural Language Processing Using Very Large Corpora, Text, Speech and Language Tech- nology, pages 157-176. Springer Netherlands, Dor- drecht.
William E. Winkler. 1990. String Comparator Metrics and Enhanced Decision Rules in the Fellegi-Sunter Model of Record Linkage. Technical report. ERIC Number: ED325505.
John Kirtland Wright. 1929. The study of place names recent work and some possibilities. Geographical Review, 1901):140-144
Reuben Rose-Redwood Derek Alderman, and Maoz Azaryahu. 2010. Geographies of toponymic inscrip- tion: new directions in critical place-name studies. Progress in Human Geography, 34(4):453-470
Mihai Stelian Rusu. 2019, Shifting urban namescapes: Street name politics and toponymic change in a Ro- manian(ised) city. Journal of Historical Geography, 65:48-58.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistiBBRR,, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint 100000090..0.088.
Bjørn Schiermer. 2014. Late-modern hipsters: New tendencies in popular culture. Acta sociologica, 177-)81
Neil Smith. 1987. Gentrification and the Rent Gap. Annuls of the Association of American Geographers, 4763)462-465.
Timo Sysiö, Onat Ülker, and Nurevyan Tokgöz. 2023. Assembling critical toponymy of diplomacy: The case of Ankara, Turkey. Geopolitics, 28(1)44164338

---

Model
Precision
Recall
DistiRoBBRTTa-CC
0.807
0.784
0.832
DistiRRoBETTaCRE
0.814
0.803
0.826
DistiRoBBERTa-CREBBLLLSS
0.812
0.789
0.836
ChatGPT In-Context Leamming
0.398
0.506
0.328
Table 2: Among the four NER architectures implemented, DistilRoBERTa with CRF layer performed the best on the validation set.
NER Models
Table 2 shows the performance of the top model for each implemented architecture: (1) DistilRoBERTa with linear classifier, (2) with CRF classifier, (3) with CRF-BiLSTM classifier, and (4) ChatGPT with In- In-Context Learning. We ran all experiments using both the initial set of annotations well the second, corrected set of annotations. All custom models 10-4 learning rate, 10-5 weight decay, gradient clipping,
trained over five epochs with 2718 total examples,
80/10/10 sein-test-validation split, and early stopping. A grid search was done over dropout values from 0.3, batch sizes from 4 to 32, and hidden layer size from 100 to 400. Overall there was not a great difference in model performance within each architecture.
Among the models, DistilRoBERTa with CRF layer performed best, while ChatGPT lagged behind even with in-context examples. For the former architecture, the best performing model ran for five epochs, with a batch size of four, dropout of 0.3, and 300 hidden layers. The performance of this model on the test set and individual NER tags can be seen in Table 3. In addition to the tags in the table, the dataset contains the tag "TN-OTHER", which was ultimately excluded from the training process due to its infrequency and irrelevance.
N
N
N
Label
Annotated (all)
Predicted (listings)
Predicted (reviews)
F1
Recall
Precision
1
TN:AIRPORT
441
5839
4637
0.877
0.909
0.847
2
TN:BEACH*
46
361
147
0.609
0.538
0.7
3
TN:BOROUGH
2005
30599
45236
0.974
0.970
0.978
4
TN.BRIDGE, BR..RRGGTTTNNLL TUNNEL.
81
1688
1091
0.375
0.5
0.3
5
BUSUSESS
948
22720
9760
0.832
0.832
0.832
TN:CITY
1223
20601
$3001
0.890
0.906
0.875
7
HN.HOSPITAL
85
1021
219
0.769
0.833
0.7142
8
TN:NAT FEAT
64
2239
543
0.667
0.714
0.625
9
INNEEIGIBORIOOD
2792
67430
45131
0.937
0.955
0.919
10
TN:PARK
578
17224
9454
0.946
0.984
0.910
TN:REGION
332
655
0.462
0.375
0.6
12
SC:OOL
158
3491
1128
0.837
0.857
0.818
STATION
464
17124
6624
0.696
0.722
0.672
STREET
22623
8795
0.706
0.712
0.700
TOURIST A
812
19151
6228
0.713
0.662
0.773
GEOG ENTITY
9024
190203
289271
0.827
0.833
0.821
HOST BUILDING
8122
277925
12391
0.733
0.705
TRANSIT
4998
97983
108324
0.840
0.843
0.837
SPAT TEMP ENT
11485
358506
203545
0.789
0.758
20
WALK RUN BIKE
128
2915
7368
0.552
0.571
0.533
Overall
26777
769150
756132
0.812
0.830
0.795
NNo tags in tost sor, FJ, recall, preciston results from validation set.
Table 3: Summary of NER label frequencies in the training data and the overall data, as well as performance metrics (FI, recall, and precision) the DistiRRoBERTa-CC model on the test set.

---

Index
Toponym
Frequency
1
spanish harlem
164
2
metropolitan museum of
162
3
domino, park madison_ avenue
152
4
150
5
wholefoods
129
6
rohertas
120)
cloisters
114
8
brooklyn_ Bpcllnnpppl_____ library
105
9
north_wiile williamsburg
104
10
bedford
104
Table 4: The sample from the top toponyms that were predicted by the DistiRROBERTA--RRF NER model but that were not present in the training data.
B Out Of Data Performance
To demonstrate the capacity of our NER model to generalize beyond the training set, we look at some of the toponyms the model predicted from the rest of the Airbnb data. In total, the model finds 97, 908 toponyms that were not in the train set but were among the other listings and reviews. These include well-known museums, alternative neighborhood names ("Spanish Harlem" for East Harlem), partial areas of neighborhoods ("North Williamsburg"), and popular parks and restaurants, all with fairly high frequency in the data. While we did not perform extensive experiments on this task, Table shows some examples from the most common toponyms predicted by the model, providing some preliminary evidence that the model is indeed able to generalize broadly.
C KDE & Toponymy Resolution
All Kernel Density Estimation (KDE) models in this paper are fitted using the default models in scikit- learn." Specifically, we use Gaussian kernel with Euclidean distance and bandwidth of 1.0.
For the toponymy resolution pipeline, we consider the 20 nearest centroids, 100 nearest neighbors for both word2vec and fastText as well as the 100 nearest Jaro- Winkler neighbors. Some spans tagged as neighborhoods fall outside of the pipeline, because they are not used sufficient times to form a convex hull. All in all, this pipeline achieves precision of 0.745, recall of 0.984, and Fi-score of 0.848, These results were calculated dedulatteddee after first running the model, then correcting the pairing of canonical and non-canonical toponyms manually, and finally, comparing the original model output with the manually corrected results.
D Expanded Dataset Details
The following table describes each category of entity in our NER model. The prefix TN indicates that a category is toponymic rather than generic.
challenge to labeling training data that many tokens could fall into multiple categorics. There are at least two common reasons for this. One that the word or phrase refers to different things in different contexts. For example, many subway stations are named for their street or neighborhood (e.g.. Forest Avenue M station). In situations such as these, coders whole label the whole phrase as TN:STATION rather than labeling Forest Ave TN:STREET, since the author subway station. A second source of ambiguity is that word phrase could reasonably be placed into multiple categories. For example, Central Park is both park and tourist attraction. In situations such these, coders opted for the most specific category (TN:PARK in this example).
using it in the context of named
htttpp///sckkkeeeeeooo////aaaaee semesssssssss..ggssssseeeeeeeeis KernelDensity....mmmmmmmomm html

---

Category
Definition
Examples
INNNEIGHBORHHOOD
Any named neighborhood
Soho, UWS
IN-BOROUGHI
Any named borough
New York City, Jersey City Munhattan, Bk
TN:CITY
Any named city
TN-REGION
Any named larger than city
Hamptons, Upstate
TN:STREET
Any named street
Broadway, Fifth Ave
TN:RRIDGE TUNNEL
Any named bridge tunnel
Brooklyn Bridge, Verrazano
IN:STATION
Any named transit center
Grand Central, Penn Station
TN:AIRPORT
Any named airort
JFK, LaGuardia
TN:PARK
Any named park
Central Park. Riverside
IN:SCHOOL
Any named educational institute
Columbia, NYU
IN:BUSINESS
Any named business
Dunkin, CVS
TN:HOSPITAL
Any named business
Dunkin, CVS
TN:TOURIST ATTR
Any named tourist muscum, landmark, etc.
the Met. Statue Liberty
TN:BEACH
Any named beach
Jacob Riis Park, Rockaways
TN:NAT FEAT
Any named natural feature
Atlantic, Fast River
TN:OTHER
Toponyms poorly captured in other categories
zip codes
GEOG_ENTITY
GEOG ENTITY
Any generic institution or other entity the environment
grncery stores, beaches
TRANSIT
Any generic reference to transit options
F train, the bus walk, bike
WALK RUN BIKE
Any generic reference to walking, running. biking as leisure activities
SPAT TEMP ENT
Any expression of spatiotemporal relation Any reference to the host's 'ss' unit as whole, or to amenities associated with the property but outside of the unit
25 minutes from, 3 blocks away 25 minutes from, 3 blocks away
HOST BUILDING
E Complete Annotation Guidelines
This document describes the process for annotating Airbnb listings and comments order create training data for a named entity recognition (NER) model. The goal to use this model to automatically extract many sorts of references to place these listings. There are 15 categories of spatial reference, and the rules for identifying which is appropriate are listed below. Before detailing the categories, there are three universal ground rules for determining annotations:
Always omit determiners from noun phrases.
i... tag Metropolitan Museum of Art, not The Metropolitan Museum of Art; grocery stores instead of few grocery stores; corner of 29th and 6th instead of the corner of 29th and 6th. Inconsistency with this will harm model performance.
If a noun phrase contains an adjective, omit the adjective unless it is an essential part of the noun phrase.
"Essential" introduces bit of interpretation, but here are some guidelines: tag "Mexican restaurant," * "grocery store", or "cocktail bar", but omit the adjectives in phrases like "nice restaurants," "24/7 stores", or "fun bar".
Do not tag any references to the interior of the Airbnb unit itself.
We have a category, HOST BUILDING, which is for references to certain building features, but we don't want to tag things like bathroom attached to the bedroom, or anything else *inside* the apartment.
Annotation Categories. There are categories that start with the prefix TN. This stands for toponym. and means that the reference to a location or spatial relationship involves directly referencing specific identity by name. Walgreens, the Hudson River, The Whitney Museum, Central Park, 5th Avenue, NYU, and Brooklyn are all toponyms. TN categories are listed below, in no particular order:
INNEIGHBORHOOD

---

These should be mostly captured automatically by list of neighborhood names, but things like misspellings might require human annotation. If the text modifies the neighborhood with words like downtown/uptown upper/lower. include those words in the description (i.e., Downtown Flushing, Lower Manhattan should be tagged IN:NEIGHBORHOOD)
IN:STATION
TN:STATION refers specifically to specific transit stations (e.g.. the Bergen 2/3 stop, Myrtle- Willoughby Station, Grand Central Station)
-References to subway, bus, ferry lines without mentioning a specific stop (e.g., the 6 train) should be tagged as TRANSIT, not TN:STATION
Should include well known stations that could potentially be TN:TOURIST ATTR (e.g. Grand Central) and train stations (e.g. Penn Station)
TN:CITY
This will almost always be New York City, NYC, New York/NY (when clearly referencing the city and not the state)
TN.BOROUGH
Any reference to borough. These should mostly be automated, but misspellings shorthands (e.g.. BK for Brooklyn) may need be done by hand
TN:PARK
Any named park: Central Park, Prospect Park, Greenwood, etc. Some neighborhoods in NYC have the word Park in them (e.g.. Ozone Park), make sure to double check you are unsure.
TN:SCHOOL
- These will mostly be colleges and universities CClumbia NYU, Fordham, CUNY, etc. thouht references to other schools should also be marked with this.
INNTOURIST_ATTR
Primarily museums, performance venues, and landmarks (Statue of Liberty). Other things that could be considered tourist attractions, such as universitics or famous stores, should be marked in those respective categories rather than the more general SUURISTAATTE
Ambigious cases that should be tagged here:
The Highline
Apollo Theater, etc.
All botanical gardens and zoos
TN:STREET
Street names; street corners (c.g., "29th and 5th Ave" should be split ("29th" and "5th Ave" as separate street names)
Include "square" and "Sq."
Other ambiguous cases that should be tagged here include Columbus Circle.
TN::USINESS
Any named business: CVS, Walgreens, Domino's, Macy's 's etc.
TN:OTHER
- This is a grab-bag category.

---

Zip codes
TN:AIRPORT
Any named reference to JFK, LGA, EWR, or other airports
TN:REGION
Any toponymic area larger than city: counties, subregions ("The Hamptons"), states, countries -"NY" can be ambiguous to the city or the state; use judgment according to sentence of context
TN:BEACH
Any toponymic beach. The difficulty here is that many beaches also share name with the neighborhood. The coder will have to use judgment to determine whether it is the beach itself the neighborhood being referenced. We expect the majority of usages to refer to the beach
SEABRIDGETTUNNEE
Any toponymic reference bridges and tunnels (e.g., Holland Tunnel, Williamsburg Bridge), Brooklyn Bridge, although tourist attraction, should be labeled EN.BRIDGE_TUNNE
TOURIST_ATTE
TN.HOSPITAL
Any toponymic reference hospitals. Include university hospitals (e.g., Columbia University Medical Center/CUMC)
"IN:NAT_FEATE FEATS
Forests, named places for hiking, mountains, etc.
Bodies of water (e.g. Hudson River, the East River)
NOT man-made parks inside the city; that would be TN:PARK
The remaining categories do not refer to specific, unique, named entities, but generally to categories of things.
SPAT TEMP ENT
Short for spatio-temppral entity. This is probably the most frequent category, and is used denote words and phrases describing proximity, distance, adjacency, location, and so on.
In general, these will be variations prepositional phrases: located the corner of, situated the intersection of, just ten minute walk from, five minutes by bike to, less than 20 minutes from, two subway stops away, within walking distance, only minutes away, between 9th and 10th ave, easy access to
When distances and times are modified (e.g.. roughly 20 minutes away, just around the
corner), include these modifying words
Occasionally could be adjectives: in central location
Include "view of", "see all of" (e.g. see all of Manhattan from my window)
If mode of transit is invoked
TRANSIT
- -oon-toponymic references to transit. This will often be names of subway and bus lines without the stop (the I train; nearby the F / G lines) or to the mode of transit general: subway, subway stop, bus, bus stop, PATH, etc.
WALK_RUN BIKE

---

Walking, running and biking an activity that's covered by STE
E
Also one of the most common categories. Geographic entities comprise most other generic references certain places/establismmntt.. Often these will be "things to do": parks, restaurants, bars, shopping malls, etc. Other times it will be more abstract: quiet neighborhood, on a bustling street, in a residential area
Trees, street parking
Tag "community" here if it's synonymous to neighborhood. (e.g. East Harlem is still quite a poor community.)
BUILDING
References to the building in which the unit is located. This can be reference to the building spatialized amenities of the building but outside
itself (located in historic brownstone), the unit itself: guest access the rooftop terrace.
OR anything that implicitly refers to the building in its entirety (the room is close to the subway) OR, references to amenities associated with the building that are spaces/that one can spend time in: backyards, roofs, patios, gyms (but not washing machines, kitchens, or anything else INSIDE the apartment itself)
-OR interfaces between world and building (e.g. there is a secure door with a code)
Includes features such as "stoop"

---

F Inductively generated STE set indicating membership
the center of
in center of
in
in this part of centrally located on
in the heart of
located
idcally located in the heart of the hub of
located in the heart of
right in the heart of
right the heart of
in the middle of
very right in the middle of
in the center of
located at the center of
centrally located in
located on the heart
located in the center of
truly the heart
conveniently located in
located in the western section of in great part of at the nexus of
nestled in the heart of
right in the middle of
situated in the heart of
conveniently located in the middle of within the heart of
at the heart of
conveniently located in the heart of
very centrally located in
located right the heart of
conveniently situated in
nestled in
within walking distance from the heart of
ideally located in
located in the heart
centrally located in the heart of
steps away
located in heart of
the south side of
located in the middle of
perfectly situated in the heart of
tucked away in
set in the heart of
perfectly located in
located in very convenient area of
located in the epicenter of
very conveniently located in
in the very heart of
the very center of
located in the best part of
locate in
perfectly located in the heart of
perfectly situated
located right in the middle of
best part of
in the epicenter
in the midst of
in the
steps away to
located within
short walk away from the heart of
remarkably located in the heart of
in great location in
absolute heart of
primely located in
close to the heart of
inside
located center of
convenient location in central area of

---

in prime position in
just a few steps away from
in the most desirable part of
in the lower part of
within steps of
footsteps away from
located in the south of
in the best part
located in the midst of
in the heat of located in part of
conveniently in
by the heart of
in middle of
located at the nexus of
in the coolest part of
in the northernmost portion of
only steps away from
located conveniently in
literally steps away from
strategically situated at the center of
located right in
nestled in the best part of
in the prime location of
convenient in
perfectly placed in the middle of


Reviews:
[{"review_id": "pEzhIQVilw", "reviewer": "Reviewer_QGQw", "summary": "", "questions": "A - Section 4.1 should ideally report on the agreement between the annotators of the NER data, even if only leveraging on a small sub-set of the annotated data.\n\nB - Section 4.2 mentions that the DistilBERT-CRF model performed best in the NER experiments, but the section does not present the actual F1 metric (i.e., in states \"an F1-score of XX\"). The authors should also explain why they haven't considered using a larger BERT model for producing the representations, or why they haven't considered fine-tuning the DistilBERT model directly for the NER task (e.g., the \"linear classifier\" approach seems to correspond to the training of a linear layer on top of the BERT representations for each token, and this setting could be easily extended to consider also the training of the DistilBERT model itself). Ideally, additional technical details about the training and inner-workings of the model(s) should also be given. For instance, what were the hyper-parameters considered for model training (this is only shown in Appendix A, but it is somewhat strange that different values for parameters such as the batch size were used for the different models)? How was span recognition modeled as a token-classification problem in terms of tags associated to the tokens (e.g., did the authors use a B-I-O tagging scheme, or some other approach)? How was the BERT tokenization scheme, based on sub-words, combined with the tokenization scheme used for creating the annotated data?\n\nC - On what regards Section 4.4, the authors mention the use of a Kernel Density Estimation procedure to filter out locations more than three standard deviations from the mean of their coordinates. This procedure should be presented in more detail (e.g., what KDE parameters, such as the kernel function or the kernel bandwidth, were considered? What do the authors mean with the expression \"mean of their coordinates\"?), and the authors should also justify this choice against some other procedure to remove outliers. In connection to this, the authors also mention the use of a convex hull to delineate the region formed by the coordinates, but a more accurate procedure could have been used instead (e.g., threshold the results of the KDE function, to keep the high density region, or using a different computational geometry approach such as alpha-shapes, which is not restricted to the formation of convex polygons).\n\nD - Also on what regards Section 4.4, which discusses methods for resolving spatial language, the authors mention the use of a \"canonical list of toponyms.\" This should be further detailed in the paper. Does the canonical list correspond to a sub-set of geonames entries? The entire contents of Section 4.4 are actually somewhat confusing (e.g., it is also not clear why and how different string similarity metrics are used), and perhaps this section should be re-written.\n\nE - The models used for detecting spatial language should ideally be evaluated comparatively to other existing approaches. I understand that the NER classes considered by the authors, useful for the type of study that they present, go beyond the classes that are considered in previous work. Still, it would be nice to see how the model compares to other approaches in terms of its ability to detect place names. Similarly, the method for \"resolving spatial language\" should also be compared against other alternatives (e.g., against existing open-source systems such as Mordecai).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors would like to thank all three reviewers for their thoughtful, constructive, and knowledgeable feedback. We appreciate the opportunity to offer our responses here. Below, we respond to each reviewer’s “Reasons to reject” and “Questions for the authors” sections. For the sake of brevity we do not respond to “Reasons to accept,” “Missing references,” and “Typos” feedback; reviewers can assume that we will incorporate all suggested references and correct all typos in the final draft. We reproduce each reviewer comment in quotation marks below.\n\nReasons to reject\n\n“The descriptions associated to the methods for \"resolving spatial language\" are somewhat confusing. This part of the proposed approach was also not quantitatively evaluated, and it may be the case that this component is introducing many errors.”\n\nWe agree with this feedback and will devote more space to the “resolving” methods description, including a performance comparison of alternative approaches in an additional appendix. We already used internally a list of neighborhood synonyms (e.g. “Spanish Harlem” and “El Barrio” for East Harlem, or common misspellings like “Bushwhick”) to validate our model. We will motivate our choice of synonyms and include the list along with results for our chosen methods and some common alternatives for spatial outlier detection (e.g. Robust Covariance, SVM, Local Outlier Factor) in this appendix.\n\n“Some of the choices regarding the methods for \"detecting spatial language\" should be better justified in the paper.”\n\nThe final draft will include more pointed justifications for our methodological choices, grounded in the appendix outlined above as well as relevant literature\n\nQuestions for the authors\n\n“A - Section 4.1 should ideally report on the agreement between the annotators of the NER data, even if only leveraging on a small sub-set of the annotated data.”\n\nThis is a fair point. The two main coders spent several meetings and considerable time together developing the coding scheme and protocols, including frequent discussions of edge cases during the coding process. However, we agree that an intercoder reliability test would be appropriate and will do so on a small sample as suggested.\n\n“B - Section 4.2 mentions that the DistilBERT-CRF model performed best in the NER experiments, but the section does not present the actual F1 metric (i.e., in states \"an F1-score of XX\"). The authors should also explain why they haven't considered using a larger BERT model for producing the representations, or why they haven't considered fine-tuning the DistilBERT model directly for the NER task (e.g., the \"linear classifier\" approach seems to correspond to the training of a linear layer on top of the BERT representations for each token, and this setting could be easily extended to consider also the training of the DistilBERT model itself). Ideally, additional technical details about the training and inner-workings of the model(s) should also be given. For instance, what were the hyper-parameters considered for model training (this is only shown in Appendix A, but it is somewhat strange that different values for parameters such as the batch size were used for the different models)? How was span recognition modeled as a token-classification problem in terms of tags associated to the tokens (e.g., did the authors use a B-I-O tagging scheme, or some other approach)? How was the BERT tokenization scheme, based on sub-words, combined with the tokenization scheme used for creating the annotated data?”\n\nWe thank the reviewer for these questions, which are all very relevant. The F1-score was omitted from the paper due to human error and will be added in the final version. We will also run all our experiments with BERT and/or RoBERTa. We will add details about hyperparameters and run all models across similar batch sizes. For tagging, we used the BIO/IOB scheme. Subwords retained the tag of the actual word for model training. Predictions were then made by first tokenizing words using the BERT tokenization scheme, with subwords merged after prediction. We will add details on this to the appendix and a mention of it in the main text.\n\n“C - On what regards Section 4.4, the authors mention the use of a Kernel Density Estimation procedure to filter out locations more than three standard deviations from the mean of their coordinates. This procedure should be presented in more detail (e.g., what KDE parameters, such as the kernel function or the kernel bandwidth, were considered? What do the authors mean with the expression \"mean of their coordinates\"?), and the authors should also justify this choice against some other procedure to remove outliers. In connection to this, the authors also mention the use of a convex hull to delineate the region formed by the coordinates, but a more accurate procedure could have been used instead (e.g., threshold the results of the KDE function, to keep the high density region, or using a different computational geometry approach such as alpha-shapes, which is not restricted to the formation of convex polygons).”\n\nThank you for these thoughtful methodological comments and suggestions. We will devote more discussion—depending on space, perhaps in an appendix—to describing and justifying the KDE/outlier removal process, up to and including a systematic comparison with other methods. The use of alpha-shapes is a good idea and we will try this approach. However, this method might result in regions that do not correspond to human intuitions of neighborhood shape. We will also experiment with using the high-density KDE region, per the reviewer's recommendations. Finally, we will replace the “mean of the coordinates” with the centroid of the region.\n\n“D - Also on what regards Section 4.4, which discusses methods for resolving spatial language, the authors mention the use of a \"canonical list of toponyms.\" This should be further detailed in the paper. Does the canonical list correspond to a sub-set of geonames entries? The entire contents of Section 4.4 are actually somewhat confusing (e.g., it is also not clear why and how different string similarity metrics are used), and perhaps this section should be re-written.”\n\nWe note the reviewer’s confusion and will try to improve the clarity of the section without sacrificing methodological detail. The “canonical list of toponyms” refers to a widely used shapefile for New York neighborhoods. We will add references to other papers that use this same list to clarify its status in research on neighborhood-level phenomena in New York.\n\n“E - The models used for detecting spatial language should ideally be evaluated comparatively to other existing approaches. I understand that the NER classes considered by the authors, useful for the type of study that they present, go beyond the classes that are considered in previous work. Still, it would be nice to see how the model compares to other approaches in terms of its ability to detect place names. Similarly, the method for \"resolving spatial language\" should also be compared against other alternatives (e.g., against existing open-source systems such as Mordecai).”\n\nThis is a good idea and we thank the reviewer for the suggestion. We will our toponym tags and compare them with results from Mordecai as well as the Stanford NER parser “LOCATION” tags. Evaluating the toponymy resolution step is challenging because we don’t have extensive gold-standard labels. However, we will perform a limited evaluation using the validation set described in our response above.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper describes creative/original work (interdisciplinary, although also on a very a niche topic that will not be of interest to all conference attendees), presenting an interesting analysis on the interplay between language, text and place.\n\n* The paper is clear and well-written, properly contextualizing the problem and the contributions\n\n* The authors plan to release the training data and models on Github, and detailed information about the annotation process and NER results is given in appendix.", "reasons_to_reject": "* The descriptions associated to the methods for \"resolving spatial language\" are somewhat confusing. This part of the proposed approach was also not quantitatively evaluated, and it may be the case that this component is introducing many errors.\n\n* Some of the choices regarding the methods for \"detecting spatial language\" should be better justified in the paper.", "questions_to_authors": "A - Section 4.1 should ideally report on the agreement between the annotators of the NER data, even if only leveraging on a small sub-set of the annotated data.\n\nB - Section 4.2 mentions that the DistilBERT-CRF model performed best in the NER experiments, but the section does not present the actual F1 metric (i.e., in states \"an F1-score of XX\"). The authors should also explain why they haven't considered using a larger BERT model for producing the representations, or why they haven't considered fine-tuning the DistilBERT model directly for the NER task (e.g., the \"linear classifier\" approach seems to correspond to the training of a linear layer on top of the BERT representations for each token, and this setting could be easily extended to consider also the training of the DistilBERT model itself). Ideally, additional technical details about the training and inner-workings of the model(s) should also be given. For instance, what were the hyper-parameters considered for model training (this is only shown in Appendix A, but it is somewhat strange that different values for parameters such as the batch size were used for the different models)? How was span recognition modeled as a token-classification problem in terms of tags associated to the tokens (e.g., did the authors use a B-I-O tagging scheme, or some other approach)? How was the BERT tokenization scheme, based on sub-words, combined with the tokenization scheme used for creating the annotated data?\n\nC - On what regards Section 4.4, the authors mention the use of a Kernel Density Estimation procedure to filter out locations more than three standard deviations from the mean of their coordinates. This procedure should be presented in more detail (e.g., what KDE parameters, such as the kernel function or the kernel bandwidth, were considered? What do the authors mean with the expression \"mean of their coordinates\"?), and the authors should also justify this choice against some other procedure to remove outliers. In connection to this, the authors also mention the use of a convex hull to delineate the region formed by the coordinates, but a more accurate procedure could have been used instead (e.g., threshold the results of the KDE function, to keep the high density region, or using a different computational geometry approach such as alpha-shapes, which is not restricted to the formation of convex polygons).\n\nD - Also on what regards Section 4.4, which discusses methods for resolving spatial language, the authors mention the use of a \"canonical list of toponyms.\" This should be further detailed in the paper. Does the canonical list correspond to a sub-set of geonames entries? The entire contents of Section 4.4 are actually somewhat confusing (e.g., it is also not clear why and how different string similarity metrics are used), and perhaps this section should be re-written.\n\nE - The models used for detecting spatial language should ideally be evaluated comparatively to other existing approaches. I understand that the NER classes considered by the authors, useful for the type of study that they present, go beyond the classes that are considered in previous work. Still, it would be nice to see how the model compares to other approaches in terms of its ability to detect place names. Similarly, the method for \"resolving spatial language\" should also be compared against other alternatives (e.g., against existing open-source systems such as Mordecai).", "paper_topic_and_main_contributions": "The paper combines critical toponymy with automated text analysis. Using a dataset of 47,440 New York City Airbnb listings from the 2010s, the authors report on an study that focuses on how property owners refer to places within the city and how this relates to cultural and economic capital. The authors introduce a new Named Entity Recognition (NER) model tailored to identify places and spans of text that are important to the characterization of place, and also new lightweight and accurate methods for toponymy resolution and geospatial dependency parsing. Results show that the residents of different neighborhoods indeed use different strategies to describe their property locations, and that strategies such as mentioning the neighborhood by name positively correlate with gentrification. The authors also highlight the fact that while some previous work within the NLP community has focused on toponym recognition and resolution, few studies have actually been reported using these methods to address concrete questions related to other fields (e.g., human geography and/or the computational social sciences).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Section 4.2 is missing the F1 score the the DistilBERT-CRF model, and also the reference to the appendix with a full comparison of models (i.e., it mentions Appendix XX instead of Appendix A).\n\nAs mentioned before, Section 4.4 is somewhat difficult to follow, and perhaps it should be re-written.", "Missing_References": "There's a recent paper that discusses current challenges associated to text geoparsing, which should be analyzed in the context of the parts that concern \"detecting spatial language\" and \"resolving spatial language\":\n\nGeoparsing: Solved or Biased? An Evaluation of Geographic Biases in Geoparsing \nhttps://agile-giss.copernicus.org/articles/3/9/2022/\n\nThere's also paper describing a benchmark platform named EUPEG for evaluating text geoparsing over multiple datasets, which can perhaps be referenced in the analysis of related work:\n\nEnhancing spatial and textual analysis with EUPEG: an extensible and unified platform for evaluating geoparsers https://geoai.geog.buffalo.edu/EUPEG/ \nhttps://github.com/geoai-lab/EUPEG\n\nAnother recent system for toponym recognition and disambiguation is Mordecai, which is conveniently available as open-source software. Passing texts through Mordecai and comparing results against the proposed approaches for \"detecting spatial language\" and \"resolving spatial language\" should be relatively simple (although this comparison would not use the detailed categories used in the proposed NER system):\n\nMordecai 3: A Neural Geoparser and Event Geocoder \nhttps://arxiv.org/pdf/2303.13675.pdf \nhttps://github.com/ahalterman/mordecai3"}}, {"review_id": "6qamle4r6N", "reviewer": "Reviewer_AMJs", "summary": "", "questions": "A. At the final step of the lightweight toponym resolution algorithm $\\S$4.4, how is the number of points determined to rank the candidates?\n\nB. It is not clear precision/recall/F$_1$ corresponds to what data in Table 2.\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors would like to thank all three reviewers for their thoughtful, constructive, and knowledgeable feedback. We appreciate the opportunity to offer our responses here. Below, we respond to each reviewer’s “Reasons to reject” and “Questions for the authors” sections. For the sake of brevity we do not respond to “Reasons to accept,” “Missing references,” and “Typos” feedback; reviewers can assume that we will incorporate all suggested references and correct all typos in the final draft. We reproduce each reviewer comment in quotation marks below.\n\nReasons to reject\n\n“The case study of toponym spans for two neighborhoods is interesting, yet, we cannot draw concrete conclusions based solely on one instance. A broader approach would be to conduct a similar analysis on all gentrified and non-gentrified neighborhoods, which provides a more robust base for hypothesis testing. This does not mean visualizing all cases, but the area of the estimated convex hulls can be used for comparison.”\n\nWe agree that a two-neighborhood comparison is insufficient for more general inference. In a sense, this is the goal of Figure 4, panel 3, in which we show a correlation between the gentrification index and neighborhood span. If accepted, however, we will consider supplementary ways to demonstrate and visualize this relationship across all neighborhoods.\n\n“The correlations depicted in Figure 4, while present, appear to be moderate. It is not clear what correlation metric is used. My guess is the Pearson coefficient. In that case, 0.37 and 0.44 may not attain a level of correlation deemed substantial.”\n\nWe will clarify that we are using Pearson’s correlation. We would argue that in the social science literature, correlations around 0.4 are commonly reported and generally considered meaningful.\n\n“The performance of the lightweight toponym resolution method is not evaluated. An analysis is required to show how accurately this algorithm performs.”\n\nThe first reviewer said similarly and we will provide a comparison in an appendix (see our response above for further details).\n\n“The performance of the NER model on TN:OTHER and TN:REGION in Table 2 stands out as notably low. For TN:OTHER, it might be attributed to the small number of annotations, but the reasons behind the low performance on TN:REGION is not clear.”\n\nBoth categories are quite low-N and we anticipate that this is why they perform poorly. We will clarify this.\n\nQuestions for the authors\n\n“A. At the final step of the lightweight toponym resolution algorithm § 4.4, how is the number of points determined to rank the candidates?” \n\nWe are not completely sure we understand the question correctly, but a candidate gets one point if it’s within the n nearest neighbors for fastText and word2vec and two points if it’s among the top n Jaro-Winkler neighbors. We arrived at this ranking system through empirical trial-and-error. Hopefully this addresses the question? To clarify how performance depends on the ranking system, we will compare different point ranking systems (only one point for Jaro-Winkler, different sizes of n etc) in the appendix, alongside performance comparisons with Mordecai recommended by reviewer QGQw.\n\n“B. It is not clear precision/recall/F corresponds to what data in Table 2.”\n\nWe will clarify this in a final draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper explores an interesting and under-studied topic in the NLP community.\n2. The insights in the paper could spark future work to bridge the gap between NLP and other disciplines such as social studies.\n3. The manual data collection process is thorough and well-executed, especially the curated taxonomy of toponyms.\n4. The paper is well-written and I really enjoyed reading it.\n", "reasons_to_reject": "I do not find any major reasons to reject this paper. Nevertheless, I do have a few minor concerns:\n\n1. The case study of toponym spans for two neighborhoods is interesting, yet, we cannot draw concrete conclusions based solely on one instance. A broader approach would be to conduct a similar analysis on all gentrified and non-gentrified neighborhoods, which provides a more robust base for hypothesis testing. This does not mean visualizing all cases, but the area of the estimated convex hulls can be used for comparison.\n\n2. The correlations depicted in Figure 4, while present, appear to be moderate. It is not clear what correlation metric is used. My guess is the Pearson coefficient. In that case, 0.37 and 0.44 may not attain a level of correlation deemed substantial.\n\n3. The performance of the lightweight toponym resolution method is not evaluated. An analysis is required to show how accurately this algorithm performs.\n\n4. The performance of the NER model on TN:OTHER and TN:REGION in Table 2 stands out as notably low. For TN:OTHER, it might be attributed to the small number of annotations, but the reasons behind the low performance on TN:REGION is not clear.\n", "questions_to_authors": "A. At the final step of the lightweight toponym resolution algorithm $\\S$4.4, how is the number of points determined to rank the candidates?\n\nB. It is not clear precision/recall/F$_1$ corresponds to what data in Table 2.\n", "paper_topic_and_main_contributions": "The paper presents a framework for critical toponymy that aims to study place names (toponyms) within a broader social, cultural, historical, and political context. To this end, the authors curated a dataset comprising nearly 47K New York Airbnb listings and reviews. They have then undertaken the manual annotation of toponyms within nearly 2.6K of these listings and reviews based on a taxonomy they devised. Subsequently, an NER model was trained on the annotated data and was used to identify toponyms in the rest of the dataset. A statistical toponym resolution model is introduced to ground toponyms using syntactic features (dependency parse tree connections) and semantic similarity in the embedding space (word2vec and fastText). \n\nGiven the toponymy information, the paper provides empirical support to corroborate a number of sociodemographic phenomena:  (1) users exhibit a tendency to use neighborhood names more frequently for neighborhoods that are central and deemed desirable (Figure 2); (2) A disparity between the perceived boundaries of neighborhoods and their canonical boundaries is underscored in two contrasting neighborhoods in terms of prestige: Williamsburg and Bedford-Stuyvesant (Figure 3); (3) The correlation of several factors with gentrification index is visualized (Figure 4). The number of listings, the ratio of self-referential mentions to neighborhoods, and the span of neighborhood names exhibit positive correlations with gentrification.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Section 2.2 could be improved by surveying more relevant works in toponym resolution from the NLP literature. The following references can be a starting point:\n\nDeLozier, G., Baldridge, J. and London, L., 2015, February. Gazetteer-independent toponym resolution using geographic word profiles. In _Proceedings of the AAAI Conference on Artificial Intelligence_ (Vol. 29, No. 1).\n\nGritta, M., Pilehvar, M.T., Limsopatham, N. and Collier, N., 2018. What’s missing in geographical parsing?. _Language Resources and Evaluation_, 52, pp.603-623.\n\nKamalloo, E. and Rafiei, D., 2018, April. A coherent unsupervised model for toponym resolution. In _Proceedings of the 2018 World Wide Web Conference_ (pp. 1287-1296).\n\nSchneider, N.R. and Samet, H., 2021, November. Which Portland is it? a machine learning approach. In _Proceedings of the 5th ACM SIGSPATIAL International Workshop on Location-based Recommendations, Geosocial Networks and Geoadvertising_ (pp. 1-10).", "Typos_Grammar_Style_and_Presentation_Improvements": "- The paper is verbose at times. For instance, the algorithm described in $\\S$4.4, could be illustrated as a pseudo-code. Similarly, Section 6 mostly reiterates the findings presented in earlier sections. By condensing some sections, it would be possible to accommodate important details like Table 1 in the main body.\n- L320 and L322: XX\n- L364: Appendix C $\\rightarrow$ Appendix E\n- Some citations include an unrelated number, e.g. L88, and L135.\n"}}, {"review_id": "7uI3tVbZw8", "reviewer": "Reviewer_B1cM", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors would like to thank all three reviewers for their thoughtful, constructive, and knowledgeable feedback. We appreciate the opportunity to offer our responses here. Below, we respond to each reviewer’s “Reasons to reject” and “Questions for the authors” sections. For the sake of brevity we do not respond to “Reasons to accept,” “Missing references,” and “Typos” feedback; reviewers can assume that we will incorporate all suggested references and correct all typos in the final draft. We reproduce each reviewer comment in quotation marks below.\n\nReasons to reject\n\n“This method was tested on a single city (NY) of which the authors appear to have good knowledge. It remains to be seen how well it generalizes to other cities.”\n\nWe agree that, in an ideal world, we would be able to annotate data from multiple cities and test performance on data from unseen cities. Due primarily to practical limitations, data from other cities were not included in this paper. We argue that this paper is proof-of-concept for a general way of using a class of NLP methods along with social science theory to think about a certain kind of linguistic practice and its relationship to urban social life, in line with the purview of the “Computational social science and cultural analytics” track. As such, we feel that a single city is sufficient for the substantive argument we make, if not ideal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work focuses on a sociological topic that is both highly relevant and of general interest: the phenomenon of gentrification in big cities such as NY. The work makes use of modern NLP techniques (transformers, embeddings, fine-tunning) to improve the state of the art in toponymic NER, showing convincingly that it can identify informal toponyms. This seems particularly important in a context where the perceived boundaries and names of neighborhoods are an emergent collective phenomenon, nevertheless of strong sociological interest, impacting the perception of prestige of neighborhoods and affecting / being affected by the dynamics of gentrification.\n\nI really appreciate the successful inter-disciplinary nature of this work, which is a great example of high quality computational social science. Beyond the soundness of the NLP methods proposed and the improvement in this specific domain, interesting sociological observations are actually extracted, in the correlations found between gentrification and informal toponomycal practices.", "reasons_to_reject": "This method was tested on a single city (NY) of which the authors appear to have good knowledge. It remains to be seen how well it generalizes to other cities.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this work, a novel NER model is proposed to identify both formal and informal toponyms as well as spatial relationships in unstructured text from Airbnb listings. For this purpose, the authors develop a taxonomy of 21 toponymic categories, including spatio-temporal relations (\"5 min away from...\") and manually annotate 2600 listings. Three fine-tuned transformer models are compared, with the CRF performing the best and presenting a good F1 score.\n\nThis model is then used to analyze the NYC Airbnb dataset in relation to a gentrification index. It is found that the proportion of Airbnbs listings that mention its neighborhood my name is positively correlated with gentrification, as is the neighborhood toponomyc span, suggesting that the perceived \"prestige\" of a neighborhood creates a tendency for its name to be referenced in a larger surrounding area. Inversely, toponomyc references to entities other than neighborhood are inversely correlated with gentrification.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "F1-score and Appendix number are missing in section 4.2."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors use a named entity recognition model trained on a novel annotated dataset to analyze how property renters in New York City describe the locations of their listings on Airbnb. The authors connect this linguistic study with a spatial analysis of gentrification in New York. They also contextualize their findings using theories of “critical toponymy” from social science. 

Overall, the review process converged to a consensus on this paper. The paper gained high soundness and excitement scores from all reviewers. Reviewers also offered very positive qualitative feedback.

Positives which emerged from the review process:
- All reviewers assigned a score of 4 for soundness and a 4 for excitement. One praised the paper for being “clear and well-written,” and “properly contextualizing” a computational analysis. Another noted that “the paper is well-written and I really enjoyed reading it.” A third described the paper as a “great example of high quality computational social science.”

Negatives which emerged from the review process: 
- While the review process did surface some issues surrounding soundness in the original draft, these were addressed during the rebuttal period. Reviewer QGQw asked for missing details about KDE parameters, the NER tagging scheme, the outlier detection procedures and the F1 score for a specific model. Reviewer AMJs also asked for missing information about the correlation statistic. However, the authors provided detailed answers to these questions, and both reviewer QGQw and reviewer AMJs replied to the rebuttal to say that their concerns had been addressed. Each mentioned they still had a positive assessment of the paper after reading the rebuttal.

- Reviewer B1cM also noted that one limitation of this work is that it focuses on a single city (New York). The authors replied that this was due to practical considerations, and that their paper offers a proof-of-concept which might be applied to other cities in the future. They noted that annotated data for other cities could improve their analysis. B1cM read this rebuttal and did not change their already high score.
Decision: Accept-Main