Event-Location Tracking in Narratives: A Case Study on Holocaust Testimonies
Authors: Eitan Wagner
Renana Keydar
Omri Abend
Venue: EMNLP 2023 Main (2023)
Keywords: Location tracking
Narrative understanding
Holocaust testimonies

Abstract: This work focuses on the spatial dimension of narrative understanding and presents the task of event-location tracking in narrative texts. The task intends to extract the sequence of locations where the narrative is set through its progression. We present several architectures for the task that seeks to model the global structure of the sequence, with varying levels of context awareness. We compare these methods to several baselines, including the use of strong methods applied over narrow contexts. We also develop methods for the generation of location embeddings and show that learning to predict a sequence of continuous embeddings, rather than a string of locations, is advantageous in terms of performance. We focus on the test case of Holocaust survivor testimonies. We argue for the moral and historical importance of studying this dataset in computational means and that it provides a unique case of a large set of narratives with a relatively restricted set of location trajectories. Our results show that models that are aware of the larger context of the narrative can generate more accurate location chains. We further corroborate the effectiveness of our methods by showing similar trends from experiments on an additional domain.

Event-Location Tracking in Narratives: A Case Study on Holocaust Testimonies
Eitan Wagner " Renana Keydar
Omri Abend
Department of Computer Science $ FFaculty of Law and Digital Humanities
Hebrew University of Jerusalem
{first_name) [last_name name Jamalemm 1 wuw...jja.. ac. il
Abstract
Inputt Segrentnn
Ousput Locations
This work focuses on the spatial dimension of narrative understanding and presents the task of event-location tracking in narrative texts, namely the extraction of the sequence of loca- tions where the narrative is set. We present sev- eral architectures for the task that seek to model the global structure of the sequence, with vary- ing levels of context awareness. We compare these methods to a number of strong baselines and ablated variants. We also develop meth- ods for the generation of location embeddings and show that learning to predict sequence of continuous embeddings advantageous in terms of performance over predicting string of locations. We focus on the test case of Holo- caust survivor testimonies, motivated by the moral and historical importance of studying this dataset using computational means. The dataset further provides unique case of a large set of narratives with a relatively restricted set of location trajectories. Our results show that models that are aware of the global context of the narrative can generate more accurate loca- tion chains. We corroborate the effectiveness of our methods by showing similar trends in an additional domain.
part7 and sike hags loas wae an importann pooduct by las 1o wer Farth
coker thon
fro toaring hou rouulcaee
"Forth (Sermanyj'
higs wulling
my sttthrr and my sure qull your
Weer whice work tu to Naccebent waillaan the lowding cediiiiuioi
"Nuremberg [Middle Franconts, tavaria. Germany)"
Cornnny burr Hummitaeut And ih 1901 caree hoo
Figure 1: Event-Location Tracking in Holocaust testi- monies. The output the list event locations which is not identical to the list of location mentions.
Nevertheless, and despite the wealth of NLP lit- erature that has studied the expression of locations in texts (see $2), we are not aware of works that attempted to extract the trajectory sequence of locations where a narrative story takes place.
In this work, we focus on transcriptions of Holo- caust survivor testimonies, given in English. It is difficult to overstate the importance of this dataset for the study and memory of the Holocaust. With the inevitable perishing of the last living survivors, there is an imminent need to develop new modes of engagement with the vast amount of Holocaust testimonies stored in the archives. The applica- tion of NLP technology to the analysis of these testimonies has recently been strongly advocated (Artstein et al., 2016; Wagner et al., 2022). Indeed, NL. can aid in allowing researchers to gain insight from the entire collection of testimonies (tens of thousands of them), instead of focusing small- scale, mostly manual, studies. We further argue that Holocaust testimonies have the additional, unique value for NLP, due to its combination of large number of testimonies and a relatively restricted domain, in terms of themes and locations. This quality stands in contrast to that of typical narrative datasets (Sultana et al., 2022).
1 Introduction
A primary goal of narrative analysis is to represent essential dimensions of stories in a schematic man- ner. One such essential dimension is the location sequence of locations, in which the story takes place. In fact, location is such an important ele- ment in a story, that the ability situate a story in a place is often viewed as one of the elements that distinguish narrative text from other types of texts (Piper and Bagga, 2022). Characterizing stories by their sequence of locations is also beneficial in that it provides backbone for alignment between different stories-ar an important task in its own right (see, e.g., Ernst al., 2022).
'Code is provided at ei llaawagner/lccation-rraake

---

In the narrative framework, event-location track- ing is unique in two ways. First, as in Gius and Vauth (2022), the focus is on event location, i.e., the location where the event takes place. For example, if a witness recounts events in the ghetto, and men- tions someone who came from some town, then al- though the town's s name is mention of a location, it is not part of the event location. Second, the loca- tions in narratives should be viewed as trajectory. For example, migration from Europe America is not merely a change of location since it follows a common post-war theme of Holocaust-relact trajectories. The task event-location tracking is therefore different from entity linking in its desired output- a sequence of segment-wise predictions for an entire document.
processing; (4) we present method for creating domain-dependent location embeddings, and show its value to the task at hand.
2 Previous Work
Narrative Analysis and Segmentation. Narra- tive schema analysis attempts to model the essence of event sequences. It is beneficial to extract a high- level sequential progression of long story, giving an overview of the events and allowing alignment between relevant parts. For example, Antoniak et al. (2019) visualized the frequent topic paths in birth stories using segment-wise topic modeling.
To extract an interpretable sequential progres- sion it is necessary divide the long story into shorter segments. Zehe al. (2021) formulated the task of dividing a long text into segments by scenes, Hotho et al. (2021) summarized shared task for scene segmentation and concluded that the task re- mains challenging. Our work is somewhat similar to scene segmentation since location trajectories in- duce some type of segmentation and locations are a key component in scenes. Nevertheless, the defini- tion of scene involves other aspects, such as time and plot, making it harder to formalize prediction task and apply domain knowledge to aid the task. We focus, therefore, on locations only, and seek to extract the trajectory of the whole narrative. For this purpose, we focus on coarse locations (such as cities) and not on detailed scenes (such as specific houses).
Trajectory extraction can be valuable for visual- ization and trajectory clustering (Bian et al., 201 8). In addition, successful location extraction indicates aspects of long-range narrative understanding, currently open problem in NLP (Yao et al., 2022).
In this work, present various architectures for location tracking that vary in their degree of context awareness. In one architecture, we use learnable transition matrix, thus taking into account the adja- cent locations. In another, we design a hierarchical transformer that can attend to the whole document. We also design a method to generate embeddings for the locations. We compare our methods to number of baselines that use strong language mod- els but are limited in the context-length that can be taken into account.
Event Locations. Some recent works have ex- pressed the significance of event locations for nar- rative analysis. Piper al. (2021) formulated definition of narratives with event locations as key part. Soni al. (2023) formulated a task of grounding characters to locations. Other works extracted event locations for single events, such as events described in tweets (Kumar and Singh, 2019). However, no previous work, to the best of our knowledge, has studied the trajectory of loca- tions in a full narrative.
We show that our models significantly outper- form the baseline models. We find that, in general, models with larger context capabilities have higher performance in location tracking. We also find that the use of location embeddings, trained with addi- tional metadata, can contribute to the performance. For validation, we construct an experiment in an ad- ditional domain and show that similar trends arise.
In conclusion, we find that the task of event- location tracking belongs, both conceptually and empirically, to the level of full narratives. We argue that this is important step towards the under- standing of narratives as full stories.
Toponym Resolution. Toponym resolution de- scribes the task of extracting and identifying loca- tion mentions in text. This task be seen subtask of Named Entity Linking, by consider- ing only location ("GPE") entities. Therefore, to- ponym resolution can be carried out with general- purpose entity linker. Cao et al. (2021) released a purpose entity linker based on autoregres-
Our contributions in this work are follows: (1) we present the task of event-location tracking for an entire narrative text; (2) we establish an experimen- tal setup for the task, in the domain of Holocaust survivor testimonies; (3) we present a number of ar- chitectures for the task, capable of document-level

---

sive language generation. As in most entity linkers, the entities are Wikidata items.
segment x;. The location sequence induces another segmentation that is based on locations. We work with a closed set of locations V i.e., Vi.y; E.. It is instructive to compare this task to NER for locations. Whereas NER is a prediction task at the phrase level, our task's s focus is on the document as long sequence of events. This change of focus implies some unique properties:
Nevertheless, has been shown that special train- ing for spatial entities has better performance, lever- aging specialized location gazetteers such as Geon- ames 2, which is more detailed than Wikidata lo- cation items (Hu et al., 2023). Hu et al. (2022) presented a survey about location-specifie entity extraction. Zhang and Bethard (2023) provided a system that first resolves the con-ambiguuus lo- cation mentions and used them as context for the ambiguous oncs. Wang and Hu (2019) provided an evaluation platform for toponym resolution.
In location tracking, the locations are to be as- signed to spans of multiple sentences, that de- scribe a locally confined cluster of events, and not to individual phrases in NER. We do not see much value in marking the exact token-level boundaries of event locations since they are not necessarily clear from the text. In our case, we make a practical choice and use units of multiple sentences, as this is the scope of the available labeled data.
Location Embeddings. Many methods exist for general-purpose text-span embeddings (Le and Mikolov, 2014; Reimers and Gurevych, 2019). Specifically for locations, Tian al. (2022) gen- crated location embeddings from human mobility trajectories using a graph- based approach. Kejri- wal and Szekely (2017) generated location embed- dings based on random walks over the Geonames location graph. Dassereto et al. (2020) proposed evaluation methods for location embeddings.
In location tracking, the prediction is at the docu- ment level, with possible dependencies through- out the entire document. This property requires strong long-context capabilities. For example, a segment describing a visit to the narrator's home- town might not explicitly mention the town't 's name. The location name should be predicted as the event-location based on earlier mentions or external knowledge.
Trajectory Modeling, In contrast to toponym resolution, which focuses on short mentions, a dif- ferent line of work seeks to extract document-level trajectories. Mathew et al. (2012) modeled human location trajectories by Hidden Markov Models (HMM). Sassi et al. (2019) replaced HMMs with convolutional neural networks applied on location embeddings. Lui al. (2021) used LSTM-based models for pedestrian trajectory prediction.
Whereas NER annotates mentions, event- location tracking often requires inferring the lo- cation implicitly. Indeed, the task requires as- signing location events if no explicit lo- cation is mentioned. Conversely, many locations are mentioned without acting as event locations.
In our work, we propose the extraction of loca- tion trajectories for events in texts. There are two key differences between this and the existing work, which focuses on trajectory modeling for lists of location coordinates. First, event locations may not be explicitly mentioned in the text, therefore requiring non-trivial mention extraction. Second, locations in texts represent more than simple coor- dinate values. For example, concentration camp might be close to a city but they clearly represent different thematic locations.
We also note that, specifically for testimonies, differentiate between the locations of described events, which the task addresses, and the locations of the event of giving the testimony (the ground), which are not part of the location trajectory.
Methods
Location tracking is a structured prediction task, as there are dependencies between the segment la- bels y;. For tasks like this, the global context of the document, and not just the individual segments, can be helpful. To this end, we propose three dif- ferent approaches with different degrees of context awareness.
3
Task Definition
Our setting is the following: given a text, divided into initial segments, x = X2, Kn. we wish to predict sequence of locations = y1, -aa such that each location y; corresponds to the text
4.1 Deep CRF
We design and implement linear CRF model (Laf- ferty et al., 2001) with transformer inputs. We first
2nttp://geonaees.org

---

Impeat Iesorrorry
Oriput Locatirra
Irput Test mony
Motel
Outbur Locos ors
Location
That Segment
Sopment
Text
Segimnnt
Toand
help
Location A2
Sigmen
Location
(a) CRF with global transition weights.
(b) CRF with pegment-depenteee transition weights.
Input: Testimony
Model
Output: Locations
0
Text Segment
Tearcfarmet Twst-
loration #1
ninpuE
Transforme
Encoder for Sepment Fmoedddnng
Text Segment #2
Transformer Tral- Encoder
Posted
Location #2
(c) Hierarchical transformers.
Figure 2: Overview of the proposed models. Notation is defined in Section 4.
finetune a transformer for segment location clas- sification and use the output logits as inputs for the CRF module.3 We propose two variants: the first is a more straightforward approach that uses a global transition potential matrix, while the second predicts the transition potentials "on the fly."
limitations. For example, in most cases, the proba- bility of staying the previous location is highest, but if the previous segment describes migration or deportation then the probability of staying should be low. For this reason, we present another version with local probabilities.
Global Transition Probabilities. In the first sion, the model consists of one transition matrix that is learned during training. The transition ma- trix is a parameter of the CRF component. Sce figure 2a. We denote the global transition model with CRF-G. Formally we can write:
Local Transition Probabilities. In the second version, the transition matrix the output of a transformer that receives the segments as inputs and outputs a transition matrix. This allows the use of different transition probabilities based on the content of the current segment. The architecture of the transition estimator is similar to the classifier, but the output is (flattened) logit matrix instead of logit vector. Each entry in the matrix represents the potential of transitioning between two location labels. In this version, the learned parameters are part the transformer component and its heads. See figure 2b. We denote the local transition model by CRF-L. Formally:
exp Σ (Eo(xi) w My
PCRF
(1)
O(x:0,w,M)
where Eg is the transformer encoder, the weights of the linear layer on top of the classifier, and M is the transition matrix. function. 0, w and are learnable parameters.
the partition
In this version, the transition matrix a function of the entire training dataset and does not depend on the specific local segment. This assumption has
PORF-L(y(x)
*Our implementation for the CRF model is based on the TorchCRF package - / con/ 4142244 TorchCRF - with minor modifications.
exp Σn_ (Eo, (xx) (k))-w wi + Eo, 1yy wz)
Z(x: 0, 0z, w1.

---

where Eg, and Eg are the transformer encoders; Ed, outputs vector for a text span and Es, outputs a matrix. w1 and w2 are the weight vectors of the linear layers on top of Ea, and Eo,, respectively. is the partition function.
hidden states for each vector inputs. the weight vector for the linear layer on top of Es.-
Location Embeddings. The previous models use discrete list of locations. In this setting, classi- fication is done in "one-hot" setting. A better option is to leverage the properties of the locations by creating location embeddings. This allows the use of zero-shot capabilities the models, as well as the use of similarity between locations.
Learning. In both versions, the segment-level classifier is first finetuned with locally labeled seg- ments. Then the CRF optimization is performed with the negative log-likelihood loss of the se- quence. As in CRF models, the likelihood includes an exponential number of summands but can be computed efficiently with dynamic programming. Gradient updates can either include only the tran- sition component (a matrix for the global version classification head for the local version), with the local classifier frozen, or can include further finetuning the segment-level classifier too.
The embeddings obtained by sentence em- bedding model trained on natural language infer- ence (NLI) data, with additional domain-specific data derived from the thesaurus and the segment annotations.
Using embeddings does not require structural changes to the architecture. The only difference is that the classification layer (i.e., the layer before the softmax) is initialized with the location embed- dings and remains frozen during training. By doing so, we effectively predict the locations by project- ing the inner representations onto the embeddings. We denote this model by HITRF+EMB.
4.2 Hierarchical Transformer
a similar manner to the CRF model, we design a hierarchical transformer with one transformer that encodes the text of each segment, and a second transformer that encodes the sequence of segment encodings. See Figure 2c.
4.3 Baseline Models
This architecture has two potential advantages over the CRF ones. The first is that a linear CRF attends to one previous location only, thus limit- ing the available context. Extending the context involves a high computation cost. The second ad- vantage is the larger flexibility of the hidden states, As opposed to the CRF model, which outputs prob- abilities, the transformer component outputs raw hidden states that can be used either inputs for classification layer or the actual representations given a set of predetermined embeddings.
For baselines, we present several methods that do not take the entire document into account. Pre- dictions are made for each segment independently. taking into account very limited additional context. These methods are simple and efficient. We intend to show that simplicity comes with a significant fall in performance.
Independent Classification. This method ex- tracts a location sequence as independent classifica- tion tasks. We train transformer model to classify the location and use this predict the segment loca- tion independently, i.e., without taking into account the relationship between predicted classes. This is the same as Eg (x) w in the proposed models. We denote this model with INDEPENDENT.
Given the flexibility of this architecture, we de- sign two versions. The first uses discrete locations, and the second uses location embeddings.
Discrete Locations. This model is constructed with two components. First, segment-level trans- former text encoder receives a text segment and out- puts an embedding vector. Second, a transformer encoder model receives a sequence of embeddings and outputs embeddings for sequence classification. We denote this model with HITRF. Formally:
Greedy Decoding. In this method, we train a transformer to predict a location for segment given some previous locations and segments.
For decoding, we predict location given the first segment and two 'START" (one for an empty segment), and then recursively predict the loca- tion for a segment given the previous segment and
V In]: Px)l)(y)))) softmax([ (Eo, (1)))==))ww)
'In our experiments, we used the two preceding locations and one preceding segment, The input format was: y;-2 [SEP) x [SFP] y;-1 [SEP] x;, where x-s are text segments and y-s are locations. The model was trained to prodict y;-
where Eg, and Es, are the transformer encoders; Eo, outputs a vector for a text span and Eo, outputs

---

two previously predicted locations. We denote this model with GREEDY. Another version of this method used beam search for decoding. The results were not significantly different.
For the purpose of location tracking, we used in- dex labels that can be seen as describing locations, such as cities, ghettos, concentration camps, etc. We performed our experiment with the category name (e.g.. "Ghettos in Poland") and not the ex- act location name (e.g.. "Warsaw (Warsaw, Poland: Ghetto)"). The conversion to categories was based on the hierarchy in the SF thesaurus, with minimal adaptation, that was done by Holocaust research experts. We ended up with 105 categories for loca- tions and two more labels for 'START' and 'END'. The label set has 55 categorics for cities (including moshavim and kibbutzim), 13 for concentration camps, 11 for administrative units, 9 for ghettos, 9 for DP camps, 9 for refugee camps, and one cate- gory for death camps. The full list can be found in appendix A.
In the independent and greedy baseline methods, we also divided the testimony into 10 bins and inserted the bin number of the segment as part of the text. This gives the classifier some additional information beyond the segment itself.
Location Entity Selection. In this method, we apply NER and predict the location based on the entity mentions. We use a pretrained zero-shot classifier to predict the location based on the con- catenation of all mentions of the locations in the segment, obtained from NER tool. We denote this model with ENTITY.
ChatGPT. Another bascline experiment with is zero-shot classification with ChatGPT: We give the model instructions to return location category from the specified list. For each segment, we input the instructions and the previous segment and its response, with an exception for cases where the response was not a category on the list, in which case, we did not input the previous segment and response for the next segment. 6 After receiving list of predictions, were not the list and instead kept the previous lo- cation category that was on the list (or used START if there was no previous location). See appendix B for the full prompts.
Using location categories allows us to signifi- cantly reduce the number of categories (we have over 1500 exact locations). Reducing the number of categories inevitably results in some loss of infor- mation, but the categories still provide rich informa- tion useful for many tasks such as location-basel segmentation, alignment between testimonies, and creation of high-level summaries.
We filtered out testimonies that did not follow the same labeling as the others, such as those with longer segments or very few location labels. We clarify that we did not discard any parts of the tes- timony text. For segments without location labels, we assumed the previous location is maintained, An exception is a case of segment that was la- beled as including visit, in which case the next segment assumed the location before the visit. We ended up with 585 testimonies labeled with loca- tions for each 1-minute segment. As the described process includes some heuristic labeling, which in- evitably introduces noise, we manually proofed the labeling of a random set of 53 testimonies. These testimonies were held out for all steps of training, including embedding training and hyperparameter tuning, and were used only a test set. Out of the remaining testimonies, used 90% for training and 10% for validation.
ignored all responses that
5 Experimental Setup
5.1 Data
Location Data. Our main data consists of 1000 Holocaust survivor testimonies, received from the Shoah Foundation (SF).7 All interviews were con- ducted orally by an interviewer, recorded on video. and transcribed as time-stamped text. The lengths of the restimonies range from 2609 to 88 $8105 words, with a mean length of 23536 words. Each testimony recording was divided into seg- ments, typically a segment for each minute. Each segment was indexed with labels, possibly multi- ple. The labels are all taken from the SF thesaurus." The thesaurus is highly detailed, containing 8000 unique labels across the segments.
5.2 Model Specifics
General Considerations. In all models that in- volved training, text segments were encoded with a pretrained transformer model. We decided use the LUKE model (Yamada et al., 2020) since it was explicitly trained for entity-awar tasks. We used
Sht //chat. openai. com/ This is done to avoid propagation of this type of error. Phttps://sff usc. edu/
/https://sfi, //sfi, wssssfiiusceeee. www.uu.cuuuuu edu/content/
howoord-thesaurde

---

the pretrained luse-base from Huggingface hub, The outputs of the encoder were pooled using the pooler. Specif-
01). This type of training requires some form of gradient accumulation for memory efficiency. In our experiments, we applied full gradient up- dates to all models besides CRF-L. This is due to technical complexity and the fact that the hierarchi- cal transformer models yield better performance.
default HuggingFFac-trrnnsfooe ically, the LUKE-pood poooler applies a linear transfor- mation and tanh activation to the embedding of a special token that is placed at the beginning of the sequence.
For all architectures, we used batch-size=1. We trained CRF-L, CRF-G and HITRF+EMB for 20 epochs, and HITRF 10 epochs. For the segment- level text-tramsformer in the full gradient setting, we used learning-rate=5.. 10-6, For the CRF ma- trix, we used sarminngratee-)) 10 and weight- decay=10 6. For the transformer with segment embeddings, used learning-rate=10-
Baseline Models' Specifies For the independent and greedy baselines, we finetuned LUKE for loca- tion classification with the standard Huggingface Trainer settings. For independent classification, the input was only the segment text and the relative lo- cation in the testimony, and for the greedy decoder, the input included the previous location-category label. For beam search, we used k = 10 beams.
Location Embeddings. Using the sentence- transformer package, 13 we finetuned LUKE for sentence embeddings. First, finetuned with the combination of the SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018) datasets and with the Multiple Negative Ranking loss." 14 We added domain-specific training examples and fur- ther finetuned with them. The extra examples were testimony segments with the SF location labels and location terms from the SF Thesaurus with their cor- responding descriptions. We note that even though the predicted output is on the category level, the location embeddings were trained also for the ex- act location level (i.e., for each city name). The pairs were converted to NLI form with standard prompts. See appendix C for further details on the embedding training.
For the entity selection baseline, we used the Holocaust NER model for entity recognition," 10 and bart-large" 11 for zero classification.
Initialization. The CRF transition matrix can be initialized either randomly or manually. Our exper- iments showed that initializing the prior transition matrix as a (smoothed) diagonal matrix yields the best results and this is what we used.
In HITRF, for the transformer for segment em- beddings, used randomly initialized DeBERTa (He et al., 2021) architecture.
Intermediate Training. In the trained models, besides the pretraining for the first transformer, we added an intermediate step of training at the seg- ment level. In all architectures, the segment-level text-transformer components (both classification and transition ones) were trained with the standard Huggingface trainer, with batch-size=4, learning- rate=5. 10 5, for 3 epochs. This was done also in the hierarchical transformer architectures, with the only difference being that the classification head was used for this step only and no further.
5.3 Evaluation Methods
Location Chains. Our models output sequence of locations for predefined list. For pointwise comparison, we only consider exact matches and
12 the forward pass, Eg, receives segments and outputs corresponding (pooled) encoding vectors, and Eg, receives all these vectors and outputs sequence of locations. The loss is calculated by the outputs of Fg, which are determined only after all segments were encoded (i.c., the input segments cannot be divided into batches). Encoding all segments before updating requires many copies of the computational graph, which requires large amount of GPU memory. To allow batching. we do the following: We run forward pass for Eg without gradients, encoding each segment. With these encodings. we run forward pass for Lo. with gradients, compute the loss, and backpropagate through Ea, obtaining (partial) gradients. We nun forward again through Fo, this time with gradicnts, and backpropagate with the previously obtained gradients. The last forward pass can be done in batches.
Finetuning. The main training step is finetuning with whole documents. In one version this finetun- ing updates only the weights of the last component (0,) - the CRF in the first architecture and the sec- ond transformer in the others. This method is fast and uses less memory, but it does not enable the usage of global properties the segment level. In another version, we update all weights (i.e., also
luke-base base
13 1hhttps://ww.. sbert. hhtttp://www.sbert.e/////ee wtwpe//ccaaaeeeeeeeeneeeeeeeeee 14https:/
10 ht //ner htw.hierrppttoooumnnttiett com/ http//ntro..ttt www.aaakkbbbrrt--t/a hawp:///bbrr--rg-mmmmim Iarge-mnli
losses. http://wnliipilleggtttteseaaaaan

---

ignore the different levels of similarity between different locations. Although it possible to take similarity into account, for example, by using em- beddings, this will introduce bias to the evaluation.
Model
Edit
SM
ACC
INDEPENDENT
0.64
0.19
0.36
GREEDY
0.58
0.32
0.42
ENTITY
0.7
0.17
0.28
One measure we used was Python's difflib Se- quenceMatcher (SM) score, which is based on the gestalt pattern matching metric (Ratcliff and Met- zener, 1988). This metric sums the longest com- mon substrings in recursive manner, and divides by the total length, attempting to reflect human impression for similarity. In this metric, a higher score means stronger similarity.
CHATGPT
0.72
0.19
0.27
CRF-G
0.57
0.27
0.42
CRF-L
0.53
0.38
0.47
HITRF
0.4
0.43
0.59
HITRF+EMB
0.37
0.49
0.63
Table Performance the various models for event- location tracking in Holocaust testimonies, in terms of Sequence Matching (SM), Edit distance, and Accuracy (AcC). For SM and Acc higher is better and for Edit lower is better. The models in the top part are the base- line models. CRF-G CRF-L are the global and local versions of the proposed CRF architecture, and HITRF and HiTRF+FEMB are the proposed hierarchical transformer architectures without and with location em- beddings. All models except CRF-L were trained with full gradient updates.
Another sequence measure we used is the Dam- -rae-Levenhhtein edit distance (Edit, Damerau 1964). This measure defines the distance between two sequences as the minimal number of insertions, deletions, substitutions, or transpositions in order to get from one sequence to the other. Since the number of edits depends on the number of elements in the sequence, we normalized the distance by the number of locations in the reference document." For the Edit distance, lower is better.
tracking compared to local methods. This includes both fine-tuned transformers and strong zero-shot models, such as ChatGPT, and it is also true for such models as LUKE that are specifically designed to be aware of named entities.
For completeness, we also report the element- wise accuracy (AcC) of the predictions.
6 Results
Scores for location tracking in Holocaust testi- monies are in Table 1. We see that the proposed models significantly outperform the baseline mod- els, with the best baseline (GreEDy) showing simi- lar performance to the lowest performing proposed model (CRF-G). Regarding the CRF models, we can see that using locally predicted transition ma- trices can improve performance. The hierarchical transformer models show improvements, with the best results for the location transformer model with location embeddings.
Among the baselines, we see that simple lo- cal classification obtains poor results. This in- dicates that the task benefits from context. The zero-shot models seem to struggle to comply with the strict prediction format. This might be due to the abundance of implicit locations. The greedy model shows relatively good results, but its infe- riority compared to the global models shows the importance of taking into account a larger context. Greedy's 's rough similarity in performance to the CRF-G model shows that the greedy models can balance the local signal (i.e., prediction based on the current text segment) and the transition signal (i.e., accounting for the previous location), without requiring global normalization.
We report one score for the greedy algorithm as we find that adding beam search had very little effect. For comparison, we mention here that fixed prediction for the most common category (which is "cities in Germany") gave much worse results: Edit = 0.84, SM = 0.07, and Accuracy = 0.11.
Among our models, we see that the hierarchi- cal transformer models perform better than the CRF models. This might be due to the larger con- text for the second level and the ability to capture more information in the first-level embeddings. In general, the experiment results follow pattern in which models that are more expressive in terms of global context have better performance. Hierar- chical transformer models are also robust to large sets of locations, in terms of memory, since the
7 Discussion
Our experiments show that document-level archi- tectures are significantly better at event-location
1he normalized distance might be larger than if the predicted number of topics is larger than the real number. This normalization is commonly known in the literature as word emoor rate.

---

embedding size is fixed. This is opposed to the CRF models, for which the transition matrix grows quadratically with the number of locations.
rectional model is not strong as the best models presented here, underscoring the importance of tak- ing into account larger context.
We also see that using specialized location em- beddings improves performance over the test set. The location embeddings are also important in or- der to allow better adaptation to other sets of loca- tions, such as more fine-grained categories, or even the exact location names, using the zero-shot capa- bilities of the embedding models compensate for sparsity of data.
An interesting feature of the CRF models is the ability to extract matrices with first-order transi- tion scores. In CRF-G, we have a single matrix for the entire set of testimonies. In CRF-L, we have function that assigns unique matrix for a given segment. Although the matrix depends on the segment, our inspection shows that the gen- erated matrices tend to be similar, allowing us to observe both general and segment-specific trends. In appendix E we present examples of matrices and discuss the trends observed in them.
Manual inspection of the HITRE outputs, shows that most cases the predicted trajectory very similar to the gold one. The cases which the performance was low seem to involve origin coun- tries that are less frequent in the testimonies (like Lithuamia or Luxembourg).
Although our experiments focus on the test case of survivor testimonies, the techniques developed here are applicable more broadly. To validate this claim, we report in Appendix results for an ad- ditional test domain. The results show a similar trend, thereby lending support to our claim as to the importance of viewing this task as global (document-level) task.
We also consider metrics that asses the models' performance in predicting location changes, while ignoring the exact location. For this, we use the FI- score for binary prediction of whether a location change is made after a given segment. Interestingly, we find that with this metric, the greedy model (F1 -0299) and CRF-L (FI - 0.24) outperform the HITRF models (F1 & 0.17). CRF-G (F1 = 0.16) and INDEPENDENT (F1 = 0.1) under-perform the HITRF models. This result implies that the pre- dicted trajectories, while fairly accurate, are not reliable enough as a means of segmentation. That is, it seems that the HITRF models succeed in cap- turing an overview of the testimonies, but struggle with exact boundarics, while models with local- transition components are more accurate in this regard.
8 Conclusion
We presented the task of event-location tracking in narrative texts and proposed various methods for tackling it.
We showed that our methods significantly out- perform several baseline methods in terms of the trajectory of locations that the model generates. Be- tween architectures, we showed that additional con- text improves results. We also showed that using location embeddings improves results, in addition to their importance for possible generalization. We corroborate our findings in another domain.
In order to learn about the impact of bidirec- tionally on performance (separately from the im- pact of structured prediction), we designed an ad- ditional oracle baseline, based ensemble of greedy models. We trained greedy model pre- dict a location given the following locations and segments (in a similar manner to the greedy base- line). Its results were slightly inferior to the origi- nal greedy model. We then constructed an oracle baseline in which we predict sequences in both directions and choose the backward predicted loca- tion if it is the correct one. The results were Edit 0.41, SM 0.45 and Accuracy 0.59. These results are similar HITRF but still not as good as HrTRF+EMB. despite the oracle information available to this method. These results then show that even under idealized conditions, a simple bidi-
In addition to the technical contribution of this work, it also makes important first steps in analyz- ing spoken testimonies in systematic, yet ethi- cal, manner. As such, our work contributes to the domain of Holocaust and memory studies, design- ing new methods of browsing and analysis that augment the current efforts of scholars and prac- titioners alike. More broadly, this work makes a significant contribution to the growing field of com- putational literary studies by offering new models of narrative analysis, based on event locations.
Limitations
Our experiments involve a specific and unique do- main. While we did test on an additional domain, the main data and conceptual framework, such as

---

the location categories, are limited in this respect. Nevertheless, the domain provided an important well-motivated test case also from the NLP per- spective.
In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lishon, Portugal. Association for Compu- tational Linguistics.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval.
Ethical Considerations
We abided by the instructions provided by each of the archives. We note that the witnesses identified themselves by name, and so the testimonies are not anonymous. Still, we do not present in the analy- sis here any details that may disclose the identity of the witnesses. We intend to release our code- base and scripts, but those will not include any of the data received from the archives; the data and trained models used in this work will not be given to a third party without the consent of the relevant archives. The testimonies can be made accessible for browsing and research by requesting permission from the SF archive.
Fred J. Damerau. 1964. A technique for computer de- tection and correction of spelling errors. Commun. ACM, 1737777
Federico Dassereto, Laura Di Rocco, Giovanna Guer- rini, and Michela Bertolotto. 2020. Evaluating the effectiveness of embeddings in representing the struc- ture of geospatial ontologies. In Geospatial Technolo- gies for Local and Regional Development: Proceed- ings of the 22nd AGILE Conference on Geographic Information Science 22, pages 41-57, Springer.
Ori Ernst, Avi Caciularu, Ori Shapira, Ramakanth Pa- sunuru, Mohit Bansal, Jacob Goldberger, and Ido Dagan. 2022. Proposition-level clustering for multi- document summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. pages 1765-1779, Seat- tle. United States. Association for Computational Linguistics.
Acknowledgements
The authors acknowledge the USC Shoah Founda- tion- - The Institute for Visual History and Educa- tion for its support of this research. We thank Prof. Gal Elidan, Dr. Gabriel Stanovsky, and Noam Weis for their valuable insights, and Nicole Gruber, Ye- lena Lizuk, Maayan Aharoni, Inbal Neumark, and Asaf Gershon for their research assistance. This research was supported by grants from the Israeli Ministry of Science and Technology, the Israeli Council for Higher Education, and the Alfred Lan- decker Foundation.
Evelyn Gius and Michael Vauth. 2022. Towards an event based plot model. computational narratology approach. Journal of Computational Literary Studies, 1(1).
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced beri with disentangled attention.
Andreas Hotho, Fotis Jannidis, Lucas Kaufmann, Markus Krug, Frank Puppe, Nils Reiter, and An- nekea Schreiber. 2021. Shared task on scene segmen- tation konvens 2021.
References
Maria Antoniak, David Mimno, and Karen Levy. 2019.
Xuke Hu, Yeran Sun, Jens Kersten, Zhiyong Zhou, Friederike Klan, and Hongchao Fan. 2023. How can voting mechanisms improve the robustness and generalizability of toponym disambiguation? Inter- national Journal of Applied Earth Observation and Geoinformation. 117:103191.
Narrative paths and negotiation of power in birth sto- ries. Proceedings of the ACM on StuaanCoompun Interaction, 3(CSCW):1-27.
Ron Artstein, Alesia Gainer, Kallirroi Georgila, Anton Leuski, Ari Shapiro, and David Traum. 2016. New dimensions in testimony demonstration. In Proceed- ings of the 2016 Conference of the North American Chapter of the Association, for Computational Lin- guistics: Demonstrations, pages 32-36, San Diego California. Association for Computational Linguis- tics.
Xuke Hu, Zhiyong Zhou, Hao Li, Yingjie Hu, Fuqiang Gu, Jens Kersten, Hongchao Fan, and Friederike Klan. 2022. Location reference recognition from texts: A survey and comparison. arXiv preprint arXiv:2207.01683.
Jiang Bian, Dayong Tian, Yuanyan Tang, and Dacheng Tao. 2018. A survey on trajectory clustering analysis.
Mayank Kejriwal and Pedro Szekely. 2017. Neural embeddings for populated geonames locations. The Semantic Web-ISWC 2017: 16th International Semantic Web Conference, Vienna, Austria, October 21-25. 2017, Proceedings, Part 16, pages 139-146. Springer.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. large anno- tated corpus for learning natural language inference.

---

Abhinav Kumar and Jyoti Prakash Singh. 2019. Loca- tion reference identification from tweets during emer- gencies: A deep learning approach. International Journal of Disaster Risk Reduction, 33:365-375.
Sandeep Soni, Amanpreet Sihra, Elizabeth F. Evans, Matthew Wilkens, and David Bamman. 2023. Grounding characters and places in narrative texts.
Sharifa Sultana, Renwen Zhang, Hajin Lim, and Maria Antoniak. 2022. Narrative datasets through the lenses of nlp and hci.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Proba- bilistic models for segmenting and labeling sequence data.
Chenyu Tian, Yuchun Zhang. Zefeng Weng, Xiusen Gu, and Wai Kin Chan. 2022. Learning fine- one-grained lo- cation embedding from human mobility with graph neural networks. In 2022 International Joint Confer- ence on Neural Networks (IJCNN). IEEE
Quoe Le and Tomas Mikolov. 2014. Distributed repre- sentations of sentences and documents. In Interna- tional conference on machine learning, pages 1188- 1196. PMLR.
Eitan Wagner, Renana Keydar, Amit Pinchevski, and Omri Abend. 2022. Topical segmentation of spoken narratives: A test case on holocaust survivor testi- monies. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6809-6888 6821, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with ap- plication to the biography domain. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 1203-1213, Austin, Texas. Association for Computational Linguistics.
Jimin Wang and Yingjie Hu. 2019. Enhancing spatial and textual analysis with eupeg: An extensible and unified platform for evaluating geoparsers. Transac- tions in GIS, 23.
Andrew Kwok-Fai Lui, Yin-Hei Chan, and Man-Fai Le- ung. 2021. Modelling of destinations for data-driven pedestrian trajectory prediction in public buildings. In 2021 IEEF International Conference on Big Data (Big Data), pages 1709-1711
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad- coveraee challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Wesley Mathew, Ruben Raposo, and Bruno Martins. 2012. Predicting future locations with hidden markov models. In Proceedings of the 2012 ACM Confer- ence on Ubiquitous Computing, UbiComp 12, page 911-918. New York, NY. USA. Association for Com- puting Machinery.
Andrew Piper and Sunyam Bagga. 2022. Toward a data- driven theory of narrativity. New Literary History, 54(1):879-95
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextualized entity representations with entity- aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 6442-6454, On- line. Association for Computational Linguistics.
Andrew Piper, Richard Jean So, and David Bamman. 2021. Narrative theory for computational narrative understanding. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Processing, pages 298-311, Online and Punta Cana. Dominican Republic. Association for Computational Linguistics.
Bingsheng Yao, Ethan Joseph, Julian Lioanag, and Mei Si. 2022. A corpus for commonsense inference story cloze test. In Proceedings the Thirteenth Lan- guage Resources and Evaluation Conference, pages 3500-3508, Marseille, France. European Language Resources Association.
John W Ratcliff and David E Metzener. 1988. Pattern- matching-the the gestalt approach. Dr Dobbs Journal, 13(7):46.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNIIP), pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Albin Zehe, Leonard Konle, Lea Katharina Dümpel- mann, Evelyn Gius, Andreas Hotho, Fotis Jannidis, Lucas Kaufmann, Markus Krug, Frank Puppe. Nils Reiter, Annekea Schreiher, and Nathalie Wiedmer, 2021. Detecting scenes in fiction: A new segmenta- tion task. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: Main Volume, pages 3167-3177, Online. Association for Computational Linguistics.
Abdessamed Sassi, Mohammed Brahimi, Walid Bechkit, and Abdelmalik Bachir. 2019. Location embedding and deep convolutional neural networks for next location prediction. In 2019 IEEE 44th LCN Symposium on Emerging Topics in Networking (LCN Symposium), pages 149-157.
Zeyu Zhang and Steven Bethard, 2023. Improving to- ponym resolution with better candidate generation, transformer- reranking, and two-stage resolu- tion.

---

A List of Location Categories
30. cities in Argentina
1. START
31. citics in Australia
2. END
32. cities in Austria
3. German concentration camps in Austria
33. cities in Belgium
4. German concentration camps in Belgium
34. cities in Bulgaria
5. German concentration camps in Czechoslo- vakia
35. citics in Canada
36. cities in Chile
6. German concentration camps in Danzig (FC)
37. citics in China
7. German concentration camps Estonia
38. cities in Colombia
8. German concentration camps in France
39. cities in Cuba
9. German concentration camps in Germany
40. cities in Cyprus
10. German concentration camps in Latvia
41. citics in Czechoslovakia
11. German concentration camps in Poland
42. cities in Danzig (FC)
12. German concentration camps in the Nether- lands
43. cities in Denmark
44. cities in Dominican Republic
13. German concentration camps the USSR
45. citics in Ecuador
14. German death camps in Poland
46. cities in Egypt
15. German prisoner of war camps in France
47. cities in Estonia
16. German prisoner of war camps in Germany
48. cities in Finland
17. Slovakian concentration camps in Czechoslo- vakia
49. citics in France
18. Soviet concentration camps in the USSR
50. cities in Germany
19. administrative units in Belgium
51. cities in Ghana
20. administrative units in Canada
52. citics in Greece
21. administrative units in Denmark
53. cities in Hungary
22. administrative units in Hungary
54. cities in Iran
23. administrative units in Israel
55. cities in Iraq
24. administrative units in Italy
56. cities in Ireland
25. administrative units in Lithuania
57. cities in Israel
26. administrative units in Romania
58. cities in Italy
27. administrative units in Spain
59. cities in Japan
28. administrative units in the Netherlands
60. cities in Latvia
29. administrative units in the United Kingdom
61. cities in Libya

---

62. cities in Lithuania
91. ghettos in Hungary
63. cities in Luxembourg
92. ghettos in Latvia
64. cities in Mexico
93. ghettos in Poland
65. cities in Morocco
94. ghettos in Romania
66. cities in New Zealand
95. ghettos in the Netherlands
67. cities in Nigeria
96. ghettos in the USSR
68. cities in Norway
97. kibbutzim in Israel
69. cities in Paraguay
98. moshavim in Isracl
70. cities in Poland
99. refugee camps in Australia
71. cities in Portugal
100. refugee camps Austria
72. cities in Romania
101. refugee camps in Italy
73. cities in South Africa
102. refugee camps in Jamaica
74. cities in Spain
103. refugee camps Sweden
75. cities in Sweden
104. refugee camps in Switzerland
76. cities in Switzerland
105. refugee camps in the Netherlands
77. cities in Uruguay
106. refugee camps the United Kingdom
78. cities in Yugoslavia
107. refugee camps in the United States
B ChatGPT Prompt
79. cities in the Netherlands
For the ChatGPT baseline, for the first segment, we input the following prompt with the "system" role (filling in the actual list):
80. cities in the USSR
81. cities in the United Kingdom
82. cities in the United States
"Here is list of Holocaust-related event location categories:
83. displaced persons camps or installations in Austria
<Here comes the list of categories in ap- pendix A>
The user will give you a text segment and you will give back the location, out of the previous list, where the events in the segment took place. We emphasize that we want the event locations (from the list) and not the location of the inter- view. Give only the name of the category from the list with no additional text. If there is no location give the one that is the most probable.
84. displaced persons camps or installations in Germany
85. displaced persons camps or installations in Germany: British zone
86. displaced persons camps or installations in Germany: US zone
87. displaced persons camps installations in Italy
Then we input the following with the "user" role (filling in the actual text):
88. ghettos in China
89. ghettos in Czechoslovakia
Segment: <Here comes the text seg- ment>.
90. ghettos in Greece

---

From the response, we extract a predicted location. After a response was received we input the last query and response as part of the message history, addition to the instructions. In cases where the response did not fit to any location in the list, we did not use it as part of the message history.
Model
Edit
SM
IND.
0.16
0.83
CRF-G
0.18
0.83
CRF-L
0.2
0.8
HITRE
0.11
0.89
HITRF+EMB
0.2
0.79
Location Embedding Training
Table 2:
Performance for NBA biography team-
The training of location embeddings was done NLI training with Multiple Negative Ranking loss. The first of NLI examples was general- domain NLI data, which is combination of the SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2). We trained with learning- rate=2-107>
tracking.
D Location Tracking in Biographies
As validation, we implemented and evaluated our methods on another test domain.
We also obtained sets of domain -opeiin-peecffic ex- amples. One set was generated from our annotated testimonies. For each segment that has explicit location label, generated entailment exam- ples of the form:
Data. We extracted from Wikipedia professional biographies of basketball players who played in the NBA. We applied our methods to the task of predicting the professional stage for each paragraph in the article.
Starting with the WikiBio dataset (Lebret et al., 2016), we selected all the biographies of basket- ball players who played in the NBA. We then ob- tained the full Wikipedia article (retrieved May 11, 2023) and filtered out the summary and sections that do not describe professional stages. We used the Wikipedia subsection titles as labels for each paragraph in the section.
<text segment> ==> "The event location is <label>" <text segment> ==> "The event location category is <label category>".
For each entailment example, we also used the opposite direction as entailment. Also, we created a contradiction example with different location (or location category).
We used labels for all current teams and legacy teams. We added labels for early careers, coaching and executive careers, and non-NBA teams. The final label set consists of 35 labels. For the full list see D.1.
Another set generated from the SF descrip- tion for the locations and categories. For example, the location "Warsaw (Warsaw, Poland)" has the description:
Coordinates: 52"15'N E Capi- tal of Poland 1929 Voivodship: War-
The process yielded 655 biographies with an average of 16.9 labeled paragraphs. The average length is 1300 words.
saw Jewish population 1939; 393,950
1900-1917: Russia 1917-1918: Occu-
Training. We followed the same training pro- cedure as with the Holocaust data. We changed rearning-rate=10- for the segment-embedding transformer.
pied by Germany 1918-1939: Poland 1939- 1944: Occupied by Germany (Gen- cralgouvernement) 1944-1945; Liber-
ated by Soviet troops 1945- Poland.
For each of these descriptions, generated an NLI example of the form:
Results and Discussion. For the NBA dataset, the location embeddings hurt the performance, even compared to the local classifier. In general, since the local classifier has such good perfor- mance, it proved difficult to add extra components without hurting the performance. Nevertheless, we still found that the of hierarchical transformers yields a substantial improvement over the indepen- dent predictions. This shows that the global context is beneficial.
<description> => =>> "The event location is <location>".
We used the opposite direction too and took random other description contradiction.
We remark that although in our experiments we predicted the location categories, we still trained
the vectors for detailed locations.

---

It's worth mentioning that we assume that the transformer models saw the NBA data with the headers during pretraining. This might be a reason for the strong performance of the local classifica- tion model. The LUKE model might have already learned useful embeddings, which outperform the ones we use here.
27. Orlando Magic
28. Philadelphia 76ers
29. Phoenix Suns
30. Portland Trail Blazers
31. Retirement
D.1 List of NBA Biography Categories
32. Sacramento Kings
1. Atlanta Hawks
33. San Antonio Spurs
2. Boston Celtics
34. Toronto Raptors
3. Brooklyn Nets
35. Utah Jazz
4. Charlotte Horncts
36. Washington Wizards
5. Chicago Bulls
E Transition Matrices
6. Cleveland Cavaliers
We attach here (partial) heatmaps of the transition matrices obtained by the CRF models.
7. Coaching career
In figure is a heatmap for the 35 most common location categories, obtained by the global CRF model. We can see some trends:
8. Dallas Mavericks
9. Denver Nuggets
10. Detroit Pistons
For any category the most probable next step is to stay in the same category.
11. Early careers
After START the next common category is "concentration camps in poland". In general, after START, the common categories are in European countries.
12. Executive career
13. Golden State Warriors
14. Houston Rockets
Figures 4 and 5 are heatmaps for the 35 most common location categories, obtained by the local CRF model. Figure 4 is obtained for segment describing transport to camp and figure is obtained for segment describing life in Tel Aviv after the war. We see that:
15. Indiana Pacers
16. Los Angeles Clippers
17. Los Angeles Lakers
18. Memphis Grizzlies
The two matrices are very similar. This means that although the transitions depend on spe- cific segments, it is still possible to extract some global trends.
19. Miami Heat
20. Milwaukee Bucks
21. Minnesota Timberwolves
The second matrix has higher values in gen- eral. This might mean that this segment is interpreted as having more "mobility".
22. New Orleans Pelicans
23. New York Knicks
As in the matrix for the global transitions, the highest values are for remaining in the same category. Also, there are similar patterns for transitions from START.
24. Non-NBA team
25. Oklahoma City Thunder
26. Old team

---

Global Iransition Matrix
400
Deekk
Seual Wher
den teeee
Dow
Figure 3: Heatmap of the global transition matrix for global CRF model. We plotted only the 35 most frequent categories.

---

Local Tonntt on Horts Segnent
Figure 4: Heatmap a local transition matrix for the local CRF model for segment describing a transport to a camp. We plotted only the 35 most frequent categories.
Figure 5: Heatmap of local transition matrix for the local CRF model for segment describing life in Tel Aviv after the war. We plotted only the 35 most frequent categories.


Reviews:
[{"review_id": "arU5X3D4D1", "reviewer": "Reviewer_adQB", "summary": "", "questions": "Question a: The motive to use category names (e.g., \"Ghettos in Poland\") instead of exact location names (e.g., \"Warsaw (Warsaw, Poland: Ghetto)\") should be clearly explained along with its potential implications if any", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We will address the issues you raised and incorporate your suggestions.\n\n**Regarding the Complexity vs. Performance Trade-off (reasons to reject #1):**\n> The paper should clearly discuss the trade-off between complexity and performance to provide practical insights for implementation.\n\nWe will discuss this. In our experiments, the document-level models were significantly more complex than the baselines, but the performance was also significantly better. The time complexity of the deep CRF models and the HiTrf models was similar. The deep CRF ones were actually a bit more time-consuming, as they required more training epochs (as we reported in section 5.2). We did mention the complexity of the models regarding interpretability (end of section 7).\n\n**Regarding Heuristic Labeling, you wrote (reasons to reject #2):**\n> The process of selecting location labels and handling segments without location labels involves heuristic approaches. This might introduce biases or errors in the labeling process, potentially affecting the quality of the training data.\n\nThe selection of location labels and categories was based on the SF thesaurus hierarchy, which was created by Holocaust research experts, so it was not arbitrary. The full thesaurus may be acquired from the SF together with the testimonies. As we mention in the paper, the annotation for segments without explicit labels was based on heuristics, and we realize that it introduces noise. For this reason, annotators manually corrected the test set so the heuristics’ effect on evaluation was minimal.\n\n**Regarding Label Granularity, you wrote (reasons to reject #3):**\n> The decision to use category names instead of exact location names might impact the precision of location tracking. The paper should discuss the implications of this choice and its potential effect on performance.\n\nThe main reason for choosing the categories was to reduce the number of categories (from >1500 exact locations to ~100 categories). Reducing the number of categories inevitably results in some loss of information, but the categories still provide rich information useful for many tasks such as location-based segmentation, alignment between testimonies, and creation of high-level summaries. We will add a discussion about this. \n\n**Regarding the motive to use category names, you wrote (questions for authors a):**\n> The motive to use category names (e.g., \"Ghettos in Poland\") instead of exact location names (e.g., \"Warsaw (Warsaw, Poland: Ghetto)\") should be clearly explained along with its potential implications if any\n\nSee response to reason to reject #3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Architectural Exploration: The paper systematically explores various architectures for solving the event-location tracking task, ranging from global contextual models to narrow context baselines. This in-depth analysis provides valuable insights into the effectiveness of different approaches, aiding researchers and practitioners in selecting appropriate methods for similar tasks.\n\n2. Location Embeddings: The introduction of methods for generating location embeddings, and the demonstration of their benefits over traditional string-based location representations, can have broader implications for representing geographical information within text data.\n\n3. Ethical and Historical Relevance: The paper's focus on Holocaust survivor testimonies demonstrates a deep ethical and historical relevance. It showcases how NLP can contribute to the understanding and preservation of significant historical events and narratives.\n", "reasons_to_reject": "1. Complexity vs. Performance Trade-off: The more advanced architectures may come with increased computational complexity. The paper should clearly discuss the trade-off between complexity and performance to provide practical insights for implementation.\n\n2. Heuristic Labeling: The process of selecting location labels and handling segments without location labels involves heuristic approaches. This might introduce biases or errors in the labeling process, potentially affecting the quality of the training data.\n\n3. Label Granularity: The decision to use category names (e.g., \"Ghettos in Poland\") instead of exact location names (e.g., \"Warsaw (Warsaw, Poland: Ghetto)\") might impact the precision of location tracking. The paper should discuss the implications of this choice and its potential effect on performance.", "questions_to_authors": "Question a: The motive to use category names (e.g., \"Ghettos in Poland\") instead of exact location names (e.g., \"Warsaw (Warsaw, Poland: Ghetto)\") should be clearly explained along with its potential implications if any", "paper_topic_and_main_contributions": " The paper introduces and explores the task of event-location tracking in narrative texts, proposes novel architectures with varying levels of context awareness, compares them against baselines, introduces methods for generating location embeddings, focuses on the Holocaust survivor testimonies as a case study, and validates the findings across different domains. The contributions of this paper lie in its formulation of a new task, the exploration of multiple architectural approaches, and the insights gained from analyzing the spatial dimension of narratives.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jXPqNCiUgJ", "reviewer": "Reviewer_VE7r", "summary": "", "questions": "* L072-L076 and also L225-L231: can one can predict locations with NER and then filter out locations that are not part of the main event?\n* L078-084: Can you please clarify the concept of trajectory? how the “post-war theme” is an event location?\n* L102: “Larger context capabilities”: is there evidence to suggest that this is due to larger context or the nature of the task/dataset?\n* L396: As mentioned in 479-480, BART-large was used as for zero-shot classification? Why not training this method instead of using a zero-shot classifier? Have you tried to use other models such as Flan-T5?\n* L511: What form of gradient accumulation has been used?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review. We address here some of your concerns and we will incorporate many of your suggestions.\n\n**Regarding the results and their implications on structured prediction, you wrote (reasons to reject #1):**\n> The proposed CRF variants underperform the hiererchical transformers (HiTRF). This shows that, for this task, global context is more important than modeling sequential information, which in turn contradicts the focus of the paper on modeling dependencies as a structured prediction task.\n\nWe are not sure we understand the point about the potential contradiction. By “structured prediction” we mean that the task should be addressed as predicting a whole structure and not independent local values. A common example of structured prediction is NER in IOB format, where an I label cannot come after a B label, so we cannot independently predict a label per token. Examples of non-structured prediction are most text classification tasks. In our case, the predicted structure is one sequence, corresponding to the whole document, and not a sequence of independent predictions for each segment. The reason for this is as we mentioned, that the sequence of locations exhibits dependencies between different parts of the document. By this definition, the decision is actually supported by our findings that models with larger contextual capabilities perform better.\n\n**Regarding the benefits of location embeddings for other domains, you wrote (reasons to reject #1):**\n> Moreover, location embeddings seem to provide additional information on the main dataset but not in the extra dataset (Appendix D). This may suggest that they can be useful for specific types of datasets only, but there is no discussion about this in the paper.\n\nThis part seems to be a different concern. As we mention briefly in section 4.2, we believe that embeddings are beneficial for two reasons - (1) the ability to leverage the internal structure (like geographic proximity) of the labels, which is relevant for labels with descriptions, and (2) the ability to use the zero-shot capabilities of the models, which is mostly beneficial in cases of labels that are common in the unsupervised data but less in the supervised data. In the testimony data, the internal structure (1) follows the descriptions of the locations (from the SF Thesaurus), in which connections between locations are mentioned (e.g., for “Berlin”, the description includes its coordinates, its distance to Hamburg, and being part of Prussia). The zero-shot capabilities (2) were important because of the large number of possible locations (even if we measure prediction of the location categories, this is still based on mentions in the text, in which the number of mentioned locations is very large). In the Biography data, the local fine-tuned classifier showed very good performance (as described in Appendix D, results and discussion), so it seems that for the list of possible teams, which is not large, the zero-shot capabilities (2) are less crucial. It is true that the domain also seems significant, which is related to the internal structure (1). It seems that for the NBA team case, the geographic layout was less important compared to other factors, as geographical proximity does not seem like a significant factor in team switching. Other types of connections are possible but require explicit data which we did not have. We note that the biography domain mainly served as a proof of concept to demonstrate replicability across domains. Another consideration, which we mentioned in the discussion (in the appendix), is that the baseline method is strong, implying that the local signal is dominant. This means that adding the contextual signal is delicate, as we do not want to weaken the local one.\n\n**Regarding the analysis of failure cases of the HiTrf models, you wrote (reasons to reject #3):**\n> The paper is lacking analysis on failure cases for the best model (HiTRF), and discussion on why certain cases fail. \n\nWe will add a discussion of this. Thank you for the suggestion.\n\n**Regarding the idea to filter out non-event locations, you wrote (questions for authors #1):**\n>  can one can predict locations with NER and then filter out locations that are not part of the main event?\n\nWe actually tried similar approaches (like extracting all named locations and training a classifier based on this data). The problem is that identifying the location of the main event is hard. In addition, this approach failed miserably with segments that do not mention the main location (and sometimes do mention other locations). We note that even the locations that are relevant can be on different scales (city/country/district), or with different forms (requiring entity linking which is a problem in itself). \n\n**Regarding trajectories and the post-war theme, you wrote (questions for authors #2):**\n> Can you please clarify the concept of trajectory? how the “post-war theme” is an event location?\n\nThe post-war theme is not an event location but rather a concept guiding the locations. What we try to convey in this example is the fact that migration from Europe to the US is very different from a post-war migration to a European country, in which the hometown usually plays a stronger role. Because migration to the US is a common post-war theme, it should not be viewed in the same way as migration to Europe, and both post-war migrations should not be viewed in the same way as other location changes before or during the war.\n\n**Regarding larger context capabilities, you wrote (questions for authors #3):**\n>  is there evidence to suggest that this is due to larger context or the nature of the task/dataset?\n\nIn this paragraph, we just mention the correlation between context modeling abilities (which are inherent to the proposed architectures) and performance. Since the data is the same across the models, the most immediate potential conclusion is to ascribe this to context modeling capabilities. Of course, it is possible that our results are limited to specific properties of our domain, although we did try to address this point by experimenting in another domain.\n\n**Regarding the different models for zero-shot classification, you wrote (questions for authors #4):**\n>  BART-large was used as for zero-shot classification? Why not training this method instead of using a zero-shot classifier? Have you tried to use other models such as Flan-T5?\n\nWe do not expect this to make much difference, since the main purpose of the zero-shot models is to leverage the connection between the mentioned locations and the label name. We interpret our findings as suggesting that the difficulty in the task is the implicitness of the locations (and the importance of context) and not the specific model’s identity. The same holds true for other similar-scale models. For this issue, training will be difficult, as we do not have many high-quality labeled segments with implicit-only locations. Also, in general, for closed-set classification with sufficient data, zero-shot methods (without large scale) are not much better than encoder-based ones. \n\n**Regarding the special form of gradient accumulation, you wrote (questions for authors #5):**\n> What form of gradient accumulation has been used?\n\nWe will clarify this in the paper. What we do is the following: in the forward pass, one transformer ($E_{\\theta_1}$) receives segments and outputs corresponding (pooled) encoding vectors, and a second transformer ($E_{\\theta_2}$) receives all these vectors and outputs a sequence of locations. The loss is calculated by the outputs of  $E_{\\theta_2}$, which are determined only after all segments were encoded (i.e., the input segments cannot be divided into batches). Encoding all segments before updating requires many copies of the computational graph, which requires a large amount of GPU memory. What we do is to run a forward pass for the $E_{\\theta_1}$ without gradients. With these encodings we run a forward pass for $E_{\\theta_2}$, with gradients, compute the loss, and backpropagate through $E_{\\theta_2}$. With these (partial) gradients we run again through $E_{\\theta_1}$, this time with gradients, but the previous calculations allow the use of batches. This whole procedure is purely for memory efficiency and does not affect the actual outputs and gradients.  \n\n**Regarding the correspondence between the input and prediction sequences, you wrote (typos and improvements #1):**\n> What is $x$? a sequence of paragraphs/sentences/tokens? does the predicted sequence of locations $y$ correspond to each element in $x$?\n \nIn the first paragraph, we describe the task in general, for any segmentation of the document into spans. The sequence corresponds element-wise (we will clarify this). Afterward, in L222 we mention that we use spans of multiple sentences, and in L425 we describe our data.\n\n**Regarding examples for implicitly mentioned locations, you wrote (typos and improvements #2):**\n> Please provide an example that motivates that implicitly mentioned locations\n\nWe’ll add an example (here’s a simple one - when an event occurs over multiple segments, the location might be mentioned explicitly only in the first one).\n\n**Regarding pooling methods, you wrote (typos and improvements #3):**\n> please explain inputs and outputs, i.e., how tdo you agreggate transformer outputs?\n\nWe used the default pooling module in the HuggingFace-transformers models, in the same manner that is used for sequence classification heads. Specifically, for sequence encoding we used LUKE, where the LukePooler applies a linear transformation and tanh activation to the embedding of a special token that is placed at the beginning of the sequence. We will clarify this in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Interesting task of tracking locations in narratives.\n* Interesting dataset on Holocaust survivor testimonies that could potentially also be used in other NLP tasks in the future (e.g., QA).", "reasons_to_reject": "* The proposed CRF variants underperform the hiererchical transformers (HiTRF). This is the case both for the main dataset and for the dataset discussed in Appendix D.  Therefore the focus of the paper on modeling this task using a CRF (local and global context) should be down-graded. Moreover, location embeddings seem to provide additional information on the main dataset but not in the extra dataset (Appendix D). This may suggest that they can be useful for specific types of datasets only, but there is no discussion about this in the paper.\n* More precise notation and details are needed in Sections 3 and 4 (see comments in Questions to Authors and Presentation improvements). \n* The paper is lacking analysis on failure cases for the best model (HiTRF), and discussion on why certain cases fail. There’s some analysis in the Appendix but only for the CRF variants that underperform HiTRF.", "questions_to_authors": "* L072-L076 and also L225-L231: can one can predict locations with NER and then filter out locations that are not part of the main event?\n* L078-084: Can you please clarify the concept of trajectory? how the “post-war theme” is an event location?\n* L102: “Larger context capabilities”: is there evidence to suggest that this is due to larger context or the nature of the task/dataset?\n* L396: As mentioned in 479-480, BART-large was used as for zero-shot classification? Why not training this method instead of using a zero-shot classifier? Have you tried to use other models such as Flan-T5?\n* L511: What form of gradient accumulation has been used?", "paper_topic_and_main_contributions": "This paper proposes the task of event-location tracking in narrative texts. It proposes a set of methods to address this task (CRF variants, hierarchical transformers), and it focuses the evaluation on Holocaust survivor testimonies. It finds that the hierarchical transformer obtains the best results both on the main dataset and on an additional dataset discussed in the Appendix.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* L212-L216: What is $x$? a sequence of paragraphs/sentences/tokens? does the predicted sequence of locations $y$ correspond to each element in $x$? Please clarify.\n* L238-239: Please provide an example that motivates that implicitly mentioned locations\n* L319: This has to be has to be more formal; please explain inputs and outputs, i.e., how tdo you agreggate transformer outputs (e.g., mean pooling)?\n* L359: Please provide a reference to section 5.1 where more details are provided.\n* L511: Please provide more details"}}, {"review_id": "owtLidY1nB", "reviewer": "Reviewer_vwcJ", "summary": "", "questions": "For the gold location sequences, is it the case that they always reflect the chronological order of locations in the story? I am wondering if locations can be given out of order, eg. \"We had moved to City B from City A the year before,\" where City A is not mentioned in any other sentence. Section 5.1 mentions the special case of visits; were any other special cases observed in the dataset? \n\nI am not sure that sequence-labeling metrics are the most appropriate to evaluate this task. For most of the segments, the label does not matter; the authors mention that, for most of the segments, the label is the same as that of the previous segment (and they use this heuristic to perform gold labeling). So if, for example, the gold location sequence is City A, City A, City A, City B, City B, then the second City A prediction seems to me less important than the first City B prediction. I would like to see some segmentation metrics added to the evaluation to focus on whether movement between locations is successfully predicted.\nUpdate: This was addressed in the author rebuttal.\n\nDo you have any intuition as to why the Greedy baseline underperforms Independent? I would have thought that access to some sequence information would be better than none, and it is not affected by the zero-shot baselines' formatting problems. I am also curious if you tried ensembling the Greedy baseline with a reversed version? Is global information required for good performance, or simply bidirectionality?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of our work and suggestions. We will address some issues you raised and incorporate your suggestions.\n\n**Regarding the filtering of labels, you wrote (reasons to reject #1):**\n>The dataset used in the main experiments are not freely available (requiring permission from the Shoah Foundation archive), and the filtering step seems somewhat subjective, so the experimental results would be hard to reproduce. While the proposed approaches are domain-agnostic, they do require there to be a fixed number of location labels, making them hard to apply to open-domain narratives.\n\nThe selection of location labels was based on the SF thesaurus hierarchy, which was created by Holocaust research experts. The thesaurus can be obtained for research purposes from the SF together with the testimonies, so this should not affect reproduction. It’s true that the methods require adaptation in order to generalize. We do, however, provide an additional dataset showing that generalization is possible.\n\n**Regarding unordered locations, you wrote (questions for authors #1):**\n> For the gold location sequences, is it the case that they always reflect the chronological order of locations in the story? I am wondering if locations can be given out of order\n\nThe testimonies were conducted with general guidelines, and the order of events is mostly chronological, from childhood to post-war experiences. Nevertheless, the testimonies were not edited, and the stories can jump between events, times, and locations.\n\n**Regarding other metrics, you wrote (questions for authors #2):**\n> I am not sure that sequence-labeling metrics are the most appropriate to evaluate this task. \n>\n> I would like to see some segmentation metrics added to the evaluation to focus on whether movement between locations is successfully predicted.\n\nSince we also assume a correspondence between the text and locations (inducing a segmentation, for example), we do see importance in the exact sequence. We experimented with measures that are less sensitive to this (e.g., DTW or F1 score over changes only), but they were highly correlated to the sequence-evaluation methods and omitted for brevity. We will add them to the paper.\n\n**Regarding the greedy baseline’s performance and bidirectionality, you wrote (questions for authors #3):**\n> Do you have any intuition as to why the Greedy baseline underperforms Independent?\n>\n> I am also curious if you tried ensembling the Greedy baseline with a reversed version? Is global information required for good performance, or simply bidirectionality?\n\nAs we mentioned in the discussion, it seems to be due to the delicate balance between the local signal and the global one (which includes the tendency to stay in the same place). For example, in cases where the location clearly changed but it’s not so obvious what the new location is, the greedy models showed a strong tendency to stay in the same place. This is in contrast to independent predictions which tended to change locations too often.\nWe did not explore a bidirectional approach in the greedy baseline (although the CRF and HiTRF models are essentially bidirectional). Since we believe the issue is the tendency to stay in the same place, we believe that it won’t make a significant difference. We will check this empirically to verify.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed task is interesting and challenging. The authors motivate it well and chose a remarkably clean and well-suited dataset to demonstrate the task. The proposed approaches are sensible and perform well in the experiments.", "reasons_to_reject": "The dataset used in the main experiments are not freely available (requiring permission from the Shoah Foundation archive), and the filtering step seems somewhat subjective, so the experimental results would be hard to reproduce.\nUpdate: This second half of this concern was addressed in the author rebuttal.\n\nWhile the proposed approaches are domain-agnostic, they do require there to be a fixed number of location labels, making them hard to apply to open-domain narratives.", "questions_to_authors": "For the gold location sequences, is it the case that they always reflect the chronological order of locations in the story? I am wondering if locations can be given out of order, eg. \"We had moved to City B from City A the year before,\" where City A is not mentioned in any other sentence. Section 5.1 mentions the special case of visits; were any other special cases observed in the dataset? \n\nI am not sure that sequence-labeling metrics are the most appropriate to evaluate this task. For most of the segments, the label does not matter; the authors mention that, for most of the segments, the label is the same as that of the previous segment (and they use this heuristic to perform gold labeling). So if, for example, the gold location sequence is City A, City A, City A, City B, City B, then the second City A prediction seems to me less important than the first City B prediction. I would like to see some segmentation metrics added to the evaluation to focus on whether movement between locations is successfully predicted.\nUpdate: This was addressed in the author rebuttal.\n\nDo you have any intuition as to why the Greedy baseline underperforms Independent? I would have thought that access to some sequence information would be better than none, and it is not affected by the zero-shot baselines' formatting problems. I am also curious if you tried ensembling the Greedy baseline with a reversed version? Is global information required for good performance, or simply bidirectionality?", "paper_topic_and_main_contributions": "The paper introduces a new task, narrative event-location trajectory extraction. The authors use a dataset of Holocaust survivor testimonies, which they convincingly argue is both extremely well-suited as a test bed for this new task (being a relatively large collection of topically-related narratives that take place in a small set of common locations) and a socially important domain in which to apply NLP techniques.\n\nThe authors encode the input narrative segments using a pretrained, transformer-based encoder and experiment with four sequence labeling approaches to predict the segment locations. Two approaches use a CRF on top of the encodings, one with a global location transition matrix learned from the entire training set, and one with a local transition matrix predicted for each segment from its encoding. The other two approaches use a second, hierarchical transformer encoder: one performs a standard multiclass probability prediction, and the other uses a pretrained location embedding matrix as the output layer. \n\nThe experimental results show that the proposed approaches outperform baseline methods that either do not use sequence information or perform local greedy prediction using the previous segment's predicted location. The authors also find that an LLM-based approach using ChatGPT did not perform well because it could not reliably produce predictions matching the given label set.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos:\n- Section 5.1, line 450: , -> ."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces the novel and socially impactful task of tracking event locations in personal narratives. The compelling application to Holocaust survivor testimonies underscores the moral importance of computationally analyzing such historical documents. The key technical contribution is a rigorous exploration of structured prediction architectures for extracting location sequences, including sequential conditional random fields (CRFs) and hierarchical transformers. Experiments demonstrate that modelling the global narrative context significantly improves prediction accuracy compared to greedy localized approaches.

The proposed task taps into a broader interest in narrative understanding and temporal commonsense reasoning. The testimony dataset appears remarkably well-suited, containing a large collection of topically-related personal accounts centered around a small set of canonical locations. However, transparency is needed around the subjective data filtering process to enable reproducibility without public access.

Reviewers praise the novelty, motivation, and dataset potential, but note a few areas for improvement. For example, Reviewer 2 believes the CRF focus should be downplayed given the superior performance of transformers. Reviewers 1 and 3 raise important concerns about aspects of the data creation process, such as the subjectivity of filtering location labels. Reviewer 3 offers multiple insightful suggestions like evaluating segmentation explicitly and analyzing bidirectionality. Overall the reviews indicate an intriguing task and technical approach while recommending improvements to evaluation, transparency, and framing.


Personally, I note the following: The CRF models underperform simple transformers, this suggests to me an overemphasis on sequential dependencies in framing and analysis. Additional segmentation metrics are recommended to better evaluate location trajectory extraction. Also, I am unconvinced about the results reported for ChatGPT. I have tried to replicate the task with ChatGPT and Claude and both appeared to have performed excellently on event location extraction. I concur that I have used different data for my experiments (mostly pseudo-text and not a real-life dataset like the one used in the paper). Also, I have used a simplified prompt than the example presented in the appendix of the paper. However, I do not think both factors should greatly impact the outcome. I think this calls for a more in-depth analysis on the part of the authors, especially to justify their approaches against out-of-the-box finetuning of LLMs for the same task. Also, an in-depth analysis of model failures could provide useful insights into remaining challenges. Lastly, details on hyperparameter tuning and model specifications would further support reproducibility.
Decision: Accept-Main