Semi-Structured Object Sequence Encoders
Authors: Rudra Murthy
Riyaz Ahmad Bhat
Chulaka Gunasekara
Siva Sankalp Patel
Hui Wan
Tejas Indulal Dhamecha
Danish Contractor
Marina Danilevsky
Venue: EMNLP 2023 Findings (2023)
Keywords: Structured object encoders
long sequences

Abstract: In this paper we explore the task of modeling semi-structured object sequences; in particular, we focus our attention on the problem of developing a structure-aware input representation for such sequences. Examples of such data include user activity on websites, machine logs, and many others. This type of data is often represented as a sequence of sets of key-value pairs over time and can present modeling challenges due to an ever-increasing sequence length. 
We propose a two-part approach, which first considers each key independently and encodes a representation of its values over time; we then self-attend over these value-aware key representations to accomplish a downstream task. This allows us to operate on longer object sequences than existing methods. We introduce  a novel shared-attention-head architecture between the two modules and present an innovative training schedule that interleaves the training of both modules with shared weights for some attention heads. Our experiments on multiple prediction tasks using real-world data demonstrate that our approach outperforms a unified network with hierarchical encoding, as well as other methods including a {\em record-centric} representation and a {\em flattened} representation of the sequence.

Semi-Structured Object Sequence Encoders
Rudra Murthy Vl, Riyaz Bhat', Chulaka Gunasekara', Siva Sankalp Patel Hui Wan', Tejas Indulal Dhamecha"; Danish Contractor', Marina Danilevsky'
IBM Research Al
Microsoft India Development Center
muurthy @ww.iim.coom....mmmmmmmm aaw.iaaa.baa..calaaaaaaaaaa auunaassaar..svaassnnapppp pppell @ibm.com, hwan us. us.ibm.com. tdhamecha microsoft.com, danish.contraclus @ ibmm.com, mdanile @ (uw.w...unnn us..bbm.com
Abstract
attributes of the object (Figure (a)). Examples in- clude recordings of user interactions with websites, logs of machine activity, shopping decisions made by consumers, and many more (Figure ((b).. The data is usually stored in semi-structured formats such as JSONs, or tabular forms.
this paper we explore the task of modeling semi-structured object sequences; in particu- lar, we focus our attention on the problem of developing structure-aware input represen- tation for such sequences. Examples of such data include user activity on websites, machine logs, and many others. This type of data is often represented as a sequence of sets of key- value pairs over time and can present model- ing challenges due quence length thereby affecting the quality of the representation. We propose two-part ap- proach, which first considers each key inde- pendently encodes a representation its values over time; then self-attend over these value-aware key representations to accomplish a downstream task. This allows us learn bet- ter representation while being able to operate on longer object sequences than existing meth- ods. We introduce a novel head architecture between the two modules and present an innovative training schedule that interleaves the training of both modules with shared weights for some attention heads. Our experiments on multiple prediction tasks using real-world data demonstrate that approach outperforms a unified network with hierarchical encoding, well as other methods including a record-centric representation and alatteeedd representation the sequence.
In this paper, explore the task of modcling semi-structured object sequences; in particular, we focus our attention on the problem of developing a structure-awar input representation for such se- quences. we think the parallel to natural lan- guage data, we could treat each sentence of text (Figure I(c)) akin to the set of key-value pairs at a particular time step.
ever-increasing se-
The challenge of sequence length: A trivial method of representing such sequences would be to flatten each structured object and view its con- stituents as individual words for tokenization in natural language (Figure (d)). However, this causes the sequence length to become extremely large (thousands of tokens) when operating on real- word semi-structured sequences. For instance, in our study of semi-structured objects from user- interaction sessions on software from large cloud- based service provider, we found these objects could contain 60 fields on average. The values of these fields include timestamps, identifiers, log messages, etc., with average of 5 words each. A session length of 15 minutes results in 105 such session objects, amounting nearly 31, 500 words which would further increase the sequence length after sub-words are created.
1 Introduction
Semi-structured object sequences comprise sig- nificant portion of the myriad of data created daily. This data usually has temporal aspect, with the data created sequentially and representing events happening in some order. More generally, the data is sequence of structured objects, each repre- sented by set of key-value pairs that encode the
Record-centric representation: In contrast, to flattening structured objects, could also create record-centric representations (Figure 1(f)) where the set of key-value representations each time step are considered sequence. As our experi- ments in Section 3.3 demonstrate, such represen-
Work carried out when the author was an IBM employee Coode at https: //gi thub. com/ mur turthyyurrr/ BemiStructuredEnooers
with markers to indicate boundaries for each structured object

---

"ayy Koo o Kocee kited Keeue Tot wi kalte coft kiced keoug kateh katnky kat Recoff
(a]
(d)
to tx-tw:
Ky
tz muaett addd, pone on lan blow boxes nucq
tz eivent add paxt collert whitme texth time_scarp:
Kst { ot vi v/
ovent: tocuoooonoo nodd aod ber nell and
(b)
(e)
nodo__d;
node, 1d1 stennstan::
tomm, stteppt
ket (ff oy of vb
T1
tz tes of 2022, nillionn"
tus (krob) kitef. ketef keof
heee "New Delles 10 The cacital
Toxt: The zivez Yanura flows thrre gh city."
ta: (krrk) lasw,, keoed Rese
(c)
(f)
tat (kickk kaceb. kscek. kaced
Figure I: Semi-Structured Object Sequences: (a) Generic representation of sequence of semi-structured objects, consisting of multiple key-value pairs at time steps 4 ... Ix- (b) Example object: a sequence of events triggered by the use of graphical user interface. (c) Viewing a text paragraph example object in (a) by flattening.
sequence of sentences. (d) Encoding the (e) Encoding (a) by encoding representation of the values for each key (to be followed by a key aggregation step, not shown). (f) Encoding (a) using record-centric representation for each time step.
tations do not work well when operating on long sequences. We hypothesize that this could be be- cause the record-representation does not adequately model the dependencies between the constituent values of the same key across different time steps (records).
port longer sequences than what would have been impossible with flattening (due to memory con- straints). Moreover, this encoding strategy also accommodates input sequences that may be con- sidered non-structured c.g. natural language text as sequences of words in sentences (Figure ((c)). Specifically, if we consider sequence of structured objects where each structured object consists of only one key, whose values contain a sentence, then our TVM effectively encodes the text sequence us- ing whichever encoder has been employed."
Key-centric representation: To address these challenges we use a modular, two-part hierarchical encoding strategy. First, we decompose the se- quence of semi-structured objects into independent sets of sequences based on keys. This allows us consider each key separately (Figure 1(e)), and encode representation of how the values of that key evolve over time (we refer this as Temporal Value Modeling - TVM). This may be achieved using any encoder. 3 We then self-attend over the set of the key encodings to create a representation of the entire structured object sequence (referred to Key Aggregation KA).
The decoupling of keys and the use of two independent encoders TVM for value-aware key representations, and KA for aggregating key- representations for a downstream task requires that information shared between the two net- works so that the rey-representations generated by TVM can be optimized for downstream tasks via the KA. To facilitate this, we share few sets of at- tention heads between the two networks. First, we pre-train the TVM network with shared attention heads in place. We then use the frozen represen- tations from this network to initialize the KA net- work, which has its own untrained attention heads, and the shared attention heads from the TVM net- work as part of its trainable parameters. We utilize a training schedule that interleaves the training of both modules to iteratively train them. Doing so allows the TVM and KA modules to create richer
This key-ccntric perspective en- coding ceni-structured sequences has many advan- tages as compared to flattening and record-centric representations (Figure I()).
Advantages:
Decoupling the keys allows us to support an ar- bitrary number of keys* since each key- sequeeeee is encoded independently. So, the key-representatiion created during Temporal Value Modeling can sup-
"ee use BERT and Longformer our experiments "Real-world data can have hundreds of keys in cach object.
'The key aggregation step, in this case, redundant.

---

representations of keys, informed by their impor- tance for the downstream task. We find that this novel iterative two part-training results in better performance compared to a unified network with hierarchical encoding (with no attention-head shar- ing) as well as other methods, that either use flattened representation or record-centric repre- sentation of the sequence (Veličković et al., 2018; Mizrachi and Levin, 2019; de Souza Pereira Mor- eira et al., 2021; Padhi et al., 2021).
for each key l encode the value-aware key- representation Vi. by considering the value of the key it at each timestamp, as a sequence. Formally, V& is given by:
Vi [CLS] A [VAL_SEP] vz [VAL_SEP] ...0
where [VAL_SEP] and [CLS] are special tokens. Note that each value vf for a key k at time step j can itself consist of many tokens and those have not been shown for case of presentation. With any choice of a transformer- 2017) language encoder (TextEncoder TE), an embedding for Vi, termed the key-representatton (KR), can be obtained as:
Contributions: Our work addresses the challenges encoding semi-structured object sequences: (i) we propose a two-part approach that separately encodes the evolution of the values for each key, followed by aggregation over key-representations to accomplish downstream tasks; (ii) we present novel approach for sharing attention heads between the components; (iii) we compare our approach against baselines such as sequence flattening, joint encoding, and record-centric sequence represen- tations; and (iv) we present detailed experiments on several datasets and tasks to demonstrate the advantages of our approach.
(Vaswani et al.,
KR TexiEncoder
(2)
We use the output embedding representation at the first position (corresponding [CLS] ) as the key- representation. It is easy to see that this formu- lation allows us to accommodate the modeling of natural language text as in Figure (c). For il- lustration, if the T Texttnnooder is based on BERT (Devlin et al., 2019), Eq. 2 reduces to the encod- ing scheme typically employed in BERT for text paragraphs, where the [VAL_SEP] corresponds the [SEP] token.
2 Modeling
We now describe our approach for modeling key- value semi-structured object sequences.
AggeAggreg: Once we create key- representations we utilize them for an end-task. We encode the key-representations using the same model architecture as the TVM T Text Encoder but do not use positional embeddings since we encode set of position-invarian key representations. Note that the weights of the KA are randomly initialized. This network is directly optimized for an end-task.
Let a sequence of semi-structured objects be denoted as J = [J1, Ja...., JN] corre- sponding to N time steps. Further, let J, = {ki: of k2: u?,... ......,y:: v.....,ka, kx: o denote a structured object Jr, containing K key-value pairs ky: vi >,j = K..... The goal of our mod- eling is to learn representation of a sequence of structured objects J: and subsequently, learn f Embd(JJ
{1.2, .. for an end task, such as C-way classification task.
Embd( = KA ({KR(K) & EEX))
(3)
We develop a modular two-part modeling strat- egy to represent a sequence of structured objects.
Key-centric vs. Record-centric Representation: As an alternative the key-centric representation used by the TVM, one could construct a record- centric view to model the sequence (de Souza Pereira Moreira et al., 2021; Padhi et al., 2021). Instead of modeling the evolution of keys in a semi- structured object sequence using V/2 for each key, one could treat the sequence ure (f)). However, the record-centric representa- tion requires the network to compress information present in multiple keys and values of record (J+) which can create an information bottleneck for the downstream task. We compare and contrast these alternative views in Section 3 and Section 5.
1. The first module, called the Temporal Value Modeler (TVM), is used to learn com- bined representation (referred to the key- representations) for the different values that each key takes in the sequence.
series of Jj (Fig-
2. The second module, called the Key- Aggregator (KA), uses the reyrrepressntttionns corresponding each key, to create overall representation for J.
Temporal Value Modeling: Let & be key from the universe of all the keys IC in the sequence. Then,

---

Key Aggregator
interleaved training
Encoder
Layer
Temporal Value Modeling
Encoder
MUM Dieeciiv
V/
1/2
V/
Layar 12
Layer 11
My
Key Prstiostttions
#ncode independently
Figure 2: TVM-KA network architecture consisting a set of shared attention heads (weights) between the Temporal Value Modeler and the Key Aggregator. Each key is encoded independently to create corresponding key representation.
Challenges of Scalable Training: The training of the hierarchical two-part network, which first obtains the rey-represnntattions (Eq. 2) and then the structured object sequence representation (Eq. 3), could be done end-to-end where the network parameters are directly trained for the downstream task. However, end-to-end training of the hicrar- chical two-part network is often difficult due to the constraints imposed by the limited GPU memory. The GPU memory usage is affected by two fac- tors: (1) the length of the semi-structured object sequence; and (2) the number of keys in an object. The end-to-end model architecture operating over a batch of J sequences would exceed the memory of most commodity GPUs. By conservative es- timate, even for = 11 and N 512, typical 120M parameter model would exceed 40GB RAM limit with batch-size of 2. To address this, use iterative training paradigm, described below, which interleaves the training of the TVM and KA components by relying on attention heads that are shared between the two components.
help improve the performance of the KA on the downstream task. We, therefore, introduce hard- parameter sharing between the TVM and KA com- ponents. We hypothesize that by sharing few at- tention heads (weights) between the two networks, the KA will be able to utilize the shared attention heads. Specifically, as training progresses and up- dates the parameters used in these heads, it will have an effect of adjusting the key-representations from the TVM in way that could help improve overall end-task performance.
Interleaved Task Training: As mentioned above, we use an iterative task training paradigm where we interleave the training of the TVM and KA components. Note that our training paradigm is different from traditional training schedules for sequential task training where one network is fully trained before the next module, or from fine-tuning approaches where a part of the network may be initialized with a pre-trained model and additional layers of the network are initialized randomly and then updated for an end-task. We use the Masked language modeling (MLM) objective (Devlin et al., 2019) to train the TVM component and end- task-specific objective for training the KA. The of interleaved training, outlined in Algorithm I, prevents the problem of catastrophic forgetting (French, 1999; McCloskey and Cohen, 1989; Mc- Clelland et al., 1995; Kumaran et al., 2016; Rateliff,
Sharing A Attention Heads: Recall that the TVM network first creates representation for each key by attending to the values that occur in the sequence for each of them. The KA network then uses these representations to learn the end task. However, the KA network could influence how these repre- sentations are created for each key, it could perhaps

---

1990) when the KA is trained. Further, it is possi- ble that when the TVM is trained for the first time it may rely heavily on the heads that are shared. Thus, any change to the representation from these heads could lead to poorer key-reprresentations and attention sharing, and would therefore would be counter-productive. To address this problem, we apply DropHead (Zhou al., 2020) the shared attention heads in TVM and pre-train the model before beginning the interleaving schedule.
traces are logged by the cloud provider website. Event types include login, browsing, account cre- ttion/maintenancchupddaa. UI navigation, search, service creation/deletion, app interactions, and oth- ers. Each event has an associated payload that provides context around the event. Our raw data is snapshot of application event traces spanning 3 months and comprising about 450M events, from which we build user sessions. user session is essentially temporal sequence of event traces for that user. While the raw data has over 60 keys in each event, we experiment with a smaller set of manually selected 11 keys, so that existing ap- proaches and baselines can be meaningfully used for comparison. We constructed user sessions for 100k users. The application events corresponding to 1) plan upgrade, and 2) opening chatbot (to seek help) are considered as milestone events. These milestone events are chosen to represent revenue generation and user experience, respectively. The case of 3) no milestone event occurring is treated as third class. From the traces, temporal sequences of 300 events are extracted to predict if a milestone (or no-milestone) event will occur in next 50 time steps. We report Macro Fl-Score, as the dataset exhibits class imbalance.
Algorithm 1 Interleaved training
Initialize Temporal Value Modeler M., Key Aggregator
Mr parameters randomly.
2: Prepare the dataset D. consisting of value sequences
.p 3: for .p do
4: D. T'V training
S: Update T'M M,, model parameters with MLM ob-
jective on Da.
6: Prepare the dataset De representations KR; as per Eq. 2
consisting of key-
7: D KA training
8: Update Key Aggregator Mr model parameters with cross- loss for downstream task.
9: end for
3 Experiments
Our experiments are designed to answer the fol- lowing questions: (i) How helpful is the TVM-KA architecture over the baseline that involves flatten- ing semi-structured object sequences? (ii) How does the model compare to existing approaches based on record-centric representations? (iii) How important is the use of shared attention heads for fine-tuning KA? (iv) Does the interleaved training procedure help train the network effectively?
Instacart eCommerce Data: The publicly avail- able Instacart dataset contains 3 million grocery purchase orders of nearly 200, 000 users of the ap- plication. Each order consists of multiple products and each structured object associated with a prod- uct contains meta-data such as day of the week, product category, department, aisle, etc. We re- process this dataset to create sequences of product purchases and evaluate models on the task of the next product prediction. We predict the product name given the sequence of product orders," which is effectively classification task over a universe of 3212 products. Existing work on this dataset has focused on a simpler binary prediction task where models are asked to predict if a particular item is likely to be purchased again.*
3.1 Data
We experiment using two application/cloud logs datasets and one e-commerce purchase history dataset. The first logs dataset, referred to as 'Cloud Service Logs,' is an internal dataset consisting of interaction traces typically used for product us- age analysis. We also use the publicly available LogHub (He et al., 2020) dataset, comprising sys- tem log messages from the Hadoop distributed file system, and a publicly available e-commerce dataset, which consists of product purchase infor- mation (Stanley et al., 2017). We include additional details about the datasets in the uppendix.
LogHub Data: We use the HDFS-1 from LogHub (He et al., 2020) for the log anomaly de- tection task. As the dataset originally consisted of lines of log messages, we use the Drain log parser
"hups://tec.inseee instacart.comiiiioc wwwmmmmmlllonnnntttaaaaaaaaaaaaa -orde-s--- open-sourred-d-40229eeddee
Cloud Service Logs Data - Application event traces from a large cloud provider: In the Cloud Service Logs dataset, application event
"e use complete structured object.
"htps://www. https///wwwwkkgl..co www.o///ooppiiiio/osssstttto.ooe
hasket- aankkct-aaalyysss/cacctooot

---

Dataset
Train
Dev
Test
# Classes
Task
Keys
Time Steps
Median
Maximum
Cloud Service Logs
12,833
1,608
1,604
Milestone Prediction
11 112 (17061)
300 (177340)
LogHub
402,542
57,506
115,012
Anomaly Detection
46
(1176)
298 (18530)
Instacurt
780,003
97,501
97,500
3,212
Next Product Prediction
10 134 (9842)
3598 (267025)
Table I:
Dataset Statisties including the median and maximum length of sequences reported in number of time-steps. Values in parentheses report the sequence length after wordword tokenization using the BERT tokenizer.
(He et al. 2017) to identify 18 log templates. Using semi-automated approach, assign key names the value slots of the templates. Thus, each log line is converted value pairs. The original dataset splits the log lines into blocks, and the binary prediction task to pre- dict whether a particular block is anomalous. The dataset is highly imbalanced, with around 3% of the instances belonging to the anomalous class. We therefore report Fl Fl-Score for the anomalous class.
the key-value pairs of a JSON object. Pre-trained models for tokens cannot be mean- ingfully used to initialize record-centric baselines as these create sequences over sets of key-value pairs. On the other hand, identifying a custom pre- training objective for baselines is challenging as the downstream task of predicting the next event/prod- uct involves predicting a value for very specific key future time step (which is very similar objective might think of for pre-training). Hence, we randomly initialize the weights of the record-centric models.
structured object with 46 key-
3.2 Encoders
Baselines: We flatten each key-value pair in struc- tured object and encode them with special markers indicating boundaries for objects and timesteps. We fine-tune the pre-trained encoders for each downstream task and report their performance. We experiment with BERT (Devlin et al., 2019) and Longformer (Beltagy et al., 2020) as the pre-trained encoders.
Encoders for TVM-KA: of the TVM-KA architecture is that is agnostic the choice of the encoder. We employ the same en- coder architectures used in our baselines to enable a direct performance comparison. Recall that the TVM module and KA module share attention heads to facilitate sharing of information between them. To pre-train the TVM, mask 15% of the tokens in every Value Sequence, and the objective is predict the masked tokens. We do mask value- separator and key aggregator tokens; we only mask the values. Details on Hlper-parameete tuning and the iterative training schedule are in the appendix.
One of the advantages
We also compare our model with popular ap- proaches for creating record-centric representa- tions. These approaches first obtain the representa- tions for each object Jú, and then feed them to an inter-object transformer to create the representation of the whole sequence of objects. We experiment with three popular methods, where each object rep- resentation is obtained from the key-value pair rep- resentations by 1) point-wise summation, 2) con- catenation then project-down (Mizrachi and Levin, 2019; de Souza Pereira Moreira et al., 2021), and 3) self-attention then averaging (Zhang et al., 2019; Gu et al., 2021). Note that the record-centric base- line can be viewed as an adaptation of the TabBERT model originally designed for tabular data (Padhi et al., 2021). Padhi et al. (2021) use field trans- former (a transformer encoder similar to BERT) encode all rows in the table independently to obtain row embeddings. These row embeddings are later passed through another encoder to obtain a representation for the entire table. We use self- attention instead of a field transformer to encode
3.3 Results
Table reports the primary results from our experi- ments. We include the results on three datasets and for each dataset, we report the overall performance along with the performance of the models on slices of the dataset where the length of the sequence is greater than the median length for that dataset.
Comparison with Flattened encoding: As seen in the 'overall' scores for each dataset in Table 2, flattening (first four rows) yields significantly lower performance compared to our approach in- volving the use of interleaved training for TVM- KA (last row). For instance, on the cloud service logs, there's an increase of 3.9%-58.5% compared to flattened encodings in macro F1 scores. Similar

---

Cloud Service Logs (Macro F1)
Instacart (Recalló 10)
Loghub (Binary F'I-Score)
Configuration
Medeaan (50%)
L.Medinn (50%)
Overall
Overall
Overall
(51.37%)
Flattened Eacoding
49.55
50.22
9.6
9.4
0.00
53.62
(bert base uncased)
(BERT)
(Devlin 2019)
Pre-Triined
77.30
20.10
23.32
(hert- his---ese-mmmeee
Pre-Trained
80.06
76.59
21.07
19.11
58.30
75.86
(bert-lagg-unccseed)
Flattened Facoding
(Logfformer) (Belingy et al., 2020)
Pre-Triined
75.83
73.71
16:94
16.08
97.71
98.54
Record-centic
Summation
78.97
77.33
7.11
5.10
99.22
99.51
Representation
Concat
77.76
75.99
7.18
5.11
99.29
99.57
Souza Pereira Moreira al., 2021) (Gu 2021)
Self-Attention
79.18
77.73
7.98
6.34
99.08
99.42
Joint Modeling
80.15
77.68
17.04
16.0
46.78
70.72
TVM-KA
No Interleaving
73.19
73.22
18.32
17.5
99.05
98.64
Interleaving
81.26
79.60
23.44
22.54
98.79
99.32
Table 2: Comparison TVM-KA model with the baseline approaches on various datasets. our proposed approach, both TVM and KA components have the same architecture and the number of parameters pert-base-uncandd We additionally report results on subset of the test set whose sequence length (L) (onstttokenization) is greater than the median length for each dataset. The values parenthesis indicate percentage number instances where the length of sequence is greater than the median sequence length. The results from approach are statistically significant with respect to all other approaches on both cloud service logs and instacart datasets (p-value < 0.03).
trends reported the Instacart dataset. We find that the LogHub dataset, the Longformer model (Beltagy et al., 2020) is able obtain a comparable performance (98.54 vs 99.32).
each time-step before it processes sequence (See Figure 1). This combined representation can be lossy and it does not allow interaction across keys or time steps, which the flattening baseline permits (thanks to n² attention across each token). Further, not only the number of classes for prediction the Instacart data in the thousands (while it is <5 for the other datasets), but it also exhibits longer se- quences than other datasets, all of which are factors that likely contribute this result.
Comparison with Record-centric representa- tions: Unlike flattened encoding, the record- centric representation does not suffer from the mod- cling limitations associated with the maximum se- quence length limit. These representations code sequences in their entirety, since most datasets have fewer than 300 objects (time-steps), and the sequence length equal to the number of time steps. However, the record-centric view may not adequately model the dependencies between val- ues of the same key across different time steps. We find that our approach outperforms all record- centric representation baselines the Cloud Ser- vice Logs dataset as well the Instacart dataset, As before, the performance of TVM-KA on the LogHub data is similar to that of different record- centric baselines (99.32 vs 99.57) this perhaps indicates that the prediction task is relatively sim- pler on this dataset.
en-
Importance of interleaved training: As seen in the last two of Table 2, using interleaving training method outperforms training the model with no interleaving. This supports our hypothesis that sharing a few attention heads helps the TVM- KA model uncover better ree-repressntatiions for the downstream task as training progresses. We illustrate the increase in performance with cach stage of interleaving all three datasets in the appendix.
Joint Modeling vs Interleaved Training of TVM- KA: In the joint modeling approach, do not use shared attention heads and instead train the TVM and KA networks end-to-end (jointly). We observe that joint modeling performs poorly com- pared to our interlcaving approach. We found this surprising as we had expected to be at par
While the poor performance of record-centric baselines over flattened encoding on the Instacart dataset may appear counter-intuitive first glance, the reason this can happen is that the record-centric approach creates a combined representation for

---

with our approach when the joint models fit in memory. We hypothesize that by interleaving and sharing attention heads between TVM and KA, the fine-tuming of the KA on the downstream task introduces a task-specific bias to help im- prove the key-representations. This in-turn, ben- efits the pre-trained representations of the TVM via thared-attention weights and further improves performance in the next round of training. In the absence of this bias, the joint model perhaps con- verges at an alternative minima that is not as good.
information present in tables and unstructured text for information seeking tasks (Li al., 2021; Zhu et al., 2021; Zayats et al., 2021; Luetto et al., 2023; Cholakov and Kolev, 2022), etc. In addition, work on modeling time-series tabular data has focused on numerical data (Zhou et al., 2021; Zerveas et al., 2021; Zhao et al., 2022b) with purpose-bun built task- specific architectures that cannot be easily adapted to other tasks (Wu et al., 2021; Padhi et al., 2021).
Modeling Temporal Graph Sequences and Rec- ommender Systems: Our approach is also re- lated to a rich body of work on modeling temporal graphs and recommendation systems (Xu et al., 20216; Grigsby et al., 2021; de Souza Pereira Mor- cira et al., 2021). Temporal graph evolution prob- lems involve constructing representations to enable tasks such as link prediction (Sankar et al., 2020; Xu al., 2021a), item recommendation in user ses- sions (Hsu and Li, 2021), answering queries on graphs and sequences (Saxena et al., 2022), clas- sifying graph instances in a sequence, (Xu et al.. 2021a,b) etc. Our findings suggest that such ap- proaches do not scale for long sequences for the tasks we experimented with. However, our record- centric model baselines (de Souza Pereira Moreira et al., 2021; Mizrachi and Levin, 2019) are similar in approach to these methods.
Effect of varying the number of shared attention heads: In general, we observe that sharing of 1 and attention heads helps the most. Sharing too few or too many attention heads results in an aver- age drop of 2%-43% in performance. We include further details in the appendix.
Effect of parameter size/model capacity: We investigate if the model capacity could be a bot- tleneck for the approaches based on flattened rep- resentations. We fine-tune a bert-large-uncant model, which has 3z the parameters of the bert- base-uncused model and approximately 1.5x the parameters of the TVM-KA network. We find that our model performs better, suggesting that the im- proved performance is primarily due the value- aware key representations and the model's resultant ability to accommodate longer sequences due to the decoupling of keys.
4 Related Work
Parameter sharing in neural network models: Deep neural networks are usually trained to tackle different tasks in isolation. Networks that cater to multiple related tasks (multi-task neural networks) seek to improve generalization and process data ef- ficiently through parameter sharing and joint learn- ing. Traditional hard hard-parameter sharing uses the same initial layers and splits the network into task- specific branches an ad hoc point (Guo et al., 2018: Lu al., 2017). On the other hand, soft- parameter sharing shares features via a set of task- specific networks (Liu et al., 2019; Maninis et al., 2019). More recently adaptive sharing approaches have been proposed that decide what parameters to share across tasks to achieve the best perfor- mance (Vandenhende et al., 2019; Sun et al., 2020). The parameter sharing utilized in this work is dif- ferent from the aforementioned approaches, as we share some attention head weights between the two networks (as compared to shared layers), in a way that could improve overall end-task performance.
Our work is related to several areas of research.
Modeling Tabular and Timeseries Data: As we explicitly model the value sequence for each key, the data object we work with is reminiscent of tabular data where each row is time-step and each column comprises the values for a particular field. On the surface, the modeling of data may appear related, but the actual tasks and models de- veloped for tasks on tabular data cannot be applied to semi-structured object sequences. This is be- cause the work on modeling textual tabular data often involves developing specialized models fo- cused on retrieving information from cells (Zayats et al., 2021; Wang et al., 2021; lida al., 2021; Yang et al., 2022), multi-hop reasoning across in- formation in different cells across parts of the table (Chen et al., 2021; Zhao et al., 2022a), combining
We trained our joint models on 80GB 1100 GPUs.

---

5 Discussion and Conclusion
References
In this paper, we have presented a two-part encoder to model structured object sequences. The choice a key-centric representation enables us to encode larger objects as well as long sequences. Our exper- iments show that by using the two-part MV architecture we are able to inject downstream task information into the temporal value modeler net- work to generate key representations that are more relevant.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. ArXiv, abs/2004.05150.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, lana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Hao Huang, Bryan Routledge, and William Yang Wang. 2021. FinQA: A dataset of nu- merical reasoming over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 3697-3711, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
We additionally present a novel interleaving scheme to train our two-part encoder. We induce task bias into the model by sharing attention heads between the Temporal Value Modeler and Key Ag- gregator components. Our proposed approach out- performs both the bascline approaches that flatten structured object sequences and those based record-centric representations.
Radostin Cholakov and Todor Kolev. 2022. The gat- edtabtransformer. an enhanced deep learning ar- chitecture for tabular modeling. arXiv preprint
arXiv:2201.00199
Gabriel Souza Pereira Moreira, Sara Rabhi, Jeong Min Lee, Ronay Ak, and Even Oldridge. 2021. Transformersáree: Bridging the gap between nlp and sequential/seesiin-- bassd recommendation. teenth ACM Conference on Recommender Systems, pages 143-153.
Limitations
Fif-
The benefits of our approach are best highlighted in datasets that have large number keys each object when sequences are long and have chal- lenging prediction task. In such cases, the joint modeling of TVM-KA becomes too big to fit in memory and record-centric approaches suffer lot of deterioration due to lossy record-repress recordrrpressentttions. In experiments, the Instacart data exhibits some of these characteristics and we see significant im- provement performance as compared to record- centric baselines.
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLD 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Robert M. French. 1999. Catastrophic forgetting in con- nectionist networks. Trends in Cognitive Sciences. 3(4):128-135.
We note that the key-centric representation does not allow the model to support tasks such as se- quence tagging of the structured objects. Nor does it allow to modeling of graph sequences effectively, it does not objects. Thus, it may be able to learn patterns across fields at different time steps. For such tasks, a record-centric representation is perhaps more helpful. Both key-centric and record-centric repre- sentations have their strengths and weaknesses, and the choice should be made with the downstream task in mind. Further, while our experiments have been reported real-world datasets they do not represent the full spectrum of existing sequence modeling tasks.
Jake Grigsby. Zhe Wang, and Yanjun Qi. 2021. Long- range transformers for dynamic spatiotemporal fore- casting. arXiv preprint arXiv:2109. 12218.
global view of the structured
Xiaodong Gu, Kang Min Yoo, and Jung- Woo Ha. 2021. BERT: Discourse-aware response generation via learning to recover and rank utterances.
Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. 2018. Dynamic task prioriti- zation for multitask learning. In ECCV (16).
Pinjia He, Jieming Zhu, Zibin Zheng, and Michael R Lyu. 2017. Drain: online log parsing approach with fixed depth tree. In 2017 IEEE international conference on web services (ICWS), pages 33-40. IEEE.
Acknowledgements
Shilin He, Jieming Zhu, Pinjia He, and Michael R Lyu. 2020. Loghub: large collection of system log datasets towards automated log analytics. urXiv preprint urXiv:2008.06444
We would like to thank Vignesh for his work during internship at IBM and helping us with the processing LogHub data.

---

Cheng-Mao Hsu and Cheng te Li. 2021. Retagnn: Rela- tional temporal attentive graph neural networks for holistic sequential recommendation. Proceedings of the Web Conference 2021.
Inkit Padhi, Yair Schiff, Igor Meinyk, Mattia Rigotti. Youssef Mroueh, Pierre Dognin, Jerret Ross, Ravi Nair, and Erik Altman, 2021. Tabular transformers for modeling multivariate time series. In ICASSP 1021-2021 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pages 3565-3569, IEEE.
Hiroshi lida, Dung Thai, Varun Manjunatha, and Mohit Tyyer. 2021. Tabbie: Pretrained representations of tabular data. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Techmologies, pages 1446-3456.
Roger Rateliff. 1990. Connectionist models of recog- nition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285.
Dharshan Kumaran, Demis Hassabis, and James L Mc- Clelland. 2016. What learning systems do intelligent agents need? complementary learning systems the- ory updated. Trends in cognitive sciences, 20(7):512- 534.
Aravind Sankar, Yanhong Wu, Liang Gou, Wei Zhang. and Hao Yang. 2020. Dysat: Deep neural represen- tation learning on dynamic graphs via self-attention networks. In Proceedings of the I 13th International Conference Web Search and Data Mining, WSDM '20, page 519-527, New York, NY, USA. Association for Computing Machinery.
Alexander Hanbo Li, Patrick Ng. Peng Xu, Henghui Zhu, Zhiguo Wang. and Bing Xiang. 2021. Dual reader- parser on hybrid textual and tabular evidence for open domain question answering. In ACL- IJCNLP 2021.
Apporv Saxena, Adrian Kochsiek, and Rainer Gemulla. 2022. gequence-to-sqquenee knowledge graph com- pletion and question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 28 14-2828.
Shikun Liu, Edward Johns, and Andrew J Davison. 2019. End-to-end multi- task learning with attention. In Pro- ceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1-80-
Jeremy Stanley, Meg Risdal, sharathrao, and Will Cukierski. 2017, Instacart market basket analysis.
Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogerio Feris. 2017. adaptiv feature sharing in multi-task networks with applications in person attribute classification. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5334-5343.
Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. 2020. Adashare: Learning what to share for efficient deep multi-task learning. Advances in Neural Information Processing Systems, 33:8728- 8740.
Simone Luetto, Fabrizio Garuti, Enver Sangineto, Lorenzo Forni, and Rita Cucchiara. 2023. One trans- former for all time series: Representing and train- ing with dime-dependent heterogeneous tabular data. arXiv preprint 200.v.230..06375..
Simon Vandenhende, Stamatios Georgoulis, Bert De Brabandere, and Luc Van Gool. 2019. Branched multi-task networks: Deciding what layers to share. Proceedings BMVC 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Sys- tems, NIPS' 17, page 6000-6010, Red Hook, NY, USA. Curran Associates Inc.
Kokiisi Maninis, Ilija Radosavovic, and Iasonas Kokkinos. 2019. Attentive single-tasking of multiple tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 1851-
James L McClelland, Bruce L McNaughton, and Ran- dall C O' Reilly. 1995. Why there complementary learning systems in the hippocampus and neacortex: insights from the successes and failures of connec- tionist models of learning and memory. Psychologi- cal review, 102(3):119.
Petar Veličković, Guillem Cucurull. Arantxa Casanova, Adriana Romero, Pietro Lið, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations.
Fei Wang. Kexuan Sun, Muhao Chen, Jay Pujara, and Pedro A Szekely. 2021. Retrieving complex tables with multi- graph representation learning. In SIGIR.
Michael McCloskey and Neal J Cohen. 1989. Catas- trophic interference connectionist networks: The sequential lcarning problem. In Psychology of learn- ing and motivation, volume 24, pages 109-165, Elsc- vier.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2021, Autoformer: Decomposition transform- ers with Auto-Correlation for long-term series fore- casting. Advances in Neural Information Process- ing Systems.
Sarai Mizrachi and Pavel Levin. 2019. Combining con- text features in sequence-aware recommender sys- tems. In RecSys.

---

Dongkuan Xu, Junjie Liang, Wei Cheng, Hua Wei, Haifeng Chen, and Xiang Zhang. 2021a. Transformer-style relational reasoning with dynamic memory updating for temporal network modeling. Proceedings of the AAAI Conference on Artificial Intelligence, 35(5):4546-435 4554.
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat- Seng Chua. 2021. TAT-QA: A question answering benchmark on hybrid of tabular and textual con- tent in finance. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 3277-3287 Online. Association for Computational Linguistics.
Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long. 2021b. Anomaly transformer: Time series anomaly detection with association discrepancy. In International Conference on Learning Representa- tions.
A Dataset Details
Jingfeng Yang, Aditya Gupta, Shyam Upadhyay. Luheng He, Rahul Goel, and Shachi Paul. 2022. Tableformer: Robust transformer modeling for table- text encoding, In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 528 -377.
We provide additional details about the datasets used in our experiments.
A.1 Cloud Service Logs Data: Application event traces from a large cloud provider
Victoria Zayats, Kristina Toutamova, and Mari Osten- dorf. 2021. Representations for question answering from documents with tables and text. In EACI.
In the Cloud Service Logs (CSL) dataset, applica- tion event traces are logged in the cloud provider website. Each user is assigned a unique identi- fier. Event types range from login, browsing. account creation, account maintenance, account update, UI navigation, search, service creation, service deletion, app interactions, among several others. We have about 638 unique event types. Each event has an associated payload that provides context around the event. For example, if a user performed a search, the payload captures the search query and the page where the search was performed. If a user inter- acted with a service, the payload captures the ser- vice ID and action performed on the service, among other information.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. 2021. A transformer- based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 21 14422424.
Xingxing Zhang, Furu Wei, and Ming Zhou. 2019. HI- BERT: Document level pre-training of hierarchical bidirectional transformers for document summariza- tion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5059-5069, Florence, Italy. Association for Compu- tational Linguistics.
Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang. 2022a. Multihiertt: Numerical reasoning over multi hierarchical tabular and textual data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages
Our raw data is a snapshot of application event traces spanning 3 months comprising about 450M events. Using these, we build our user sessions. A user session is essentially a temporal sequence of event traces for that user. If there is difference of greater than 15 minutes between two consecutive events, we break the session. We constructed user sessions for 100k users. The application events corresponding to 1) plan upgrade, and 2) opening chatbot (to seek help) are considered as milestone events. These milestone events are chosen as they represent revenue generation and user experience, respectively. The case of no milestone event occur- ring is treated as the third class.
Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang. 2022b. MultiHiertt: Numerical reasoning over multi hieranchical tabular and textual data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6588-6688 1688-660, Dublin, Ireland. Association for Computational Linguistics.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11 11111.
From the traces, we identify user sessions con- taining any of the aforementioned milestone events. We consider the temporal sequences of events 350 time-steps before the milestone event occurs. To construct the data, we consider the sequence of events till 300 time-steps and the task is to predict
Wangchunshu Zhou, Tao Ge, Furu Wei, Ming Zhou, and Ke Xu. 2020. Scheduled drophead: A regularization method for transformer models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1971-.

---

if milestone (or no milestone) event will occur in the next 50 time steps.
Thus, each log line converted structured ob- ject. The log message after conversion JSON object would look as follows,
A.2 Instacart eCommerce Data
The publicly available Instacart dataset contains million grocery purchase orders of nearly 200, 000 users of the application. Each order consists of multiple products and each structured object asso- ciated with a product contains meta-data such as the day of the week, the product category, depart- ment, aisle, etc. We reprocess this dataset create sequences of product purchases and evaluate mod- els on the task of the next product prediction. We create variable-length training instances from each user's order history by sampling between 50 to 200 previous product purchases for a certain target product. Additionally, we only sample a training instance if the target product has been ordered at least 50 times across users.
"status": "addStoredBLock
Blockmap updated",
"port": "18.250. 6:5001e",
"block_ID": "bKK--1688996671118822
6". "size": "91178", "LineId": "11", "Date": "81109",
"Time": "203519",
"Pid": "29",
"Level": "INFO", "Component": "dfs. FFfSssssamssstee" "EventId"; "5d5de21c"
A block consists of a sequence of such structured objects. The task is to classify the given block anomalous or not.
As per our task formulation, predict the prod- uct name given the sequence of product orders which is effectively classification task over universe of 3212 products. Existing work on this dataset has focused on a simpler binary prediction task where models are asked to predict whether a particular item is likely to be purchased again." 12
B Hyperparameters and Training schedule
We perform a grid search for the learning rate and batch size for all the models in our experiments. We select the hyper-parameter configuration which gives the best validation set performance each dataset's metric. We now list the range of values considered for each hyper-paramette
A.2.1 LogHub Data
HDFS-1 from LogHub (He et al., 2020) is utilized for the log anomaly detection task. The dataset consists of log lines. sample log message is shown below:
Learning
Rate:
(le-4 3e 5, le 1e-5, -5 le 3e 5e 6).
081 109 203519 29 INFO dfs. FSNamesystem: BLOCK* NameSystem addstoredBlock: blockMap updated:
5e
Batch Size: 2, 4, 8, 16, 32
10. 10.250.15 10. 10.01:066:00000 is added to blk_ -1688999887919862966 size 91178
Shared Heads (p): 2, 4, 6,
The drophead mechanism is only activated dur- ing TVM training. with drophead probability set to 0.2.
We now convert the log message to a JSON object. We utilize Drain log parser (He et al., 2017) to ex- tract the static template, dynamic variables. and header information from log messages. We obtain around 48 templates. All the log messages fall into of the 48 templates. In semi-automated fash- ion, define keys for the templates. We populate key names with the value slots of the templates.
For all the baseline models, we train till conver- gence. For the TVM training in the first iteration, we vary learning rates and observe model conver- gence (in terms of train and dev loss) after fixed 100K steps. The best learning rate for TVM is identified from this exereise. A similar approach is used for identifying the best learning KAA training stage too, albeit for smaller number of update steps. In the first iteration, TVM train- ing is done for 1 epoch. Then we proceed with the interleaving step. We alternate between TVM
for the
"https://tech.i mwstacart.coo/3.... inailiionnnnstaca--oodesss open-sourced seee--ouured--44e d40x129ead612
" use the complete structured object.
12 "hrps://www.. https://www.kaggt.com baggaa.com/cmmpttionnssstttstsse -aaketttttttttt--krtt---
barkkt--aallsiislaaaeee

---

Clou Sewke Logs Legilib
32
-2
30
80
20
(Banyy F-5or
60
-44 -14
Abcnzon Moads
10
Intefloaiing Stage
Figure 3: Effect of sharing different numbers of atten- tion heads between TVM and KAA
Figure 4: TVM-KA model performance for each in- terleaving stage. The red, green, and blue lines, along with their respective colored y-axes, indicate the per- formance of the Loghub, Instacart, and Cloud Service Logs datasets, respectively.
training and KA training with their number of train- ing steps in 2:1 proportion. For the cloud service logs dataset, the number of TVM training steps is 50K and the number of KA training steps is around 100K.
In general, we observe that sharing of and atten- tion heads helps the most. Either not sharing any attention heads or sharing more than that results in poorer performance.
Class-wise Results on Cloud Service Logs
E Training schedule
Table 3 reports the detailed class-wise results on the Cloud Service Logs dataset. As seen in the "Macro F1' score column in the Table 3, flatten- ing (first four rows) yields a significantly lower performance compared to our approach involving the use of interleaved training for TVM-KA (last row). Specifically, the flattened encoding with ran- domly initialized BERT model suffers the most the Open Chatbot class. We believe that seman- tic understanding of the event names is crucial for identifying the sequences leading to Open Chatbot milestone event.
As shown in Table 2, using our interleaving train- ing method outperforms training the model with no interleaving. This supports our hypothesis that sharing a few attention heads helps the TVM-KA model uncover better key ree-eeepresntttiionn for the downstream task training progresses. Figure 4 illustrates the increase in performance with each stage of interleaving on all three datasets.
In general, we observe improvements from our TVM-KA approach all class labels compared to the baseline models. This indicates the perfor- mance gain from model is not due to improve- ments in single class subset of classes, but, on all the classes present in our dataset.
D Influence of shared attention heads
We use BERT encoder (Devlin et al., 2019) to model both TVM and KA components in our model. This allows us to share attention heads between the TVM and KA components. bert-base- uncased has 12 attention heads at each encoder layer. We experiment with sharing 0, 4, 6, at- tention heads between TVM and KA components. Figure presents the performance of our TVM- KA approach with different numbers of shared at- tention heads between TVM and KA components.

---

Browsing/ Upgrade Account
Comments
Open
Random
Flattened Encoding
50.22
72.38
70.14
80.50
(bert-base-uncuseed)
(BERT) (Devlin 2019)
Trained
74.77
80.31
79.39
84.12
60.79
(bert-base-uncasce)
Prained
76.59
81.57
80.72
85.13
63.93
Flattened Encoding
(Longformer) (Bellagy 2020)
Pre-Trained
73.71
79.91
79.31
84.26
57.57
Record-coss centrd-
Summation
77.33
85.66
90.59
56.50
Repreaentation
Concat
75.99
85.12
90.15
53.36
Souza Pereira Moreira 2021)
(Gn al., 2021)
Self- Attention
77.73
85.66
84.78
90.48
57.92
Joint Modeling
69.61
80.54
80.04
86.82
41.97
TVM-KA
No Interlcaving
73.22
84.14
83.03
90.32
46.31
Interleaving
79.60
86.49
85.47
91.24
62.
Table 3: Comparison of TVM-KA model with the baseline approaches Cloud Service Logs datasets. In our proposed upproach, both TVM and KA components have the same architecture and the number of parameters bert-base-uncased We additionally report class-wise results. The results from approach are statistically significant with respect to all other approaches (p-value < 0.03).


Reviews:
[{"review_id": "f1GoD5e2vX", "reviewer": "Reviewer_A9A6", "summary": "", "questions": "Did the authors explore pre-training in the joint modeling approach? It's crucial for the authors to demonstrate that the performance boost of the interleaved approach is attributable to the interleaved training itself, rather than to pre-training.\n\nDid the authors explore first pre-training the TVM module and then finetune the KA module?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We address the points raised by the reviewer below:\n\n**Weakness 1:The result of the record-centric approach on the Instacart dataset is not very convincing; it's even much lower than the flattening baseline.**\n\nWhile this may appear counter-intuitive at first glance, the reason this can happen is that the record-centric approach creates a combined representation for each time-step before it processes a sequence (See Figure 1). This combined representation can be lossy and it does not allow interaction across keys or time-steps, which the flattening baseline permits (thanks to n^2 attention across each token).  Further, not only are the number of classes for prediction in the Instacart data is in the thousands (while it is <5 for the other datasets), it also exhibits longer sequences than other datasets, all of which are factors that likely contribute to this result. \n\n**Weakness 2: Pre-training can have a large impact on model performance. Have the authors tried employing pre-training in the record-centric approach?**\n\nPre-trained models for tokens cannot be meaningfully used to initialize record-centric baselines  as they create sequences over sets of key-value pairs. \n\nOn the other hand, identifying a custom pre-training objective for this baseline is challenging as the downstream task of predicting the next event/product involves predicting a value for a very specific key at a future time step (which is very similar to an objective one might think of for pre-training). \n\n**Question 1:Did the authors explore pre-training in the joint modeling approach? It's crucial for the authors to demonstrate that the performance boost of the interleaved approach is attributable to the interleaved training itself, rather than to pre-training.**\n\nIn this case of TVM-KA the value modeler can be initialized with any existing pre-trained model, however we pre-train the model ourselves because of the use of shared-attention heads (see lines: 290-297).  \n\n**Question 2: Did the authors explore first pre-training the TVM module and then finetune the KA module?**\n\nYes that setting is referred to as \"No interleaving\" in our experiments -- See Table 2.\n\nWe hope we were able to satisfactorily address all queries that the reviewer had and request the reviewer to account for this rebuttal in their final scores.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Given that the proposed key-centric method appears as a natural choice for encoding record sequences, contrasting it with other approaches would offer valuable insights to the community.\nThe authors further investigates interleaving training which improves the performance of the key-centric approach.", "reasons_to_reject": "The evaluation lacks comprehensiveness; it covers only three datasets.\n\nThe result of the record-centric approach on the Instacart dataset is not very convincing; it's even much lower than the flattening baseline.\n\nPre-training can have a large impact on model performance. Have the authors tried employing pre-training in the record-centric approach?\n\nThe proposed interleaved training, being more complex than other methods, might necessitate finer parameter tuning for optimal performance.", "questions_to_authors": "Did the authors explore pre-training in the joint modeling approach? It's crucial for the authors to demonstrate that the performance boost of the interleaved approach is attributable to the interleaved training itself, rather than to pre-training.\n\nDid the authors explore first pre-training the TVM module and then finetune the KA module?", "paper_topic_and_main_contributions": "This paper investigates a key-centric approach to encoding semi-structured object sequence. The authors compared the proposed approach with the flattening approach and record-centric approach on three datasets, demonstrating the superior performance of the key-centric approach.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5Tx2NR63gl", "reviewer": "Reviewer_sSHy", "summary": "", "questions": "Please refer to the previous section", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the the reviewer for their feedback! We address all points raised by the reviewer below.\n\n**Difference between Joint Modeling and Interleaved Training.**\n\n\nIn case of joint modeling we do not use shared attention heads and instead train the TVM and KA networks end-to-end (jointly). In contrast, interleaved training process first pre-trains the TVM with shared attention heads and then the KA network where TVM parameters (except shared attention heads are frozen), followed by training the TVM again where KA parameters are frozen (except shared attention heads). We refer to this process as \"interleaving\". Please refer to the sections between lines 2292-86 in the main paper for more details, including the motivation of the approach. We thank the reviewer for this question and we will further clarify this in the revised version of the paper. \n\n**The performance is actually not that promising. In three datasets, the proposed approach only shows consistent improvement in two of them by a small margin. For the remaining one, the Record-centric approach actually works the best. What would be the kind of tabular data that would benefit from the proposed approach? Is there any proper reason why it's not working on Loghub dataset?**\n\n\nThere are three main take-aways from the results in Table 2.\n\n\n(i) When using flattened-encoding, using models with support for longer context lengths can help improve performance (eg: BERT vs Longformer)\n\n\n(ii) Introducing structure-ware representations (such as Record-centric representations with self-attention) further improve performance  (existing approaches -- our baselines).\n\n\n(iii) Key-centric representations based on the TVM-KA (our contribution) not only allow models to include significant larger input sequences, they also perform better than both flattened and record-centric representations. Note that, TVM-KA models are trained under three settings  -  \n\n- (a) Joint Modeling \n- (b) Interleaved training (our contribution) \n- (c) No interleaving - i.e, one single step of TVM training followed by KA training.  \n\nWe respectfully highlight that *TVM-KA with interleaved training indeed performs well on all datasets including LogHub*. In the case of joint modeling, the results are lower than key-centric baselines. We hypothesize that this because on LogHub there isn't a lot of cross-key interaction across time-steps i.e, the presence of a value for a key in a sequence at one time-step correlates highly with an anomaly. However, in the joint-model training setup the network is unable to pick up on correlations due to a weak signal coming from the aggregated key-representations via the KA layer. In contrast, in case of inter-leaved training inter-leaving and sharing attention heads between TVM \nand KA, the fine-tuning of the KA on the downstream task introduces a task-specific bias to help improve the key-representations. This in-turn, benefits the pre-trained representations of the TVM via shared-attention weights and further improves performance in the next round of training. In the absence of this bias, the joint model perhaps converges at an alternative minima that is not as good. We discuss this aspect in our main paper between lines 462-492.\n\n**What kind of data would it help on?**\n\nThe benefits of our approach are best highlighted in datasets that have a large number of keys in each object, when sequences are long and have a challenging prediction task . In such cases, joint modeling of TVM-KA become too big to fit in memory and record-centric approaches suffer a lot of deterioration due to lossy record-representations.  In our experiments, the Instacart data exhibits some of these characteristics and we see a significant improvement in performance as compared to record-centric baselines. \n\nWe would have liked to experiment with more challenging real-world datasets but as the reviewer will appreciate, these are hard to source and we have tried to adapt existing public datasets to demonstrate our work. \n\n**Additional baseline**\n\nThis baseline is effectively the TVM-KA approach with no interleaved training. Instead of a simple-sum we run self-attention which allows the model to construct richer representations and is likely a stronger baseline than the one suggested by the reviewer.\n\nWe hope we were able to satisfactorily address all queries that the reviewer had and we request the reviewer to account for this rebuttal in their final scores.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Compared to flatten or record-centered approaches, separating each key and encoding each sequence of the key separately before aggregating them together makes the collection of information more efficient and capture the correlation in a better way. \n2. The paper is well structured and the figure/algorithm is clear. We can see improvement in 2/3 datasets. The ablation study is well-designed and persuasive. \n3. The idea of using the shared attention head is efficient and makes sure that TVM and KA models sync with each other and share the attention learned in the sequential training. \n", "reasons_to_reject": "1. I would suggest more analysis discussion, such as the difference between Joint Modeling and Interleaved Training.\n2. The performance is actually not that promising. In three datasets, the proposed approach only shows consistent improvement in two of them by a small margin. For the remaining one, the Record-centric approach actually works the best. What would be the kind of tabular data that would benefit from the proposed approach? Is there any proper reason why it's not working on Loghub dataset? \n3. A baseline on key-centric representation would be helpful, such as simply passing it into a transformer encoder separately and aggregate them (mean/sum) for the classification. ", "questions_to_authors": "Please refer to the previous section", "paper_topic_and_main_contributions": "Authors treat semi-structured object sequences as key-centric representations by computing a representation for each set of keys separately along the time axis and then aggregating them using self-attention for an overall representation. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AyQoNNMbIx", "reviewer": "Reviewer_ZUvN", "summary": "", "questions": "Please see previous sections (reasons to reject). ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and for rating our paper strong and well motivated. We include a response to the comments made by the reviewer below:\n\n**The main baseline considered is the flattened encoding (BERT) which handles 2/3 datasets comparatively well obtaining in 2-5 points of TVM-KA. Only on LogHub, it performs significantly worse which is due to sequence length limitation, but another flattened encoder (Longformer) is competitive. As such, the central thesis of using key-centric and record-centric over flattened encoding is not fully supported (though competitive performance is obtained by different models).**\n\nAs the reviewer correctly highlights, as compared to BERT we have gains varying from 4% to 23% depending on the dataset but we likely to respectfully highlight that they are statistically significant and the performance is consistent across datasets. Further, the performance in datasets such as Instacart has an F1 of ~20-23 where gains of 2-4 points are important due the challenging nature of the task. Unfortunately datasets are difficult to source for this challenging problem and we hope the reviewer can appreciate we have tried to adapt existing publicly available datasets to demonstrate our work. \n\n**The authors should also show results on text-only benchmarks (like GLUE used in BERT paper), to see if TVM-KA degrades the performance on learning pure text representation (represented like in Fig 1 c). This would suggest if the TVM-KA is flexible enough to tackle very diverse forms of data.**\n\nAs discussed in the main paper, for text only representations, the TVM network is effectively the base model (and can be any existing pre-trained model). In such a setup the use of the KA network is unnecessary - as an example, if the TVM model is based on BERT and lets say our NLP task is classification, the KA network will only need to learn to copy the CLS embedding from TVM.  \n\n**A key-centric baseline should be explored. Very naively, one could use same models as for record centric representation but with k and t changed (in Fig 1f).**\n\nWe apologize that we are unsure we understand what the reviewer is suggesting.  Could you please clarify? Our flattened baseline, uses a series of key:value pairs in sequence as shown in Figure 1(d), our record-centric baseline aggregates all key-values within an object before encoding it as a sequence and key-centric approach models values of each key across time-step before aggregating them for use in an end-task.\n\n**It is not intuitive as to why interleaving works better than simple joint training, and the hypothesis in L477-486 is not convincing to me. Is it possible that it is a case of simply weighting the losses in the joint setup? Some visualizations of the outputs and failure cases would be useful (in the supplementary)**\n\nWe agree that analyzing the effect of shared attentions is difficult but we have tried to empirically demonstrate that in our work with rigour. Our hypothesis is perhaps best understood by keeping in mind the performance of no-interleaved training (when shared attention-heads are not used - Table 2) as well as how the performance of the model improves with each stage of interleaved training (Please refer to Figure 4 in the appendix). \n\nWe thank the reviewer for the suggestion about include visualizations -- we will include them in the supplementary text. \n\n**Reproducibility**\n\nWe release all code and we have experimented with two publicly available datasets. We will be releasing our data processing scripts as well\n\nWe hope we were able to satisfactorily address all queries that the reviewer had and we request the reviewer to account for this rebuttal in their final scores.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Pros:\n\n1. The topic is quite interesting given the abundance of semi-structured representation (esp in tabular data) I can see lot of use cases for end-user. Simple flattening of the data is indeed sub-optimal. As such, the paper is well motivated.\n\n2. The proposed method makes sense, and performing both key-aggregation and temporal modeling in the training is helpful. \n", "reasons_to_reject": "Cons:\n\n1. The main baseline considered is the flattened encoding (BERT) which handles 2/3 datasets comparatively well obtaining in 2-5 points of TVM-KA. Only on LogHub, it performs significantly worse which is due to sequence length limitation, but another flattened encoder (Longformer) is competitive. As such, the central thesis of using key-centric and record-centric over flattened encoding is not fully supported (though competitive performance is obtained by different models).\n\n2. The authors should also show results on text-only benchmarks (like GLUE used in BERT paper), to see if TVM-KA degrades the performance on learning pure text representation (represented like in Fig 1 c). This would suggest if the TVM-KA is flexible enough to tackle very diverse forms of data. \n\n3. A key-centric baseline should be explored. Very naively, one could use same models as for record centric representation but with k and t changed (in Fig 1f). \n\n4. It is not intuitive as to why interleaving works better than simple joint training, and the hypothesis in L477-486 is not convincing to me. Is it possible that it is a case of simply weighting the losses in the joint setup? \n\n5. (Minor) Some visualizations of the outputs and failure cases would be useful (in the supplementary)", "questions_to_authors": "Please see previous sections (reasons to reject). ", "paper_topic_and_main_contributions": "Brief Summary: The paper tackles the task of how to learn over semi-structured sequences. While transformers have been used for text-sequences, here, the task is to learn series of key-value pairs which is a common data structure with many real-world use cases. The authors point out a trivial flattening of the data structure is sub-optimal due to very large sequence length. Instead, the authors explore key-centric and record centric representation. To this end, the authors propose TVM (temporal modeler) for the time-series, and KA (key aggregator) for key-centric representation, and further suggest using shared heads and training with both key-centric and record-centric representations (interleave training). \n\nExperiments are performed on three datasets, Cloud Service Logs (internal), LogHub, and Instacart, the latter of the two being public. The authors find their proposed method (TVM-KA) outperforms competitive baselines on each dataset. ", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper looks at the task of learning semi-structured sequences. They frame this as a task of learning  a key-value data structure.  They propose key-centric and record centric representations. They propose a temporal modeler (TVM) for the time-series, and KA (key aggregator) for key-centric representation, and further propose interleaved training to learn both representations. They evaluate and show good performance on 3 datasets.

The reviewers agree that the approach is exciting, the paper is well motivated, and ablations are persuasive. The reviewers also raise some clarifications on the evaluation and training. In particular,

&nbsp; (1) the evaluation with the flattened encoders is still competitive and hence questions whether the premise of the key-centric and record-centric representations is even necessary. The authors have addressed this in their response.

&nbsp; (2) there were also some clarifications requested regarding the interleaved training and the joint training.

The authors have responded to these questions. Perhaps there is still scope to clarify the writing, especially with regard to other possible baselines suggested by reviewers and how those might compare to the baselines in the paper. Incorporating the feedback from the reviewers should help improve the quality of the paper.
Decision: Accept-Findings