# Abacus: Self-Supervised Event Counting-Aligned Distributional Pretraining for Sequential User Modeling

Sullivan Castro, Artem Betlei, Thomas Di Martino, Nadir El Manouzi

► To cite this version:

Sullivan Castro, Artem Betlei, Thomas Di Martino, Nadir El Manouzi. Abacus: Self-Supervised Event Counting-Aligned Distributional Pretraining for Sequential User Modeling. WSDM 2026, Feb 2026, Boise, Idaho, United States. 10.1145/3773966.37793. hal-05456642

HAL Id: hal-05456642

https://hal.science/hal-05456642v1

Submitted on 13 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Abacus: Self-Supervised Event Counting-Aligned Distributional Pretraining for Sequential User Modeling

Sullivan Castro*

Artem Betlei

Criteo AI Lab

Criteo AI Lab

Paris, France

Grenoble, France

École Nationale des Ponts et Chaussées

Paris, France

a.betlei@criteo.com

sullivan.castro75@gmail.com

Thomas Di Martino

Nadir El Manouzi

Criteo AI Lab

Criteo AI Lab

Paris, France

Paris, France

t.dimartino@criteo.com

n.elmanouzi@criteo.com

## Abstract

Modeling user purchase behavior is a critical challenge in display advertising systems, necessary for real-time bidding. The difficulty arises from the sparsity of positive user events and the stochasticity of user actions, leading to severe class imbalance and irregular event timing. Predictive systems usually rely on hand-crafted "counter" features, overlooking the fine-grained temporal evolution of user intent. Meanwhile, current sequential models extract direct sequential signal, missing useful event-counting statistics. We enhance deep sequential models with self-supervised pretraining strategies for display advertising. Especially, we introduce Abacus, a novel approach of predicting the empirical frequency distribution of user events. We further propose a hybrid objective unifying Abacus with sequential learning objectives, combining stability of aggregated statistics with the sequence modeling sensitivity. Experiments on two real-world datasets show that Abacus pretraining outperforms existing methods accelerating downstream task convergence, while hybrid approach yields up to +6.1% AUC compared to the baselines.

## CCS Concepts

* Information Systems → Personalization.

## Keywords

Display Advertising, User Modeling, Self-Supervised Learning

## ACM Reference Format:

Sullivan Castro, Artem Betlei, Thomas Di Martino, and Nadir El Manouzi. 2026. Abacus: Self-Supervised Event Counting-Aligned Distributional Pre-training for Sequential User Modeling. In *Proceedings of the Nineteenth ACM International Conference on Web Search and Data Mining (WSDM '26), February 22-26, 2026, Boise, ID, USA*. ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/3773966.3779391

## 1 Introduction

Modeling online user behavior is central to display advertising (DA), where Demand-Side Platforms (DSPs) must decide within milliseconds whether and how much to bid for an impression. The setting is challenging: purchase events are extremely rare relative to clicks and impressions, and event timing is highly irregular, with long and variable gaps between user actions.

In practice, production systems rely heavily on handcrafted, count-based features - aggregated statistics over fixed time windows [7, 8, 19, 28]. These counters are robust and capture coarse patterns, but they under-represent the fine-grained temporal evolution of intent, motivating direct sequence modeling. Deep sequential models have proven their effectiveness in recommendation [13, 20, 29], yet in DA they are often inferior to systems built on counting features, providing stable aggregations [1, 6, 9, 12].

A common remedy across domains is self-supervised learning (SSL) pretraining before finetuning [2, 3, 5], which brings (i) label efficiency when positives are scarce, and (ii) transferable representations. However, standard SSL tasks (e.g. next item prediction, masked modeling, contrastive learning) do not explicitly align with the counting signals that drive performance.

We introduce ABACUS, a counting-aligned pretext task that trains a sequential encoder to predict the empirical event-type distribution within a user timeline. Abacus supplies stable supervision that reflects the information captured by production counters (without manual feature engineering) and complements token-level pretexts. We further propose a hybrid objective that mixes Abacus with masked modeling [4] and Barlow Twins [26], combining the robustness of aggregation with the sensitivity of temporal encoders.

On a large private dataset, Abacus-based pretraining improves AUC by up to +6.1% over a scratch-trained GRU encoder, with smaller but consistent gains on the public Taobao benchmark. Besides, Abacus yields a warm-start effect and faster convergence. Ablations show that mixing Abacus with complementary pretext tasks is beneficial and stable.

**Contributions.** We (i) show that self-supervised pretraining on event sequences improves user behavior modeling for DA, improving performance and accelerating convergence versus training from scratch, (ii) introduce ABACUS, a counting-aligned objective that predicts a sequence's empirical event distribution (including standard,

<figure><img src="image_2.png" /></figure>

<footnote>*The work was done during the master internship at Criteo AI Lab.</footnote>
random, and masking variants) and (iii) propose a hybrid, multi-task approach that mixes Abacus with masking and contrastive pretext tasks, improving accuracy and stability across used encoders.

## 2 Related Work

Early approaches to user modeling for DA relied on LSTM [14, 30] or hybrid [11] encoders of user activity sequences that captured temporal dependencies in click or browsing logs to make direct predictions. BERT4Rec [20], architecture for sequential recommendation, demonstrated combination of bidirectional attention with pretraining via masked modeling to be efficient to model user behavior. Later on, [27] extended the method by combining masked modeling with sequence rearrangement, explicitly integrating relative time embeddings. In Pretrained User Models (PTUM) [24], two pretext tasks for user model pretraining from large-scale unlabeled data are proposed: the model learns relatedness between historical behaviors with masked behavior prediction, while the relatedness between past and future behaviors is captured by next K behavior prediction. UserBERT [23] complemented PTUM by suggesting pretraining by behavior sequence matching, which aims to capture inherent user interests that are consistent in different periods. [15] explored benefits of SSL pretraining for event-based user behavior sequences. Their pretraining approach of contrasting the GRU-encoded representations of the possible future events from those of the less likely events with a contrastive loss has demonstrated good performance on the downstream conversion prediction task. [17] developed PinnerFormer, a Transformer-based model tailored for long user engagement sequences on Pinterest. Unlike methods focusing on next-item prediction, PinnerFormer employs a dense all-action loss that jointly predicts multiple future engagement types (e.g., repins, close-ups, link clicks) across long horizons, demonstrating strong performance in capturing evolving user interests for large-scale recommendation. [10] proposed taking the prediction of user behavior distribution over a defined period in the future as a pretext task, instead of the usual next K behaviors prediction. They provided the Multi-scale Stochastic Distribution Prediction (MSDP) algorithm, improving the generalization ability of user representation and satisfying various downstream tasks. While the idea of predicting behavior distribution is similar, MSDP aims to extract sequential patterns to predict future actions, differing from our approach of learning present event statistics. [16] adapted Barlow Twins - contrastive SSL method based on minimizing cross-correlation between pairs of differently corrupted inputs - to user sequence modeling, proven to learn informative representations without relying on labeled data or negative sampling.

# 3 Method

## 3.1 Notation

Let the user behavior sequence be $x = \{(e_i, \tau_i)\}_{i=1}^\ell$ with event type $e_i \in \{1, \dots, K\}$ and continuous normalized event timestamp $\tau_i \in [0, 1]$, $\ell$ is the sequence length. Pretrained and finetuned model parameters are defined as $\theta$ and $\phi$ respectively. Also, let event embedding be $e_\theta \in \mathbb{R}^{K \times d}$ where $d$ is event embedding dimension, sequential encoder be $f_\theta^{Enc}$ and sequence embedding be $h$. Then, pretext task-specific entities are defined: sequence augmentation, encoder projection head, output and loss are $a^t(x)$, $g_\theta^t$, $o^t$ and $\mathcal{L}^t$

respectively, where $t$ denotes pretext task. For downstream task model, prediction head is denoted as $g_\phi$.

## 3.2 Pretraining

On the pretraining stage, we first apply a task-specific augmentation $a^t$ to the raw sequence $x$, yielding $a^t(x)$. Input representation is a concatenation of the learnable event embedding ($e_\theta$)$_i \in \mathbb{R}^d$ with $\tau_i$, i.e. $a_\theta^{emb,t}(x) = \{(e_\theta)_i \parallel \tau_i\}_{(e,\tau) \in a^t(x), i \in \{1, \dots, \ell\}}$.

Shared encoder $f_\theta^{Enc}$ maps $a_\theta^{emb,t}(x)$ to the $h = f_\theta^{Enc}(a_\theta^{emb,t}(x))$ embedding summarizing a sequence. For the encoder architecture, two models are considered: RNN (GRU) and Transformer (BERT with [CLS] token at the end as a summary, as done in related works [21]). Then, the task-specific head $g_\theta^t$ maps $h$ to the prediction $o^t = g_\theta^t(h)$, forming the loss $\mathcal{L}^t$.

For the hybrid model training, multi-task learning (MTL) is applied: on a mini-batch, task-specific losses are computed on the same examples, so single backward pass is performed on the weighted sum: $\mathcal{L}^{MTL} = \sum_t w^t \mathcal{L}^t$ where $w^t \ge 0$ and $\sum_t w^t = 1$.

Chosen pretext tasks with corresponding augmentations, projection heads, and losses are described below.

### 3.2.1 Pretext tasks. Three pretext tasks are used in our approach.

(1) Distributional (Abacus). Abacus trains the encoder to summarize a sequence by its global event composition. Given $x$, let $p^{Abacus} = \frac{1}{|\sum_{i=1}^\ell one\_hot(e_i)|} \in \Delta^{K-1}$ where $\Delta^{K-1} = \{p \in [0, 1]^K : \sum_{k=1}^K p_k = 1\}$ be the empirical histogram of event types. A MLP head $g_\theta^{Abacus}(h)$ outputs logits histogram prediction $o^{Abacus} = g_\theta^{Abacus}(h)$ and $\hat{p}^{Abacus} = softmax(o^{Abacus})$. Cross-entropy loss $\mathcal{L}^{Abacus} = -\sum_{k=1}^K p_k^{Abacus} \log \hat{p}^{Abacus}$ is minimized. The choice of histogram prediction rather than absolute count is due to sequence length invariance and compatibility across users.

Three sequence augmentation options are considered:

* $a^{Abacus}(x)$ (no augmentation): learns event frequencies.

* $a^{Abacus-R}(x)$ (random permutation): enforces permutation invariance, acting as additional regularizer.

* $a^{Abacus-M}(x)$ (segment event masking): predicts histogram from the corrupted sequence where events are masked for some segments. By this, the encoder is forced to learn how local fragments inform aggregate behavior, i.e. how observed signals correlate with unobserved ones.

Overall, Abacus injects a counting-aware prior that is order/length-agnostic and robust to imbalance.

Besides proposed Abacus task, we introduce to the model two additional, existed tasks, focused on temporal pattern learning:

(2) Masked Sequence Modeling (MSM). In MSM, both events and timestamps are masked in contiguous sequence segments $M$, forming $a^{MSM}(x)$. Then, projection head $g_\theta^{MSM}$ inputs $h_M$ (masked elements of $h$ only) and outputs $o^{MSM} = (o_{(1:K)}^{MSM}, o_{(K+1)}^{MSM})$ - so $o_{(1:K)}^{MSM}$ are event reconstruction logits, and the last component is a timestamp reconstruction $\hat{t} = o_{(K+1)}^{MSM}$. Then we can derive $\hat{p}^{MSM} = softmax(o_{(1:K)}^{MSM})$. For the event and timestamp reconstruction, cross-entropy and (MSE) losses are used respectively:

$$ \mathcal{L}^{MSM} = -\frac{1}{|M|} \sum_{m \in M} \log \hat{p}_{em}^{MSM} + \lambda^{MSM} \frac{1}{|M|} \sum_{m \in M} (\tau_m - \hat{\tau}_m)^2. $$
<figure><img src="image_3.png" /><figcaption>Figure 1: Illustration of the proposed approach. In pretraining, Abacus, MSM and BT pretext tasks are jointly learned via MTL. Both event and sequence embeddings are then finetuned on downstream task.</figcaption></figure>

$\lambda^{MSM} = 1$ is used for the experiments. MSM trains the encoder to recover locally coherent event-time patterns, sharpening short-range temporal context and acting as a denoising regularizer, so the learned representations are more robust to missing or corrupted spans providing a stronger warm-start for finetuning.

(3) Barlow Twins (BT). Two identical inputs with independently masked segments, $a^{BT}(x)$ and $a^{BT'}(x)$, are forwarded to $f_{\theta}^{Enc}$ producing $h$ and $h'$, then, projection $g_{\theta}^{BT}$ outputs $o^{BT}(h, h') = (z, z')$.

Cross-correlation matrix of the normalized outputs ($\tilde{z}^i, \tilde{z}^j$) where $\tilde{z} = \frac{z - \mu}{\sigma}$ (with mean $\mu$, standard deviation $\sigma$) is $C_{ij} = \frac{\langle \tilde{z}^i, \tilde{z}^j \rangle}{\|\tilde{z}^i\| \|\tilde{z}^j\|}$. The Barlow Twins loss is $\mathcal{L}_{BT} = \sum_i (1 - C_{ii})^2 + \lambda_{BT} \sum_i \sum_{j \neq i} C_{ij}^2$, which enforces invariance ($C_{ii} \approx 1$) while reducing redundancy between dimensions ($C_{ij} \approx 0$). $\lambda_{BT} = 1$ has been used.

## 3.3 Finetuning

In the finetuning stage, both the event embeddings $e_{\theta}$ and encoder $f_{\theta}^{Enc}$ are finetuned on downstream task, while projection MLP head $g_{\phi}$ is learned from scratch, returning $\tilde{y}$, as seen in Figure 1.

# 4 Experiments

## 4.1 Datasets

We evaluate our method on two real-world, large-scale datasets of (event type, timestamp) pair sequences:

* **Taobao<fnref n="1" />**: user behavior log from the big chinese e-commerce platform, collected during 8 days and containing “click”, “add-to-cart”, “add-to-fav” and “purchase” events.

* **Private**: user behavior log from online advertising platform, collected on one partner during one month and containing 9 user event types.

<table><caption>Table 1: Dataset statistics (higher is better for both PPL and GS).</caption><thead><tr><th>Dataset</th><th>Size</th><th>Seq. Length</th><th>Label Mean</th><th>PPL ↑</th><th>GS ↑</th></tr></thead><tbody><tr><td>Taobao</td><td>1M</td><td>100</td><td>0.0131</td><td>1.57</td><td>0.20</td></tr><tr><td>Private</td><td>1.7M</td><td>50</td><td>0.1014</td><td>4.43</td><td>0.71</td></tr></tbody></table>

To qualitatively compare the datasets, we report general numbers (size, sequence length used and label mean) along with the event distribution statistics: i) *Perplexity* PPL = $2^{-\sum_i p_i \log_2 p_i}$, representing the effective number of equally likely events, and ii) *Gini-Simpson diversity* GS = $1 - \sum_i p_i^2$ which is the probability that two independent draws differ. Statistics are provided in Table 1. Private dataset exhibits greater event type balance and label density than Taobao.

As we consider purchase prediction to be the downstream task, for both datasets each sequence is split by time into i) a user history $x$, used for training, and ii) labeling time window. We assign label $y = 1$ if a purchase event occurs at least once in this window, otherwise $y = 0$. 70/20/10 and 60/20/20 time-based train/val/test split are used for Private and Taobao respectively.

## 4.2 Experimental Setup

*Baselines.* We use two types of baselines: i) *No Pretraining (No-PT)* so finetuning only, to check the performance gain due to pretraining stage, and ii) baseline pretrainers including *Next Event Prediction (NEP)*, *Next K Events Histogram Prediction (NKEHP)* [10], *Masked Sequence Modeling (MSM)*, and *Barlow Twins (BT)* [16] to compare to both Abacus and hybrid (MTL) pretraining approaches.

All experiments were done in PyTorch on 2 NVIDIA V100 (16 GB) GPUs. Dimensions of $e_{\theta}$ and $h$ embeddings are 3 and 8 respectively. Batch size is $2^{14}$. Learning rates were chosen by a small grid on validation AUC on 300 epochs with early stopping: 0.01 for pretraining and 0.01 and 0.001 for finetuning on Private and Taobao respectively. Models share the same tuning budget. AdamW optimizer was used. The code is publicly released<fnref n="2" />.

## 4.3 Results

For both datasets, we report mean ± std AUC over 3 random seeds computed on the test split, along with the relative improvement over No-PT. Best results are highlighted in bold.

*Taobao dataset (Table 2).* While GRU is stronger without pretraining, BERT benefits substantially more from self-supervised objectives. Naïve event-level tasks such as NEP or MSM degrade

<footnote n="1">¹https://tianchi.aliyun.com/dataset/649</footnote>

<footnote n="2">²https://anonymous.4open.science/r/Abacus/</footnote>
performance across both architectures; in contrast, BT offers small but consistent improvements. Abacus variants yield the most robust single-task gains. Finally, the hybrid MTL approach combining Abacus, MSM, and BT, achieves the best overall performance.

Private dataset (Table 3). Among single pretraining objectives, results are mixed: NEP significantly boosts GRU but fails for BERT, while MSM shows the opposite pattern, hurting GRU slightly but clearly improving BERT. Barlow Twins again yields modest but consistent gains for GRU while being neutral for BERT. In contrast, the Abacus family demonstrates robust improvements for both architectures, with Abacus-R being strongest. The hybrid MTL model achieves the best performance overall.

<table><caption>Table 2: Results on Taobao dataset.</caption><thead><tr><th rowspan="2">Model</th><th colspan="2">GRU</th><th colspan="2">BERT</th></tr><tr><th>AUC↑</th><th>Δ (%)↑</th><th>AUC↑</th><th>Δ (%)↑</th></tr></thead><tbody><tr><td>No-PT</td><td>0.6423 ± 0.0031</td><td>+0.00</td><td>0.6077 ± 0.0378</td><td>+0.00</td></tr><tr><td>NEP</td><td>0.5927 ± 0.0382</td><td>-7.72</td><td>0.6170 ± 0.0295</td><td>+1.53</td></tr><tr><td>NKEHP</td><td>0.6342 ± 0.0076</td><td>-1.26</td><td>0.6133 ± 0.0318</td><td>+0.92</td></tr><tr><td>MSM</td><td>0.6409 ± 0.0008</td><td>-0.21</td><td>0.6073 ± 0.0448</td><td>-0.07</td></tr><tr><td>BT</td><td>0.6423 ± 0.0006</td><td>+0.01</td><td>0.6024 ± 0.0474</td><td>-0.87</td></tr><tr><td>Abacus</td><td>0.6422 ± 0.0009</td><td>-0.01</td><td>0.6347 ± 0.0058</td><td>+4.43</td></tr><tr><td>Abacus-R</td><td>0.6437 ± 0.0017</td><td>+0.22</td><td>0.6326 ± 0.0074</td><td>+4.09</td></tr><tr><td>Abacus-M</td><td>0.6421 ± 0.0015</td><td>-0.03</td><td>0.6310 ± 0.0032</td><td>+3.83</td></tr><tr><td>Hybrid (MTL)†</td><td><strong>0.6439 ± 0.0010</strong></td><td><strong>+0.25</strong></td><td><strong>0.6411 ± 0.0017</strong></td><td><strong>+5.49</strong></td></tr></tbody></table>

† Weights: GRU 0.6/0.4 (Abacus-R/Barlow), BERT 0.5/0.25/0.25 (Abacus-R/MSM/BT).

<table><caption>Table 3: Results on private dataset.</caption><thead><tr><th rowspan="2">Model</th><th colspan="2">GRU</th><th colspan="2">BERT</th></tr><tr><th>AUC↑</th><th>Δ (%)↑</th><th>AUC↑</th><th>Δ (%)↑</th></tr></thead><tbody><tr><td>No-PT</td><td>0.7094 ± 0.0149</td><td>+0.00</td><td>0.7175 ± 0.0224</td><td>+0.00</td></tr><tr><td>NEP</td><td>0.7215 ± 0.0089</td><td>+1.71</td><td>0.7173 ± 0.0206</td><td>-0.03</td></tr><tr><td>NKEHP</td><td>0.7003 ± 0.0096</td><td>-1.28</td><td>0.7198 ± 0.0186</td><td>+0.33</td></tr><tr><td>MSM</td><td>0.7076 ± 0.0088</td><td>-0.25</td><td>0.7304 ± 0.0086</td><td>+1.80</td></tr><tr><td>BT</td><td>0.7258 ± 0.0108</td><td>+1.66</td><td>0.7164 ± 0.0141</td><td>-0.13</td></tr><tr><td>Abacus</td><td>0.7342 ± 0.0196</td><td>+3.49</td><td>0.7324 ± 0.0149</td><td>+2.08</td></tr><tr><td>Abacus-R</td><td>0.7358 ± 0.0171</td><td>+3.72</td><td>0.7378 ± 0.0058</td><td>+2.82</td></tr><tr><td>Abacus-M</td><td>0.7101 ± 0.0096</td><td>+0.09</td><td>0.7214 ± 0.0158</td><td>+0.55</td></tr><tr><td>Hybrid (MTL)†</td><td><strong>0.7525 ± 0.0036</strong></td><td><strong>+6.07</strong></td><td><strong>0.7401 ± 0.0185</strong></td><td><strong>+3.15</strong></td></tr></tbody></table>

† Weights: GRU 0.75/0.25 (Abacus-R/BT), BERT 0.7/0.3 (Abacus-R/BT).

Figure 2 shows that Abacus-pretrained models start from higher validation AUC and converge faster. Pretraining provides consistently higher validation performance, warm-start effect, and faster convergence, thus confirming existing observations [15]. The gap is largest early in training and narrows over time without collapsing, suggesting accelerated optimization and a shifted convergence point. MTL model has learnt more complex features, converging slower though reaching higher final AUC.

<figure><img src="image_4.png" /><figcaption>Figure 2: BERT on Taobao</figcaption></figure>

## 4.4 Discussion

1. *Distributional pretraining is effective.* Counting-based objectives consistently outperform reconstruction and contrastive pretraining, indicating that aligning with distributional statistics better matches the purchase prediction task.

2. *Randomization improves robustness.* Abacus-R matches or improves over Abacus on the private dataset and is comparable on the Taobao, suggesting that enforcing order-invariance regularizes the objective improving generalization.

3. *Proposed objectives are complementary and stable.* Mixing counting, masking, and contrastive tasks with MTL consistently exceeds the best single-task pretraining results being stable across the seeds and datasets/encoders reducing the standard deviation. This suggests that combining complementary inductive biases acts as an effective regularizer that enriches the learning.

4. *Analysis of the Abacus Loss behavior.* During pretraining, while decreasing, value of $L^{Abacus}$ is quite high. This is expected: transformers and GRU struggle with explicit counting[22, 25], being similar to distribution estimation. Nevertheless, approximate moment-matching suffices to uplift downstream performance, as representations capture global distributional regularities without exact counts.

5. *Objective misalignment and negative transfer.* Objectives that enforce exact sequence fidelity either via reconstruction or the ordered future prediction (NEP, NKEHP, MSM) optimize a harder goal that involves prediction. NEP and MSM are moreover weakly aligned with downstream purchase prediction and require precise token order and timestamp recovery. This amplifies the exposure bias and error compounding, and are brittle to time noise, which translates into higher variance and frequent negative transfer. In contrast, in sequence distributional (counting) pretraining targets global statistics that are better matched to ranking and yields more reliable and stable gains across models and datasets.

6. *Why GRU is better than BERT?* Observed sequences have a tiny vocabulary (order ~10 vs ~30,000 in NLP) and are relatively short (order ~100 tokens vs ~512 in NLP). In this regime, the Transformer advantages (global self-attention and long-range context) are underused, while its higher capacity can overfit and require more data/budget to stabilize. GRUs offer a stronger inductive bias for ordered, local temporal dynamics and count-like aggregation with far fewer parameters, leading to more stable training and better downstream AUC in our setting.

## 5 Conclusion

We introduced Abacus, self-supervised objective of predicting empirical user event distribution. Across two datasets of user behavior logs and two encoders (GRU, BERT), Abacus consistently improves performance on downstream task. A hybrid, MTL approach, combining Abacus with MSM and BT tasks, further strengthens performance, indicating pretraining tasks complementarity.

Future work will include scaling of count-aligned pretraining to longer event sequences and larger vocabularies, pretraining objectives focused on efficient event time encoding and application of the more flexible MTL architectures (e.g. MoSE [18]).
# 6 Ethical Considerations

*Data provenance and privacy.* This work uses one private, large-scale behavioral dataset and one public dataset (Taobao). The private dataset was accessed under a data-use agreement for research purposes. It contains de-identified interaction logs consisting only of event types and timestamps; no direct identifiers (e.g., names, emails, device IDs) or content-level attributes were available to the authors. We applied data minimization (event type, timestamp only), removed any residual quasi-identifiers where present, and restricted access to authorized researchers. All analyses were conducted in aggregate, and we report only summary statistics and model performance.

*Personal data and sensitive attributes.* We did not use protected attributes (e.g., race, ethnicity, gender, religion) nor attempt to infer them. Labels and features reflect user interactions, not personal characteristics. We caution, however, that behavioral logs can still enable indirect profiling. Any operational deployment should be preceded by a data protection impact assessment (DPIA) and appropriate legal review under applicable data protection laws and platform terms of service.

*Potential harms and intended use.* Our method can improve purchase prediction and, by extension, bidding efficiency in online advertising. Potential risks include over-personalization, unwanted targeting, or reinforcement of filter bubbles. We recommend guardrails in deployment: frequency capping, budget and exposure constraints, human oversight for campaign goals, transparent user controls (outlets where applicable), and continuous monitoring for unintended outcomes.

*Fairness and bias.* Because protected attributes were unavailable, we could not compute standard group fairness metrics. To mitigate risk, we recommend pre-deployment audits using available proxies (e.g., geography or time-of-day segments), calibration monitoring across segments, and periodic counterfactual or perturbation testing to detect disparate error rates. Future work should evaluate fairness with appropriate consented labels or privacy-preserving auditing protocols.

*Security and confidentiality.* Model artifacts trained on private data were not released. All experiments on the public dataset use only public data and anonymized code. No attempts were made to re-identify users or link records across sources.

*Reproducibility and transparency.* To support verification without exposing private data, we will release code, configuration files, and scripts to reproduce all public-dataset experiments, including data preprocessing and hyperparameters. We also report distributional diagnostics (entropy, effective vocabulary size, Gini-Simpson) to clarify when the method is likely to be beneficial.

*Environmental considerations.* Training used two NVIDIA V100 GPUs and modest epochs; we prioritized early stopping and reuse of pretrained checkpoints to limit compute. We encourage reporting energy use and preferring smaller backbones when performance is comparable.

*Scope and limitations.* Abacus aligns representations with distributional (counting) signals and is most effective when event-type

distributions are rich. The method does not guarantee exact counting and should not be used to infer sensitive attributes or make high-stakes decisions without human oversight and additional safeguards.

# References

[1] Davide Abbattista, Vito Walter Anelli, Tommaso Di Noia, Craig Macdonald, and Aleksandr Vladimirovich Petrov. 2024. Enhancing sequential music recommendation with personalized popularity awareness. In *Proceedings of the 18th ACM Conference on Recommender Systems*. 1168–1173.

[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. *Advances in neural information processing systems* 33 (2020), 1877–1901.

[3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In *International conference on machine learning*. PmLR, 1597–1607.

[4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. *arXiv preprint arXiv:1810.04805* (2018).

[5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers)*. 4171–4186.

[6] Xinyu Du, Huanhuan Yuan, Pengpeng Zhao, Jianfeng Qu, Fuzhen Zhuang, Guanfeng Liu, Yanchi Liu, and Victor S Sheng. 2023. Frequency enhanced hybrid attention network for sequential recommendation. In *Proceedings of the 46th international ACM SIGIR conference on research and development in information retrieval*. 78–88.

[7] Pinterest Engineering. 2018. Building a Real-time User Action Counting System for Ads. https://medium.com/pinterest-engineering/building-a-real-time-user-action-counting-system-for-ads-8ac54e569fe.

[8] Snap Inc. Engineering. 2022. Speed Up Feature Engineering for Recommendation Systems. https://eng.snap.com/speed-up-feature-engineering-for-recommendation-systems.

[9] Qixiang Fang, Zhihan Zhou, Francesco Barbieri, Yozen Liu, Leonardo Neves, Dong Nguyen, Daniel Oberski, Maarten Bos, and Ron Dotsch. 2024. General-Purpose User Modeling with Behavioral Logs: A Snapchat Case Study. In *Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval*. 2431–2436.

[10] Chilin Fu, Weichang Wu, Xiaolu Zhang, Jun Hu, Jing Wang, and Jun Zhou. 2023. Robust user behavioral sequence representation via multi-scale stochastic distribution prediction. In *Proceedings of the 32nd ACM International Conference on Information and Knowledge Management*. 4567–4573.

[11] Djordje Gligorijevic, Jelena Gligorijevic, and Aaron Flores. 2019. Time-aware prospective modeling of users for online display advertising. *arXiv preprint arXiv:1911.05100* (2019).

[12] Ruidong Han, Qianzhong Li, He Jiang, Rui Li, Yurou Zhao, Xiang Li, and Wei Lin. 2024. Enhancing CTR Prediction through Sequential Recommendation Pre-training: Introducing the SRP4CTR Framework. In *Proceedings of the 33rd ACM International Conference on Information and Knowledge Management*. 3777–3781.

[13] Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recommendation. In *2018 IEEE international conference on data mining (ICDM)*. IEEE, 197–206.

[14] Tobias Lang and Matthias Rettenmeier. 2017. Understanding consumer behavior with recurrent neural networks. In *Workshop on Machine Learning Methods for Recommender Systems*.

[15] Yiping Liao. 2020. On the Effectiveness of Self-supervised Pre-training for Modelling User Behavior Sequences.

[16] Yuhan Liu, Lin Ning, Neo Wu, Karan Singhal, Philip Andrew Mansfield, Devora Berlowitz, Sushant Prakash, and Bradley Green. 2025. Enhancing User Sequence Modeling through Barlow Twins-based Self-Supervised Learning. *arXiv preprint arXiv:2505.00953* (2025).

[17] Nikil Pancha, Andrew Zhai, Jure Leskovec, and Charles Rosenberg. 2022. Pinnerformer: Sequence modeling for user representation at pinterest. In *Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining*. 3702–3712.

[18] Zhen Qin, Yicheng Cheng, Zhe Zhao, Zhe Chen, Donald Metzler, and Jingzheng Qin. 2020. Multitask mixture of sequential experts for user activity streams. In *Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining*. 3083–3091.

[19] Abhishek Singh, Rohit Gupta, Gaurav Pandey, et al. 2023. Ad-Load Balancing via Off-policy Learning in a Content Marketplace. In *Proceedings of the ACM Web Conference (WWW)*. https://arxiv.org/abs/2302.07888
[20] Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019. BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In *Proceedings of the 28th ACM international conference on information and knowledge management*. 1441–1450.

[21] Dong Wang, Kavé Salamatian, Yunqing Xia, Weiwei Deng, and Qi Zhiang. 2023. BERT4CTR: An Efficient Framework to Combine Pre-trained Language Model with Non-textual Features for CTR Prediction. arXiv:2308.11527 [cs.CL] https://arxiv.org/abs/2308.11527

[22] Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018. On the Practical Computational Power of Finite Precision RNNs for Language Recognition. In *Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*. Association for Computational Linguistics, Melbourne, Australia. doi:10.18653/v1/P18-2117

[23] Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2022. Userbert: Pre-training user model with contrastive self-supervision. In *Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval*. 2087–2092.

[24] Chuhan Wu, Fangzhao Wu, Tao Qi, Jianxun Lian, Yongfeng Huang, and Xing Xie. 2020. PTUM: Pre-training user model from unlabeled user behaviors via self-supervision. arXiv preprint arXiv:2010.01494 (2020).

[25] Gilad Yehudai, Haim Kaplan, Asma Ghandeharioun, Mor Geva, and Amir Globerson. 2024. When Can Transformers Count to n? *arXiv preprint arXiv:2407.15160* (2024).

[26] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. 2021. Barlow Twins: Self-Supervised Learning via Redundancy Reduction. *arXiv preprint arXiv:2103.03230* (2021).

[27] Qihang Zhao. 2022. RESETBERT4Rec: A pre-training model integrating time and user historical behavior for sequential recommendation. In *Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval*. 1812–1816.

[28] Fan Zhou, Runze Wu, Zhiyuan Xu, et al. 2018. ATRank: An Attention-Based User Behavior Modeling Framework for Recommendation. In *Proceedings of the AAAI Conference on Artificial Intelligence*. https://arxiv.org/abs/1711.06632

[29] Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen. 2020. S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. In *Proceedings of the 29th ACM international conference on information & knowledge management*. 1893–1902.

[30] Konrad Żołna and Bartłomiej Romański. 2017. User2Vec: User Modeling Using LSTM Networks. In *Proceedings of the AAAI Conference on Artificial Intelligence*, Vol. 31.