MTSTRec: Multimodal Time-Aligned Shared Token Recommender
Authors: Ming-Yi Hong
Yen-Jung Hsu
Miao-Chen Chiang
Che Lin
Venue: ICML 2025 poster (2025)
Keywords: Multimodal Sequential Recommendation
Time-aligned Shared Token
Image Style Representation
Large Language Model
Primary area: deep_learning->sequential_models_time_series
TL;DR: We propose MTSTRec, a transformer-based model that temporally aligns multimodal information in sequential recommendation, achieving state-of-the-art performance without relying on private user data.

Abstract: Sequential recommendation in e-commerce utilizes users' anonymous browsing histories to personalize product suggestions without relying on private information. Existing item ID-based methods and multimodal models often overlook the temporal alignment of modalities like textual descriptions, visual content, and prices in user browsing sequences. To address this limitation, this paper proposes the Multimodal Time-aligned Shared Token Recommender (MTSTRec), a transformer-based framework with a single time-aligned shared token per product for efficient cross-modality fusion. MTSTRec preserves the distinct contributions of each modality while aligning them temporally to better capture user preferences. Extensive experiments demonstrate that MTSTRec achieves state-of-the-art performance across multiple sequential recommendation benchmarks, significantly improving upon existing multimodal fusion. Our code is available at https://github.com/idssplab/MTSTRec.

MTSTRec:
Multimodal Time-Alignel Shared Token Recommender
Ming-Yi Yi Hong" 1
Yen-Jung Hsu *2 Miao-Chee Chiang' Che Lin 123
Abstract
approaches largely focus on single-modal modal data, overlooking the potential of rich multimodal information to reveal deeper user preferences and improve recommendation quality.
Sequential recommendation in e-commerce uti- lizes users' anonymous browsing histories to personalize product suggestions without relying on private information. Existing item ID-based methods and multimodal models often overlook the temporal alignment of modalities like tex- tual descriptions, visual content, and prices in user browsing sequences. To address this limita- tion, this paper proposes the Multimodal Time- aligned Shared Token Recommender (MTSTRec), transformer-base framework with single time-aligned shared token per product for effi- cient cross moddlityy fusion. MTSTRec preserves the distinct contributions of each modality while aligning them temporally to better capture user preferences. Extensive experiments demonstrate that MTSTRec achieves state-of-the-art perfor- mance across multiple sequential recommenda- tion benchmarks, significantly improving upon existing multimodal fusion. Our code is avail- able at https: prrithub.commddssp...
Multimodal recommendation systems combine diverse data types, such as images and texts. Image-based methods use pre-trained convolutional neural networks (O' Shea, 2015) like ResNet (He et al., 2016) to capture the visual features, while text-based methods use models like BERT (Devlin et al., 2019) for product descriptions and reviews. CLIP aligns text and images through multimodal approach, en- abling zero-shot transfer for various vision-language tasks (Radford et al., 2021). Recently, large language models (LLMs) have excelled in extracting hidden textual informa- tion, enriching item representations (Geng et al., 2022; Lyu et al., 2024). Rather than full-scale LLM training, we focus on leveraging LLMs to enrich item descriptions by extract- ing hidden textual features. While many recommendation models rely on image classification recognition to predict purchase intent, we argue that textual data alone is often sufficient for identifying what product is. Instead, images should capture product aesthetics and style, which are espe- cially important on platforms that sell the same product in various patterns or designs (Ugurlu, 2023).
MISTRec.
When it comes multimodal fusion, designing uni- fied model that effectively integrates different modalities presents significant challenges. Different modalities and their specialized formats leamn at different speeds and pat- terns. This makes it challenging to combine them effectively into a single system. In the context of recommendation systems, the common modalities are typically images and text, which have fundamentally different input represen- tations. Early fusion techniques are often used to unify these features, where modalities are combined before en- tering the recommendation model (He & McAuley, 2016; Liu et al., 2019). However, this approach fails to account for the significant differences in input representations and neural network architectures across modalities. In contrast, late fusion processes each modality independently before combining outputs later, enabling modality-specifft extrac- tion but ignoring their complementarity in representing the same product at each time step (Liang et al., 2023). This delays cross interaction, forcing post hoc inference and leading to suboptimal representations.
1. Introduction
In e-commerce and online platforms, Sequential Recommen- dation Systems (SRS) play a pivotal role in delivering per- sonalized product suggestions by analyzing users' browsing histories. Throughout its gradual evolution, SRS has tran- sitioned from traditional probabilistic models to advanced neural networks, enabling more effective modeling of user behavior sequences. Recent Transformer- based methods, such as SASRec (Kang& & McAuley, 2018) and BERT4Rec (Sun et al., 2019), have further enhanced sequence model- ing by capturing dependencies across the entire interaction history. However, despite these advancements, existing SRS
Equul contribution 'Data Science Degree Program, National Taiwan University and Academia Sinica. Taiwan GGrduutee In- stitute of Communication Engineering, National Taiwan Univer- sity, Taiwan 'Deparmeett of Electrical Engineering, National Tai- wan University, Taiwan. Correspondence to: Che Lin <che- lin @ ntu.cdu.tw>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To address both the modality processing and fusion issues, we propose Multimodal Time-aligned Shared Token Rec-

---

MTSTRee: Multimodal Time-Aligned Shared Token Recommender
ommender (MTSTRecc) a novel framework for multimodal feature integration and cross-modal interaction using time- aligned shared tokens. MTSTRec consists of two main com- ponents: the Feature Extractor and the Multimodal Trans- former. The feature extractor takes the browsing history sequence, where each item includes a product ID, image, text, and price. We use different extractors for each modal- ity. For text, we enrich the data through LLMs with task- specific prompts to extract implicit consumer preferences. For images, we focus style rather than classification, using Gram metrics to capture visual patterns that influ- ence purchasing behavior. These inputs are then projected into separate feature embeddings for each modality. In the proposed multimodal transformer, a self-attention encoder first applied to model the information for each feature. During fusion, we adopt a mid-fusion strategy, where modal- ities are processed independently and then combined in the intermediate stage. Our proposed Time-aligned Shared To- ken fusion module (TST) learns cross-modal interactions by aligning features from different modalities at each time step of the product interaction sequence. This ensures effi- cient feature sharing across modalities while maintaining the chronological order and consistency of product interactions.
recommendations. The traditional Markov Chain model (Shani et al., 2005) employed simple probabilistic methods but struggled to capture complex user behavior patterns, GRU4Rec (Hidasi, 2015) introduced RNNs to improve sequence modeling by capturing temporal dependencies. Transformer-based models like SASRec (Kang & McAuley, 201 8) enhance this approach by using self-attention mech- anisms (Vaswani, 2017) and causal masking, preserving temporal order and efficiently capturing short and long-term dependencies. BERT4Rec (Sun et al., 2019) extends this further by adopting bidirectional Transformer model and using CLOZE task for training. Unlike SASRec's 's causal masking, BERT4Rec allows the model to attend to both past and future items in the sequence, capturing richer contextual information. Transformers4Ree (Moreira al., 2021) and GFormer (Li et al., 2023a) have been proposed extend Transformer-basee recommendations. Transformers4Rec adapts NLP Transformers to sequential recommendation tasks, whereas GFormer enhances user-item modeling by integrating generative self-supervised learning with a graph transformer architecture. However, these models remain focused primarily on single-modal data, overlooking the potential benefits of multimodal information.
Our contributions can be summarized as follows:
2.2. Multimodal Recommendation Systems
We propose a unified multimodal recommendation framework for multimodal recommendations that seamlessly integrates diverse information, such as ID, text, image, and other modalities, enhancing the sys- tem's 's adaptability across different tasks.
Multimodal recommendation systems are essential for in- tegrating information from multiple modalities to create more accurate and comprehensive predictions. These sys- tems generally operate through two main stages: raw feature extraction and feature fusion (Liu et al., 2024).
We introduce a novel TST module that leverages shared tokens to learn mooss-modal interactions at each time step of the sequence, ensuring consiseett alignment and fusion of modality information. By maintaining the chronological structure of the sequen- tial data, the TST module effectively captures the evoly- ing relationships between user interactions and product features.
Different modalities require tailored methods to capture their unique attributes in the raw feature extraction stage. For instance, image-based features are often extracted using CNNs (O' 2015) or, more recently, Vision Transform- ers (ViT) (Dosovitskiy et al., 2021), which excel at process- ing visual data. Textual features often rely on pre-trained language models (Devlin et al., 2019), capturing semantic meanings from product textual information. In addition, advances in LLMs (Achiam et al., 2023; Dubey et al., 2024) have significantly enhanced text-based feature extraction. L.M--Rec (Lyu et al., 2024) utilizes LLM and prompts to generate richer contextual representations, enhancing rec- ommendation quality. Other LLM-based approaches have also emerged, further refining text comprehension to opti- mize the use of textual features in recommendation systems (Zhao et al., 2023; Li et al., 2023b; Wu et al., 2024).
MTSTRee outperforms state-of-the-art methods on three real-world e-commerce datasets, setting a new benchmark in multimodal recommendation systems. Our ablation studies highlight the contribution of in- dividual features and reveal deeper insights into the distinct roles of various modalities across diverse e- commerce environments, offering a valuable under- standing of future advancements in the field.
After raw feature extraction, the system proceeds to the feature fusion stage, where the multimodal data are com- bined and processed. It can be categorized into three main approaches (Zhou et al., 2023): (i) Early fusion involves merging the features from different modalities at the initial stages of the model. For example, VBPR (He & McAuley,
2. Related Work
2.1. Sequential Recommendation Systems
SRS aims to predict the next item a user will interact with based on their browsing history, providing personalized
2

---

MTSTRee: Multimodal Time-Aligned Shared Token Recommender
2016) integrates visual features into matrix factorization approach. However, early fusion may miss important tempo- ral relationships and modality-speciffe behaviors by merging features too early. (ii) Mid-fusion delays modality combina- tion for more refined processing, as seen in MM-Rec (Wu et al., 2021), which uses a cross-modal attention mechanism to combine textual and visual information effectively. How- ever, these methods often fail to capture temporal relation- ships, which are crucial in sequential recommendations, (iii) Late fusion keeps modalities separate until the final stage, as in MMMLP (Liang et al., 2023), the final outputs from three different modalities are concatenated before making the prediction. While preserving modality-spedde -specitt---peiiicc features, fails to capture early interactions between modalities and sequences, leading to suboptimal performance with complex sequential patterns. Similarly, FREEDOM (Zhou & Shen, 2023) freezes item-item structures and denoises user-item interactions before fusion, while Mirror Gradient (Zhong et al., 2024) improves robustness via flat local minima ex- ploration, both maintaining separate modality processing until the final stage.
preserving the chronological order of product interactions.
3. Proposed Method: MTSTRec
This section presents the MTSTRec framework, designed to enhance multimodal sequential recommendation by effec- tively fusing diverse product modalities from a consumer- centric perspective. The framework comprises two primary modules: the feature extractor module, which processes browsing history to extract modalitt-speccfic embeddings, and the multimodal transformer with time-aligned shared token (TST) fusion, which synchronizes and integrates fea- tures across modalities and layers. An overview of the MTSTRec is illustrated in Figure 1.
3.1. Preliminaries
Let the set of items be defined as I = fii, i2,..., ik..... in where each item is E I consists of four essential elements: product ID by, image 0k., text lg, and price Ck. Therefore, each item én can be expressed as a tuple is = (bg, ok tg, ck), capturing the multiple modalities that describe it. For each user, their browsing history is denoted as S = [81, ... s,l, where each 8; E I represents an item the user has engaged with, and n indicates the session length. The goal of SRS is to predict the next item 8n+11 that user will engage with based on their interaction history S.
Recent works also emphasize robustness and information ef- ficiency during fusion. CGI (Wei al., 2022) enhances rec- ommendation systems through contrastive graph learning, combining adaptive structure pruning with the information bottleneck to suppress irrelevant signals. DVIB (Zhao et al., 2025) improves multimodal recommendation by applying hidden-layer perturbations and self-distillation, inducing an information bottleneck effect. These approaches concep- mally align with our mid-fusion strategy, which aims to preserve relevant information during fusion.
3.2. Feature Extractors
3.2.1. ID EXTRACTOR
The ID extractor converts raw ID data into meaningful embeddings, We construct an item ID embedding matrix Mid E Rdxi1), where d denotes the dimension of the ID embedding. The input ID) embedding matrix for each ses- sion E E Rdxx is then retrieved such that E M corresponding to the item s; in the user's 's interaction se- quence, the i-th row in the matrix. For any padding items in the sequence, a constant zero vector 0 is used as their em- bedding, typically added when sequences are shorter than the maximum length. This mechanism allows the model to capture item relationships and improve recommendation accuracy by understanding item identities within sequences.
2.3. Temporal Dynamics in Recommendation Systems
Recent research highlights the significance of temporal dy- namics in recommender systems. For instance, time-aware recommendation systems (TARS) have been emphasized (Campos et al., 2014). Temporal and feature dynamics were integrated to address data sparsity (Zhang et al., 2021), while multi-scale temporal effects were leveraged for micro- video recommendations (Jiang et al., 2020). CDTR (Wang et al., 2024) introduced a causality-based framework that addresses item- and time-level biases in user behavior to enhance the accuracy of time-aware recommender models.
The processing method for the price extractor is similar; refer to Appendix A for more details.
While these approaches explicitly model timestamps or tem- poral dependencies, our work focuses on a different as- pect of time representation. Instead of relying on absolute timestamps, we consider the relative positional structure of different modalities within sequence. This allows for structured alignment of multimodal data without requir- ing explicit time information. Our method does not predict future interactions based on timestamp trends but ensures that multimodal features at each step are temporally aligned,
3.2.2. STYLE EXTRACTOR
We are inspired by the Neural Style Transfer algorithm (Gatys, 2015: Gatys et al., 2016) to extract style features. Instead of transferring style, we focus solely on extracting it. Using the VGG-19 (Simonyan & Zisserman, 2015) model, we pass the image through the network to obtain feature maps, from which we compute Gram matrices G. These
3

---

MTSTRee: Multimodal Time-Aligned Shared Token Recommender
Medale
Time alinnd Shand Takan nuston Madule (Staning across dfferent model
Foutput
Concatenation
LLM
Share Frozen LUH
ziple
Multimodal Transformer
TST Fuston Module c+.
Style Encoder
TST Fuston
Text Encoder
TST Fuston
Prompt Encoder
TST Fusion
Price Encoder
TST Ruston Medule
Encoder
+
Sall Aell-Anteniion
Sall-Attention Encoder
Salf- Attentin Encoder Uist
Salf-Altantion Encoder
Salr Attentin Encoder
Encoder
n
@
@
Text Extractor
Prompt Test Extractor Gating
Feature Extractor
ID Extractor
Style Extractor
LLM
Price Extractor
Prompt (Past Fi
Figure MTSTRec consists feature extractors and multimodal transformers. The feature extractors take inputs as a browsing history sequence S, where each item consists of product ID, image, text, and price, transforming them into the corresponding feature embeddings, along with CLOZE embedding z for each modality. The multimodal transformers begin by applying a self-altention encoder to independently process the information for each feature. When fusion begins, we utilize the TST module across different modalities each time step and across multiple layers, ensuring both modality-specific and cross-modal information are captured effectively.
matrices capture image texture and color patterns and are in- variant to spatial transformations, ensuring similar matrices for images with similar styles. We then generate the input style embedding matrix for each session, Esyle E Rdxa. Further details are in the Appendix B.
to improve recommendation performance. Inspired by LLM- Ree (Lyu et al., 2024), we employ two prompt strategies: basic prompt and recommendation prompt. Additionally, utilize five variations tailored to the task, including three for basic tasks, along with two designed for recommenda- tion tasks (see Appendix D for more details). The LLM input is divided into system prompt and user prompt. The system prompt provides a brief description of the e- commerce context Pinoo to ensure that the LLM understands the task's background, and a task-specific prompt pi, se- lected from the five predefined prompt variations described earlier. For each product its textual information, such as the title and description, is fed into the user prompt. The LLM then generates a response P for each product i, using each prompt variant j. These responses are converted into prompt embeddings for further processing (see Appendix for more details). The process formalized follows:
3.2.3. TEXT EXTRACTOR
The text extractor focuses on extracting features from prod- uet titles and descriptions. We use Llama 3.1 (Dubey et al., 2024) as our backhone for generating text embeddings that capture the key attributes of the product (see Appendix C for details). Each product is assigned text embedding. forming the item text embedding matrix M Rd [7] which is generated by feeding the textual information of the product into an LLM text encoder to obtain the initial text embeddings, followed by one-layer projection to ad- just the dimensionality. The input text embedding matrix for each session Etext E dddd then retrieved, such that Ejext Miex corresponding the item 8; in the user's interaction sequence.
p Encoder (P!))
(1)
To further enhance the model's capacity to integrate and utilize the generated prompt embeddings, we introduce gating mechanism. Inspired by (Yu et al., 2024), the gating network G is designed to manage the varying importance of each prompt variant Pi This network controls the infor- mation flow to the final prediction layer. The weights of the
3.2.4. PROMPT-TEXT EXTRACTOR
In the second part of text processing, we explore how LLMs can be prompted to generate additional textual information

---

MTSTRee: Multimodal Time-Aligned Shared Token Recommender
prompt-text embeddingg w are calculated follows:
= G(Il,P') == foftma W ['| b), Vj E .1,2,...,55 5).
Style Encoder
The item prompt embedding matrix M E pdxll is then computed by:
Encoder
The TST module is shared across different modalities manner that aligns each item in the sequence. For example, in the fusion of style and text embeddings, each TST embedding =1h updated element fement-wise by averaging the corresponding tokens from the style and text modalities at the same time step in the sequence, ensuring cross-modal information exchange that aligns in time.
Figure 2.
M Lp
Σ
wj pi
(3)
where Lp is a linear projection layer applied to the gated prompt embeddings. Further details are provided in the Appendix E. The input prompt embedding matrix for each session E E Rd is then retrieved, such that E," M matching item 8; within the user's interaction sequence.
efffciintly transferring information between modalities, we re-design the mid-fusion module such that the embeddings for each item are aligned in time. This design introduces strong prior that the embeddings from each modality be- long to the same item in the sequence, ensuring that repre- sentations of the same item from different modalities are effectively fused.
3.3. Multimoddl Transformer with Time-aligned Shared Token Fusion
The multimodal transformer is built based on the Trans- former encoder architecture (Vaswani, 2017), where each feature has its own encoder, to begin with. The user se- quence is processed through each feature extractor, produc- ing five distinct embeddings: Eld, E Elext EP, Eprice, all in ædx=, Positional embeddings are added each em- bedding, and CLOZE embedding zez is added to the quence to signify the item to be predicted. The recommenda- tion is made based on the final representation of zez- Using the ID embedding matrix Fid as an example, the final input to the ID encoder is:
Each sequence contains n tokens, including the CLOZE embedding Zca (Sun al., 2019). The num- ber of time-aligned shared tokens (z*h) matches the se- quence length, with each shared token 2," at time step corresponding to modality-specifie tokens 2, where mod E {id, style, text, pt, price]. These shared tokens en- able cross-modal interaction for the same item at each time step. For each modality. modality-specifft tokens = tokens 2 [21 . 2 = are concatenated and fed into the self-attention encoder Atl (Vaswani, 2017). It processes input tokens from each modality independently through L transformer layers, where the transformation at layer is defined as:
and the time-aligned shared Z = [2 ..... 2n mod mood]
Z D Eid Ei 2 ① p,
(4)
where p E In+ is the learnable positional em- bedding, and @) denotes element-wise addition. Thus, we obtain the inputs to the multimodal transformer: Zdd, Zatle Z 2m Zprice, which are passed through sep- arate encoders.
z = Alt (tt mo 12hh1-11)
(5)
3.3. 1. SELF-ATTENTIO ENCODER
At each layer, modality-specific tokens are updated into mod, for the next layer L. These tokens capture the modality-ppecific features, which will be further enhanced by interaction with the TST module. Similarly, for time- aligned shared tokens, the tokens from all modalities averaged at the end of each layer to form the next layer's ss shared tokens zhh. facilitating further fusion across subse- quent layers. This process is defined as:
In the next step, input embeddings from each modality Z are processed independently through a self-attention encoder (Vaswani, 2017). This approach enables each modality to learn its unique features, and the resulting representations are then prepared for cross-modal fusion using the TST module. Further details are provided in Appendix F.
3.3.2. TIME-ALIGNED SHARED TOKEN FUSION WITH SEQUENTIAL MULTIMODAL INTEGRATION
thd
#mod Σ
Z mod, 1
(6)
Our TST fusion module, as illustrated in Figure 2, facilitates multimodal information sharing by aligning item tokens across different modalities. Inspired by the attention bottle- neck mechanism (Nagrani et al., 2021), which focuses on
Vmod E {id, style, text, pt, price}.
In the first layer of the fusion encoder, the coodality-speciiii sequences do not take inputs from the time-aligned shared
5

---

MTSTRee: Multimodal Time-Aligned Shared Token Recommender
tokens yet, as these tokens are initially unlearned. However, each shared token 2dh take inputs from its corresponding modality tokens, learning multimodal information. From the second layer onward, moodalit--ppecifi tokens and their corresponding shared tokens attend to each other, enabling cross-modal learning and information exchange.
E-commerce and House-Hold E-commerce, which have al- ready been made publicly available', and one public dataset from H&M. The proprietary datasets capture user interac- tions, including page views and purchases, while the public dataset focuses on purchase records in the trousers cate- gory from H&M. Detailed information, including dataset splitting, is provided in Appendix G.
After fusion, the learned representations are passed to the prediction layer. The sequence token Sez plays a critical role in capturing the information needed for predicting the next item in the user's interaction sequence. By integrating infor- mation from the speciit and time-aligned shared tokens, the model achieves a cohesive multimodal represen- tation, improving the accuracy of the recommendation task. The final modality representations are normalized to ensure stability, then combined and used for prediction and loss computation. The žez tokens are concatenated to form the final recommendation representation for each sequence:
Evaluation Metries. We evaluate all models using three popular top-k ranking metries: Normalized Discounted Cumulative Gain (NDCG@K), Hit Rate (HR @k), and Mean Reciprocal Rank (MRR @ k), with k set to 5 and 10. NDCG@k @k remains unchanged as inherently supports mul- tiple selections. For HR @k and MRR@k, we adjust the calculations to account for the multiple-answer format, en- suring a fair evaluation. For more detail, please refer to Appendix H.
Benchmark Models. We compare our MTSTRec model with two categories of baselines: general models using only item IDs (e.g. SASRec (Kang & McAuley, 2018) and BERT4Rec (Sun et al., 2019)) and multimodal models (e.g., MMMLP (Liang et al., 2023)). The latter also in- cludes the enhanced versions of SASRec and BERT4Rec which concatenate image and text features, similar to model. These modifications allow them to act as early fu- sion models, combining all features upfront to ensure fair comparison of multimodal inputs. For more details on the benchmark models, refer to Appendix
Zontput
-
style| text
-= price
This final representation Poutput encapsulates the key infor- mation from each modality, allowing the model to make accurate predictions for the next item in the user interaction sequence.
3.4. Loss Function
For each sequence, the model computes the cosine sim- ilarity between the final recommendation representation Zoutpot and both the ground truth and negative samples. The embeddings for both the ground truth and negative samples y are generated by concatenating features E ex- tracted for each item through the same feature extractors. Let cos (ouutuut, y represent the cosine similarity between Routpot and the k-th sample, where k E Kyt for positive sam- ples (ground truth) and & E K,, for negative samples. These cosine similarities are then employed to compute the binary cross-entropy (BCE) loss for the sequence as follows:
4.2. Performance Comparison
To assess the generalizability of MTSTRec, we conducted experiments on three datasets and compared the results with baseline models. More implementation details can be found in Appendix J. The results, summarized in Table reveal critical insights into the effectiveness of different models.
MTSTRec surpasses all baselines, leveraging its TST mod- ule to outperform both early and late fusion approaches. In the Food E-commerce dataset, MTSTRec achieves the highest NDCG@S score of O. 0.8800, and in the House-Hold Hold E-commerce dataset, it achieves 0.8942, outperforming SASRec by approximately 3.4% and 9.7%, respectively. Notably, in the H&M (Trousers) dataset, MTSTRec achieves NDCG@* @5 5 of 0.2307, further solidifying its dominance across all metrics.
LACR
Σ Kt
log (cos((zunpus y²) +
ge U Knl
cos(Zoutuut log (1 - c1-cos(22 Zoutput: y ()))
kEKn
This formula drives the model to maximize cosine similarity with ground truth and minimize it with negative samples, improving recommendation performance.
Besides, SASRec+ and BERT4Rec+ which integrate text and image features, further boost performance. In the Food E-commerce dataset, SASRec* achieves an NDCG@5 of 0,8512, and in the House-Hold E-commerce dataset, it reaches 0.8150, which demonstrates the importance of incor-
4. Experiments
4.1. Experimental Settings
'The datasets are available at https: / / thub.com/ ddsplab/MMSS TRec. For more details regarding the dataset relcase, please refer to Appendix O.
Dataset. Our experiments utilize three datasets: two propri- etary datasets from AviviD Innovative Multimedia- Food
6

---

MTSTRee: Multimodal Time-Aligned Shared Token Recommender
Table Performance comparison of henchmark models and MTSTRec three datasets.
NDCG@5
NDCG@
MRR@5
MRR
SASREC
0.8015
U.7999
0.7505
0.7836
0.7099
0.7143
BERT4REC
0.8076
0,8094
0.7658
0.8010
0.7146
0.7193
Food
SASREC
0.8512
0.8446
0.8013
0.8250
0.7729
0.7760
DERT4RES
0.8441
0.8394
0.7962
0.8218
0.7641
0.7675
MMMLP
0.7487
MMMLP
0.8164
0,8130
0.7742
0.8048
0.7338
0.7379
MTSTREC
0.8800
0.8765
0.8407
0.8651
0,8086
0,8118
SASREC
0.7563
0.7736
0.7578
0.7960
0.7024
0.7076
BERT4REC
0.7596
0.7761
0.7603
0.7968
0.7068
0.7116
House-Hold
0.8258
0.8144
0.84
0.7688
0.7723
E-commercc
BERT4RECT
0.7959
0.7477
0.7521
MMMLP P
0.8335
0.8425
0.8303
0.8525
0.7930
0.7959
MMMLP
0.8401
0.8480
0.8368
0.8569
0.7993
0,8019
MISTREC
0.8942
0,9086
0.9067
0.9358
0,8568
0.8607
SASREC
0.1520
0.1759
0.1809
0.2489
0.1258
0.1348
BERT4REC
0.1468
0.1692
0.1738
0.2378
0.1223
0.1306
H&M
SASRECT
0.1605
0,1811
0.1828
0.2415
0.1371
0,1448
(Trousers)
BERT
0.1633
0.1848
0.1878
0.2492
0.1387
0.1466
MMMLP
0.1923
0.1971
0.1502
0.1565
MMMLPT
0.1710
0.1889
0.1927
0.2436
0.1461
0.1528
MISTREC
0.2307
0.2797
0.3139
0.4481
0,1871
0.2049
porating multimodal data to better capture user preferences and provide more accurate recommendations. However, these models still fell short of MTSTRec's performance.
0.8800 to 0.7582 on the Food E-commerce dataset. This highlights the critical role of product identity in differentiat- ing items. The absence of item IDs also increased training time the model struggled to manage without clear product identifiers. See Appendix K for detailed analysis of the impact of removing II) modules across different datasets. Text information proves to be another important modality. In the House-Hold E-commerce dataset, removing both text and prompt text leads a noticeable performance drop. with NDCG@5 decreasing from 0.8942 to 0.8191. This highlights the significance of detailed product descriptions in capturing user preferences. Even when standard text is removed, the prompt text still provides valuable contextual information, enabling the model to maintain reasonable ac- curacy, as indicated by smaller drop to 0.8488 NDCG@5 compared removing both text and prompt text.
MMMLP uses the same features MTSTRee, incorpo- rating additional features compared to MMMLP. While MMMLP* shows slight improvements over MMMLP in the House-Hold E-commerce dataset, performs slightly worse in the other two datasets. This suggests that MMMLP is bet- ter suited for simpler inputs. When comparing MMMLP+ with MTSTRee, MTSTRec demonstrates a superior abil- ity to process and leverage the complex relationships be- tween multiple features. As a result, MTSTRec achieves significantly better results across all three datasets, further highlighting its effectiveness in capturing multimodal inter- actions.
In summary, MTSTRec consistently outperforms both early fusion models (SASRec* and BERT4Rec*) and the late fusion model (MMMLP) across all evaluated datasets. The model's effective integration of multimodal information via the TST module ensures that feature interactions are fully captured, leading to state-of-the-art performance, par- ticularly in complex datasets like H&M (Trousers), with performance gains of approximately 43.7% over competing models in terms of NDCG@5.
In summary, while item ID is crucial for accurate recom- mendations, approach surpasses the ID-only baselines (SASRec and BERT4Rec) by demonstrating the added value of integrating multiple modalities. Our multimodal design provides complementary strengths beyond the limitations of ID-based methods. Interestingly, despite its relevance in consumer decisions, price contributes little to recom- mendation accuracy, suggesting that preferences better captured through semantically rich and behaviorally informative features. Our approach demonstrates the power of leveraging multimodal inputs to enhance accuracy and robustness predicting user preferences.
4.3. Ablation Study of Modalities
In this section, we conduct an ablation study on Food E- commerce and House-Hold E-commerce datasets to evaluate the contribution of different modalities, including item ID) text, prompt text, images, and price. As shown in Table 2, removing the item ID modality has the most significant im- pact on both datasets, with NDCG @5 dropping sharply from
To better understand how MTSTRec allocates attention across modalities, we visualize self-attention produced just before the final output layer of the TST module the House-HHold E-commerce dataset (Figure 3). The attention maps reveal distinct podality-speciife patterns. Product

---

MTSTRee: Multimodal Time-Aligned Shared Token Recommender
Table 2. The impact of removing different modality modules across e-commerce platforms.
ABLATION STUDY
commence
House Hold coommrcce
NDC
HRWS
MR
NDCGUS
HRWS
MRR
MISTREC
0,8800 10.0023)
0.8407[10.00:1]
0,8086( 1 0.0023)
0.8942, 10,0035)
0.9067(10.0024)
0.8568(10.004]
W/O ID
0.7582((00.0049)
0.7506r20.00499
0.6459(20.0002)
(0013(40.0244)
0.8337(=000191))
0.718-((-002288)
W/O TEXT & PROMPT
0,8574(=000001
(8100((+000011)
0.7813(20.0015)
0.8191 0047)
0.8142(#0.0041)
0.7761 0062)
W/O TEXT
18829)(20.00166
0.8308(++000224
279980=0.0022
(4488(=0000995
0.8553(=0000199
0.8051(=#00003 0087)
WIO PROMPT TEXT
TEXT
0.874%(:10.0044
0.8331(+0.0056)
0,8030 0.004))
0.8770(1:0 00035)
0.8895( 0.0099
0.8366(1-00031))
W/O STYLE
0.8784(110.002)
0.8391(1:0.0038)
18068(10.0020
0.8932(1:0 0093)
0.906 0.0002)
0.8524(10.0111)
W/O PRICE
(=+0.002))))
0.8404(20.0031)
0.8077((000000)
0.8941(##00004)
0. 0,9066 00040)
0. 0.8553(20.0063
(a) ID
(b) Style
(c) Text
(d) Prompt Text
(c) Price
Figure 3. Self-attention heatmaps from the last TST layer of the MTSTRec model House-Hold E-commerce dataset, visualized across five modalities. Each 20 20 heatmap shows attention weights from queries (y-axis) to keys (x-axis). Brighter regions indicate higher attention. The specific light and dark patterns highlight how the model allocates focus based on content.
ID and price tokens exhibit scattered, weakly structured attention, while style (image) features show broader focus with noticeable vertical stripes. Text and prompt text dis- play strong vertical attention some key tokens and share consistent higher attention in the first half of the sequence (e.g.. indices 1-10) and reduced focus in the latter half. These patterns indicate that text and prompt text convey semantically aligned and informative content. Complementary attention is also observed across modalities; for instance, at indices and 6, style receives strong atten- tion while text and prompt text are relatively weak. This highlights MTSTRec's 's ability to adapt its attention across modalities, reinforcing the importance of a multimodal de- sign for recommendation.
NDCG@5 of 0.8800, compared to 0.8621 for early fusion and 0.8211 for late fusion. Similarly, on the House-Hold E- commerce dataset, TST obtains an even higher NDCG@5 5 of 0.8942, outperforming early fusion (0.8366) and late fusion (0.8773). These trends reflect two limitations of the baseline approaches. First, in early fusion, all features are processed together from the start, which can dilute unique information from each modality, as the model cannot treat them dis- tinctly. Consequently, the model cannot fully leverage the strengths of each feature, leading to reduced performance. The performance gap is particularly noticeable in the House- Hold E-commerce dataset, where early fusion reduces the NDCG@5 by 0.0576 compared to TST. Second, late fusion limits the interaction between different features until the end of the sequence. This restriction hinders the model's ability to capture cross-modal dependencies throughout the sequence, leading to decreased performance. This limita- tion is especially evident in the Food E-commerce dataset, where the NDCG@s 5 score drops from 0,8800 to 0.821 1, decreasing to 0.0589.
4.4. Impact of Fusion Strategies
In this ablation study, we evaluated the impact of different fusion strategies on the Food E-commerce and House-Hold E-commerce datasets, comparing TST (used in MTSTRec) with two alternative setups: one resembling late fusion by removing the TST and fusion encoder; another resembling early fusion by removing the entire multimodal transformer. In the late fusion alternative, features are processed inde- pendently and concatenated only at the final step, and in the early fusion alternative, all features are concatenated immediately after feature extraction and passed through a single encoder.
In conclusion, TST's ability to facilitate cross-modal in- teraction at each time step leads to better results, whereas early and late fusion approaches fall short due to inefficient handling of feature interactions.
4.5. Impact of Time-aligned Shared Tokens
We evaluate the different shared token configurations on the Food E-commerce dataset, focusing on three key setups: the proposed TST (1:1), TST with multiple shared tokens per
TST consistently outperforms both alternatives as shown in Table 3. On the Food E-commerce dataset, TST achieves
8

---

MTSTRee: Multimodal Time-Aligned Shared Token Recommender
Table 3. The impact of removing different modules across e-commerce platforms.
DATASET
FUSION METHOD
NDCG@5
NDCG@10
HR@6
HR
MRR@5
Food
MTSTREC
0.8800
0,8765
0.8407
0.8651
0.8086
0.8118
WiO TST & FUSION ENCODER (LATE FUSION)
0.8211
0.8301
0.7912
0.8378
0.7271
0.7333
E-commerce
W/O MULTIMODAL ENCODER (EARLY FUSION)
0.8621
0.8590
0.8206
0.8483
0.7862
0.7899
MISTREC
0.8942
0,9086
0,9067
0.9358
0.8568
0.8607
House-Hoke
E-commerce
W/o TST FUSION ENCODER (LATE FUSION)
0.8773
0,8896
0.8839
0.9116
0.8392
0,8429
W/o MULTIMODAL ENCODER (FARLY FUSION)
0.8366
0.8516
0.8404
0.8738
0.7929
0.7974
Table 4. The impact time aligned shared tokens on Food E-commerce dataset results.
SETTING
NDCG@5
NDCG@10
MRR
TST (OURS)
0.8800 )
0.8765(#0.0021)
0.8651c=000020
0.8086((#00002)
0.8118 000233)
(1:(1:2)
0.8795(10.00077 0002))
0.8758(#0.0000)
0.84066:E0 0025)
0.8641(=0.0023)
0. 0.8079 0,026)
0.811D0 0026)
TST (1:4)
0.8769(10 1.0 0041)
0.8737(11.000)
1.8384(1100001)
0.8631(10.00
0.8044(110.00
0.8077(110.00) 0.005
BOTTLENECKS (ALL:5)
2.8737(#0000238
0.8693(20.0024)
0.8323(++0.0030
0.8560
0.8012
0.8044C==000022
BOTTLENECKS (ALL21)
0.8754(10.0036)
0.8716(110.005)
0.8346(1.0.0444
0.8587(10.0040)
0.8037(11000033)
0.8069(11000035
time step (TST (1:2) and TST (I:4)), and fusion bottlenecks. Each configuration represents a unique mid-fusion approach to feature sharing during the sequence learning process. Further details are in the Appendix L.
our Time-aligned Shared Token (TST) module introduces only one extra token per time step (plus a prediction token), incurring minimal overhead in practice. With optimized implementations, MTSTRec typically involves only minor constant overhead compared to standard Transformer-based sequential recommendation models. For completeness, we report parameter sizes, training durations, and inference times for all models in the Appendix M. We further ana- lyze the impact of output token configurations on model performance, presented in Appendix N.
As shown in Table 4, the proposed TST (1:1) configura- tion delivers the best performance, with an NDCG@5 of 0.8800. This setup ensures that only one shared token per time step facilitates information transfer, which helps the model focus on product-speciffs features without introduc- ing unnecessary noise. TST (1:2) and TST (1:4) show a slight drop in performance, with NDCG@5 ranging from 0.8769 to 0.8795. The increase in shared tokens per time step introduces more information exchange but also adds redundancy and potential noise, slightly affecting model effi- ciency. Bottlenecks configurations perform worse than TST, with BOTTLENECKS (ALL:5) (Nagrani et al., 2021) achiev- ing an N of 0.8737 and BOTTLENECKS (ALL:21) achieving an NDCG@5 0.8754, which is 0.5% lower than TST (1:1). Despite allowing broader information sharing, these setups lack the precise, time-aligned interaction that enhances TST's performance.
5. Conclusion
We introduce MTSTRec, novel Multimodal Time-aligned Shared Token Recommender that fuses and transmits essen- tial information across different modalities. Our approach allows for precise integration of multimodal features such as product IDs, images, text, and prices while maintaining the unique contributions of each modality. Extensive experi- mentation shows that MTSTRec significantly outperforms ofte--of-the-art baselines across various evaluation metrics on real-world e-commerce datasets. Our ablation studies re- vealed the critical role of each feature type in improving rec- ommendation accuracy, particularly the importance of item identity and textual descriptions in different e-commerce scenarios. Moreover, we showed that the proposed TST fusion method consistently surpasses both early and late fusion strategies by enabling tross-modal interaction that aligns in time throughout the sequence. In summary, MT- STRec represents a significant advancement in multimodal sequential recommendation, offering a flexible and efficient framework that can be adapted to various e-commerce appli- cations. Future work will focus on extending the model to other domains and exploring additional multimodal features for even more personalized recommendations.
The TST (1:1) configuration outperforms other setups, in- cluding multi-token and fusion bottleneck approaches, likely due to its ability to maintain precise, time-aligned interac- tions between product features. This configuration effec- tively balances information sharing and efficiency, making it the most optimal choice for multimodal sequential recom- mendation in our scenario.
4.6. Complexity and Runtime Analysis
The time complexity of MTSTRec is primarily dominated by its Transformer layers, each with standard complexity of On . 2d). where is the sequence length and d is the embedding dimension. Naively fusing m modalities would result a worst-case complexity of O(m- n². d). However,

---

MTSTRee: Multimodal Time-Aligned Shared Token Recommender
Acknowledgments
Gatys, L. A., Ecker, A. S., and Bethge, M. Image style trans- fer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2414 2423, 2016.
This work was supported in part by the National Science and Technology Council (NSTC) of Taiwan under grant number 113-2622-E-002-015. We thank the National Center for fiigh-eeffrmmnnc Computing (NCHC) of National Applied Research Laboratories (NARLabs) in Taiwan for providing computational and storage resources. We are also grateful to AviviD.ai ai for providing the datasets.
Geng, S., Liu, S., Fu, Z., Ge, Y., and Zhang, Y. Recommen- dation language processing (RLP): A unified pretrain, personalized prompt & predict paradigm (P5). In Pro- ceedings of the 16th ACM Conference on Recommender Systems, pp. 299-29999111 2022.
Impact Statement
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
This research advances sequential recommendation systems with novel transformer- baseed framework for efficient mul- timodal fusion. By aligning temporal sequences across diverse modalities such as text, images, and prices, the approach enhances recommendation accuracy and personal- ization with broad applications in e-commerce, media, and digital marketing. Its scalable design supports seamless integration of additional modalities, improving user experi- ences in real-world scenarios. However, we acknowledge potential risks, including possible amplification of biases in multimodal data and privacy concerns regarding data handling practices.
He, R. and McAuley, J. VBPR: visual bayesian personalized ranking from implicit feedback. In Proceedings of the AAAl conference on artificial intelligence, volume 30, 2016.
Hidasi, B. Session-based recommendations with recurrent neural networks. arXiv preprint /5/ 1.06939, 2015.
Huang, C.-K., Hsieh, Y.-H., Chien, T.-.,, Chien, L.-C., Sun, S.-H., Su, T.-H., Kao, J.-H., and Lin, C. Scalable numeri- cal embeddings for multivariate time series: Enhancing healtheare data representation learning. arXiv preprint arXiv:2405. /6557, 2024.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Jiang, H., Wang, W., Wei, Y., Gao, 7., Wang, Y., and Nie, L. What aspect do you like: Multi-scale time-aware user interest modeling for micro-video recommendation. In Proceedings of the 28th ACM International conference Multimedia, pp. 3487-3495, 2020.
Anadkat, S., et al. GPT-4 technical report. arXiv preprint S.Xv:22333.87744. 2023.
Campos, P. G., Diez, F., and Cantador, I. Time- rec- ommender systems: comprehensive survey and analysis of existing evaluation protocols. User Modeling and User-Adapted Interaction, 24:67-119, 2014.
Kang, W.-C. and McAuley, Self-attentive sequential recommendation. In IEEE international conference on data mining (ICDM), pp. 197-206. IEEE, 2018.
Devlin, J., Chang. M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In North American Chapter of the Association for Computational Linguistics, 2019.
Li, C., Xia, L., Ren, X., Ye, Y., Xu, Y., and Huang, C. Graph transformer for recommendation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '23, pp. 1680-1689, New York, NY, USA, 2023a. Association for Computing Machinery. ISBN 9781450394000 doi: 10. I 1..144//5396188359172. URL /tRLtt////////io///// N0.1145/3539618.882
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenbom,, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Li, R., Deng, W., Cheng, Y., Yuan, Z., Zhang, J., and Yuan, F. Exploring the upper limits of text-based collaborative filtering using large language models: Discoveries and insights. arXiv preprint arXiv:2305. 1700, 2023b.
Dubey, A., Jauhri, A.. Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Liang, J., Zhao, X., Li, M., Zhang, Z., Wang, W., Liu, H., and Liu, Z. MMMLP: Multi-moda multilayer perceptron for sequential recommendations. In Proceedings of the ACM Web Conference, pp. 1109-1117, 2023.
Gatys, L. A. A neural algorithm of artistic style. arXiv preprint arXiv: 5508 06576, 2015.
10

---

MTSTRee: Multimodal Time-Aligned Shared Token Recommender
Liu, F., Cheng, Z., Sun, C., Wang, Y., Nie, L.,, and Kankan- halli, M. User diverse preference modeling by multimodal attentive metric learning. In Proceedings of the 27th ACM international conference on multimedia, pp. 1526-1534, 2019.
Shani, G., Heckerman, D., Brafman, R. and Boutilier, C. An mdp-based recommender system. Journal of machine Learning research, 6(9), 2005.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. 3rd Interna- tional Conference on Learning Representations (ICLR), pp. 1-14, 2015.
Liu, Q. Hu, J., Xiao, Y., Zhao, X., Gao, J., Wang, W., Li, Q. and Tang, J. Multimodal recommender systems: A survey. ACM Comput. Surv., 2024. ISSN 0360-0300.
Sun, F., Liu, J., Wu, Pei, C.,, Lin, X., Ou, W., and Jiang, P. BERT4Rec: Sequential recommendation with bidi- rectional encoder representations from transformer. In Proceedings the 28th ACM international conference on information and knowledge management, pp. 1441-1450 2019.
Lyu, H., Jiang. S., Zeng, H., Xia, Y. Wang, Q., Zhang, S., Chen, R., Leung. C., Tang, J., and Luo, LLM-Rec: Personalized recommendation via prompting large lan- guage models. In Duh, K., Gomez, H., and Bethard, S. (eds.), Findings of the Association for Computational Linguistics: NAACL, pp. 583-5131 612. Association for Com- putational Linguistics, 2024.
Ugurlu, B. Style4rec: Enhancing transformer-based e- commerce recommendation systems with style and shop- ping cart information. NTU Theses and Dissertations, pp. 1-59, 2023.
Meng, Z., McCreadie, R., Macdonald, C., and Ounis, I. Exploring data splitting strategies for the evaluation of recommendation models. In Proceedings of the 14th ACM conference on recommender systems, pp. 681-686, 2020.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Wang. L., Ma. C.. Wu, X., Qiu, Z., Zheng, Y., and Chen, X. Causally debiased time-aware recommendation. In Pro- ceedings of the ACM Web Conference 2024, WWW '24, pp. 3331-3342, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400701719. doi: 10. 1.1145555893343665550000 URL httptttt//ioi.orrr 10.1145/3589334.33
Moreira, G. d. S. P., Rabhi, S., Lee, J. M., Ak, R., and Oldridge, E. Transformens4Roe: Bridging the gap between NLP and sequential session- based rec- ommendation. In Proceedings of the 15th ACM Con- ference on Recommender Systems. RecSys '21, pp- 143-153, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781 9781450384582. doi: 00.1145/3460231.3477333 URL https: /doi.org/ 10.1145/3460233.3744422
Wei, C., Liang, J., Liu, D., and Wang, F. Contrastive graph structure learning via information bottleneck for recommendation. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 20407-20420. Curran Associates, Inc., 2022. URL https:/pproceddnnssserri..se paper FAles//paper/2222//1ee 80330C8884/888777221112222333333 Paper-Confereecceppt
Nagrani, A., Yang. S., Arnab, A., Jansen, A., Schmid, C., and Sun, C. Attention bottlenecks for multimodal fusion. Advances in neural information processing systems, 34: 14200-14213, 2021.
OpenAl. Openai embedding API [fext-embbeddigg---arree, nnww
2024a. URL ambedding-modes onteeddddnn..modllll-----------e
Wu, C., Wu, F., Qi, T., and Huang, Y. MM-Ree: multimodal news recommendation, arXiv preprint arXiv:2104.07407, 2021.
OpenAI. Openai GPT-4o API [gpt-4o-mini), 2024b. URL https::/ gpt-
4o-mini Nommini-edvvaniing-oot----ff- intelligence/.
Wu, L., Zheng, Z., Qiu, Z., Wang. H., Gu, H., Shen, T., Qin, C., Zhu, C., Zhu, H., Liu, Q., et al. A survey on large language models for recommendation. World Wide Web, 27(5):60, 2024.
O'haa K. An introduction to convolutional neural net- works. arXiv preprini
2015.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Yu, C.-C., Hong, M.-Y., Ho, C.-Y,, and Lin, C. Push4rec: Temporal and contextual trend-aware transformer push notification recommender. In IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6625-6629. IEFE, 2024.

---

MTSTRee: Multimodal Time-Aligned Shared Token Recommender
Zhang, Q. Cao, L., Shi, C.,, and Niu, Z. Neural time- aware sequential recommendation by jointly modeling preference dynamics and explicit feature couplings. IEEE Transactions on Neural Nenworks and Learning Systems, 33(10):5125-5137, 2021.
Zhao, W., Zhong, S., Liu, Y., Wen, W., Qin, J. Liang, M. and Huang. Z. DVIB: Towards robust multimodal recommender systems via variational in- formation bottleneck distillation. In Proceedings of the ACM on Web Conference 2025, WWW '25, pp- 2549-2561, New York, NY, USA, 2025, Association for Computing Machinery. ISBN 9798400712746. doi: 10.1 10.1445466664103.7444440. URL https:://di.o..
10.1 10.1455369644003714488
Zhao, Z., Fan, W., Li, J., Liu, Y., Mei, X., Wang, Y. Wen, Z., Wang, F., Zhao, X., Tang, J., al. Recommender systems in the of large language models (LLMs). arXiv preprint ar70v:230..02046, 2023.
Zhong, S., Huang, Z., Li, D., Wen, W., Qin, J., and Lin, L. Mirror gradient: Towards robust multimodal rec- ommender systems via exploring flat local minima. In Proceedings of the ACM on Web Conference 2024, pp- 3700-3711, 2024.
Zhou, H., Zhou, X., Zeng, Z., Zhang, I.,, and Shen, Z. A comprehensive survey on multimodal recommender systems: Taxonomy, evaluation, and future directions. arXiv preprint 23X1::2302.04473, 2023.
Zhou, and Shen, Z. tale of two graphs: Freezing and denoising graph structures for multimodal recommen- dation. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 935-943, 2023.
12

---

MTSTRee: Multimodal Time-Aligned Shared Token Recommender
A. Price Extractor
We represent all product prices in the list using a unified normalization approach for consistency. Following the Scalable Numerical Embedding (SCANE) method (Huang et al., 2024), an item price embedding matrix M E R is constructed, and an imput price embedding matrix Eprice E gdx= captures the product prices within user's interaction history. These embeddings are then scaled by their corresponding price value Pa.. such that E M Pss, for each item 8; in the sequence, enhancing the model's ability to understand pricing patterns in recommendations.
Furthermore, we experimented with a variant of the SCANE method, where the entire price embedding matrix M was replaced such that each element of MPrice was set to 1, effectively disabling embedding learning. In this case, the price embeddings were simply the expanded price values Ps, themselves. In our scenario, this variant yielded better performance, as the embedding effectively represented the expanded price without the complexity of learning additional embedding weights.
B. Implementation Details of the Style Extractor
In our experiments, we extract style features from the first two layers of VGG-19 (Simonyan & Zisserman, 2015) to capture the relevant style information. Each of these layers produces 64 feature maps, resulting in Gram matrices of size 64 x 64 (Gatys, 2015; Gatys et al., 2016).
The Gram matrix for each layer is calculated as follows:
Let F' E gQxxf represent the feature map from layer i, where QI represents the number of feature maps (or channels) in the layer, and R; is the total number of spatial positions, calculated as the height multiplied by the width of the feature map. The Gram matrix G E Q for layer I is calculated as the inner product of the vectorized feature maps Fde and Fje at spatial positions k across all feature maps and j. Mathematically, this can be expressed as:
R -¹
F
jk
where G.yy represents the element of the Gram matrix that captures the correlation between feature map and feature map j in layer I, and the summation over l accounts for all R, spatial positions in the feature map. After calculating the Gram matrices, we apply max-pooling to compress these matrices, which reduces the computational complexity while retaining the essential style information. This step compresses the Gram matrices to 2 16 x 16 style embeddings.
Once each image undergoes the process of Gram matrix computation and subsequent max- max-pooling, the compressed and flattened Gram matrices for all images are concatenated across layers to form the item style embedding matrix M E Rdx[/ This matrix encapsulates the style features of all items in the dataset. The input style embedding matrix for each session Estyee& E Rdx= is then retrieved, such that E = Matyle matching item 8; within the user's interaction history.
C. Comparison of Language Models for Text Embedding
We evaluated the pre-trained language model, BERT (Devlin et al., 2019), and large language models, text-embeddding- 3-large (OpenAI, 2024a), Llama 3 (Dubey et al., 2024), and Llama 3.1 (Dubey et al., 2024). These comparisons were conducted with the MTSTRec model, focusing solely on product ID and text. The handling of product IDs is discussed in Section 3.2.1. For the text portion, we used product titles and descriptions, converting them into embeddings via the respective Language Models, followed by a linear reduction layer that reduces the embedding dimension to
The results are presented in Table 5. Llama 3.1 achieves the best performance across all evaluation metries (NDCG, HR, and MRR). Thus, we choose Llama 3.1 as the backbone for generating text embeddings in our main experiments.
D. Implementation Details and Results of Prompt- Text Feature
As outlined in Section 3.2.4, we employed various prompting strategies to generate rich textual information, with detailed examples of these prompts provided in Table 6. For Basic Prompt, we utilize three variations: Ppares Proger and Pauese. P Ppara prompts the L.LM to rephrase the original product deseription while retaining the same information. Ptags aims for a
13

---

MTSTRee: Multimodal Time-Aligned Shared Token Recommender
Table 5. Comparison of language models for text embedding in MTSTRec.
FEATURE
NDCG@5
5
ID
0.8554
0.8074
0.7792
ID TEXTBERT
0.8585
0,8108
0.7829
TD TEXTorIw:
0.8594
0.8119
0.7838
TEXTULANO
0.8329
ID TEXTLA0060
0.8754
0.8340
0.8037
Table 6. All prompts in the prompt-tex text feature extraction module.
PROMPT TYPE
SYSTEM AND USER PROMPT
SYSTEM:
"FFE is an e-commerce website that sells fresh, healthy, high-quality food products withoud unnecersary additives. You will provided the product
Pyaro
description sold this e-commence website. Your tack paraphraze
USER: "Tiile: preduct title. Description: product deseripta iesipptinn
SYSTEM: FTE
websit that sells fress heabhy, high qughituultty food products without unncersarr additives, You will be provided with the product title and description sold on this e-commerce website. Your task to summarize this product using sugs."
Plags
Description: product descript e-comieecce website sells fresh, healthy, ligh-qualiit food products without unccesssary addltives, You be provided with the product tille and deseription sold this e-commerce website. Your task to infer what other products on the site a consumer might be interested in if they parchase this product."
USER: "Titee: product
SYSTEM: FFE
Puues
USER: "Tide: product title, Description: product descript i
SYSTEM: "FEE is an e-commerce website jellsfresh, heulihy, high quality food productss without unnecessary additives. tack is to tell me what elte I should say if wont to recomumend this product somcone."
para
USER: "Tute: product le, Description; product. descript ion"
SYSTEM: "FEE is e-commerce websue that sells frezh, healthy, high high-quality food products wilhout unnecessary addliives. Your task is tell which tags should be used want recommend this product to someome."
tagg
USER: "Title: product title, Description: product dezcription"
concise summary using tags, guiding the LLM to extract key details. Lastly, P Pquess prompts the LLM to predict what other items the user might purchase based on the product's 's title and description. The Recommendation Prompt extends the Basic Prompt by introducing a recomnendation-orientted task. We define two variations: pree and prec pora lags
In our experiments, we explored text generation with Llama 3.1 (Dubey et al., 2024) and GPT-4o-mini (Achiam et al., 2023; OpenAI, 2024b) and compared their effectiveness in creating useful text embeddings. Each prompt carefully designed with specific settings to match its intended function.
For instance:
This prompt was used to paraphrase the product title and description, configured with a temperature of 0.7, maximum token limit of 256, and lop-p=1.
para:
Progsi For this prompt, which focuses summarizing product information using tags, we set the temperature to 0.5, maximum token limit of 128, and top.p ppppp1. 1.
P This prompt aims infer potential additional products the might purchase based on the current product. It was configured with temperature of 1, maximum token limit of 512, and top-p 1.
ppeaa This recommendation focuseed paraphrase prompt had temperature of 1, maximum token limit of 384, and top 1.
prec lags: Designed to summarize product information for recommendations, this prompt used a temperature of 1, a maximum token limit of 128, and top p=
The text generated by Llama 3.1 (Dubey et al., 2024) or GPT-4o-mini (Achiam et al., 2023; OpenAI, 2024b) was uniformly converted into embeddings using Llama 3.1 (shown Appendix C) to maintain consistency across experiments. This approach allowed for reliable comparisons, foeusing on how different prompts and their corresponding embeddings affected recommendation accuracy in MTSTRee. Each experiment incorporated prompt embeddings alongside the product ID to measure performance.
We also assessed the influence of different LLMs (Llama 3.1 and GPT-4o-mini) on overall model performance by comparing how each model's 's generated text, once converted into embeddings, impacted recommendation results. The detailed outcomes are summarized in the following Table 7.
14

---

MTSTRee:
Multimodal Time-Aligned Shared Token Recommender
Table 7. Comparison of prompt strategies and LLMs (Llama 3.1 GPT-4o-mini) in MTSTRec.
FEATURE
GPT-40
Paara
0.8727
0.8747
Ptags
0.8727
0.8723
ID PROMPT
Pauesa
pree.
0.8718
0.8702
Prec
0,8725
0,8716
AVERAGE
0.8719
0.8718
Based on the results, we observe that incorporating prompt embeddings into the model helps improve performance compared to using only the product ID. While the results for prompt embeddings are not as high as directly using text embeddings, this might suggest that the original product titles and descriptions, when processed by the LLM, sufficiently capture the characteristics of the items. However, our findings demonstrate that prompt embeddings positively influence recommendation outcomes. Additionally, the prompt embeddings generated by both Llama 3.1 and GPT-4o-mini show comparable performance across the five strategies. Llama 3.1 has slight edge on average, which is why chose to primarily use Llama 3.1 generate prompt-text features in subsequent experiments.
E. Gating Weights and Performance of Prompt-Text Text Feature
As explained in Section 3.2.4, a gating network applied to the five prompt strategies to learn their relative importance. The gating weights (Table 8) are derived from the validation results of the ID Text + Prompt features. In addition, we compare the performance of ID Text versus ID Text Prompt, demonstrating improved recommendation accuracy with the inclusion of prompt embeddings, regardless of the LLM used, as shown in Table 9.
Table 8, Gating weights for different prompt strategies.
PROMPT STRATEGY
LLAMA 3.1
GPT-4O-MINI
Pparn
0.1
0.2766
Prage
0.4485
0.1846
Poueas
0.1367
0.1403
0.1328
0.1325
Pingo
0.1428
0.2659
Table 9. Performance comparison of ID text and ID) text prompt-text in MTSTRec.
FEATURE
NDCG@5
ID + TEXT + PROMPT (GPT-40-MM)) 40 MINI)
0.8790
(I.AAMA 3.1)
In conclusion, adding prompt embeddings significantly enhances recommendation performance, shown by the improved NDCG@5 scores. Furthermore, based on the Llama 3.1 gating weights, Ptags emerges critical factor in improving recommendation accuracy. Similarly, the gating weights from GPT-4o-mini align with the results from individual prompt embeddings, reinforcing the importance of specific prompts, such as Pparas Ptags, and tags: in optimizing model performance shown in Table 7.
F. Detailed Architecture and Process of the Self-Attention Encoder
Self-Attention Encoder architecture follows the transformer encoder architecture proposed by Vaswani (2017). The input embedding from each modality Z is processed independently through an independent transformer encoder with multiple layers. Each layer consists of three key components: Multi-Head Self-Attention (MSA), Layer Normalization (LN), and Multilayer Perceptron (MLP), all connected via residual connections. The transformation at layer defined as follows:
YI- MSA(LN(Z ( + LN(Z
(10)
z MLP(LN((
(11)
15

---

MTSTRee: Multimodal Time-Aligned Shared Token Recommender
In the MSA block, self-attention computes the attention scores between tokens, allowing each token to dynamically attend to other tokens in the sequence. The attention mechanism is defined as:
MSA(X) AAttention / X, w X. x)
(12)
where W W and W are the weight matrices used to transform the input tensor X into queries, keys, and values, respectively. The attention scores are calculated as the dot product between queries and keys, enabling the model to prioritize the most important parts of the input.
G. Dataset
Our experiments utilize three datasets: Food E-commerce, House-Hold E-commerce, H&M (Trousers). The Food E- commerce dataset includes total of 1,507,388 interactions, with an average session length of 4.693 and total of 216,576 sessions. The Hold E-commerce dataset consists of 94,984 interactions, average session length of 5.914, and a total of 12,345 sessions. The H&M (Trousers) dataset contains user 3,576,972 purchase records and a total of 416,794 sessions, with further details provided in Table 10.
Table 10. Statistics of datasels.
DATASET
Food
House Hold
H&M (Trousers)
# SESSIONS
216,576
12.345
416.794
# PRODUCTS
770
2,464
11.150
AVG. SESSION
5.914
7.029
AvG. PURCHASE
2.267
1,780
1.553
ACTIONS
1,507,388
94,984
3,576,972
We apply preprocessing steps filter out sessions with fewer than three interactions to ensure sufficient data for model training. Session lengths are limited to 20 items. For shorter sessions, we pad them with zeros, while for longer sessions, retain only the last 20 products, removing any repeat items at the end of the sequence. This adjustment allows the model to better focus on complex patterns, enhancing its applicability to real-world scenarios.
For the first two datasets, each session's 's purchase order involves multiple items, so we treat the answer set as a multiple- choice task. The same preprocessing steps are applied to the H&M (Trousers) dataset. Although it consists solely of purchase actions, we sort the items by purchase time and treat the products bought on the last day as the answer set. Since users may purchase multiple items on the last day, this is also treated as a multiple-choice task. To facilitate model computation, we pad all answers to fixed length of 50 items.
Dataset Splitting, Each session in the proprietary datasets consists of historical clicks and a final purchase order. The data is split chronologically into 75% for training, 12.5% for validation, and 12.5% for testing based on the purchase orders. For the H&M (Trousers) dataset, which contains only purchase actions, items are sorted by purchase time, and those bought on the last day are used as the answer set, ensuring consistency across all datasets (Meng et al., 2020). The model predicts items in purchase orders as multi-labe recommendation task, where each sequence may have multiple correct answers (purchases).
H. Evaluation Metries
For each session, we randomly sampled 100 items that the user did not interact with under the target behavior as negative samples. Finally, we report the results on the test set while selecting the best hyperparameters using the validation set. We detail the three evaluation metrics used in our experiments: NDCG@k, HR@ @k and MRR@K @k where k is set to 5 and 10. Since our problem involves multiple correct answers (multi-label),, we have adjusted the definitions of HR @ k and MRR @k accordingly. HR is calculated by treating each correct answer as a separate single-choice task. The model ranks each correct item among negative samples, and calculate the hit rate for each task. The final HR @k is the average of these hit rates across all correct answers in the sequence. MRR @k follows a similar approach, where we compute the reciprocal rank for each correct answer, and the final MRR@) k is the average across all correct answers. Below are the definitions of each metric, along with an example calculation.
NDCG@K k (Normalized Discounted Cumulative Gain). NDCG@K considers both the relevance and the position of the correct items in the ranked list, with higher-ranked relevant items contributing more to the score.
16

---

MTSTRee: Multimodal Time-Aligned Shared Token Recommender
The Discounted Cumulative Gain (DCG@k) is calculated by summing the relevance scores of the correct items, where the relevance score decreases logarithmically based the item's rank position. The formula for DCG@k is:
k
rel(r)
DCG@k
(13)
loga 1)²
where rel(r) represents the relevance score of the item at rank r, where all items in the purchase order are assigned relevance score of 1. Higher-ranked relevant items contribute more to the final score, as their positions are weighted more heavily in the DCG calculation.
The Ideal DCG (IDCG@k) represents the best possible ranking scenario, where all relevant items are ranked at the top. Since we are normalizing the DCG score, the ideal ranking is computed by assuming the best-case relevance distribution. The formula for is:
min(k.n)
IDCG@k
Σ
(14)
loga(r (r + 1)*
where min(k, n²) ensures that we only consider the smaller of & (the cutoff) and (the number of correct answers).
Finally, the NDCG@k is calculated by normalizing DCG by the ideal DCG. The formula for NDCG@k is:
DCG@k
NDCG@k = IDCG@k
(15)
This normalization makes NDCG range between 0 and 1, allowing consistent comparison across different queries. A higher NDCG score indicates that the correct items are ranked closer to the top.
HR @k (Hit Rate). HR @R@king is calculated by checking whether any of the ground truth items appear within the top k-ranked items. For the multi-label task, we calculate HR for each correct item and then average the results. The formula for HR @+ k is:
1 ml 1 Trank; S ks)
IR@K- =
(16)
1=1
where [rank; < k] is 1 if the correct item i ranked within the top and 0 otherwise.
MRR@E (Mean Reciprocal Rank). MRR @k measures the ranking of the first correct item within the top k positions. For the multi-label task, MRR @k is calculated by averaging the reciprocal ranks of the correct items:
MRR @k = In
(17)
rank
where rank; is the rank position of the correct item
I. Benchmark Models
SASRec (Kang & McAuley, 2018): self-attention-basse sequential model with causal masking that captures long-term user preferences by attending only to previous tokens.
BERT4Rec (Sun et al., 2019): A bidirectional sequential recommendation model that uses self-attention to predict masked items user behavior sequences, capturing both left and right context.
S: An enhanced version of SASRec (Kang & McAuley, 2018), which integrates item ID, text, image features. The text and image features are processed using the techniques from our MTSTRec model. To ensure stable training, we reduce the dimensionality to 256, as the model would otherwise fail to converge properly.
: An enhanced version of BERT4Rec (Sun et al., 2019), which integrates item ID, text, image features. The text and image features are processed using the techniques from our MTSTRee model. To ensure stable training, we reduce the dimensionality 256, the model would otherwise fail to converge properly.
17

---

MTSTRee: Multimodal Time-Aligned Shared Token Recommender
MMMLP (Liang et al., 2023): A multimodal MLP-bbsed model that processes text, image, and price features (with price added for fair comparison) through Feature Mixer Layer, Fusion Mixer Layer, and Prediction Layer, achieving state-of-the-art performance with linear complexity.
An enhanced version of MMMLP (Liang et al., 2023), which integrates item ID, image, text, prompt-text, and price features. The features are processed using the same techniques from our MTSTRee model, ensuring that all five feature embeddings extracted from the feature extractor identical. This ensures fair comparison that focuses solely on the differences in the predictive model architectures.
MMMLP*:
J. Implementation details
In our experiments, we tuned the hyperparameters based on validation data to ensure optimal performance. The batch size was uniformly set to 64 for all models, and the input dimension d was fixed at 512. We employed the Adam optimizer while the maximum sequence length N was set to 20. The fusion layers were standardized across models, with Luuusonn = and dropout rate of 0.1.
For the ID feature encoder, used two transformer blocks (Lnd = 2) with four attention heads, applying a hidden layer dropout of 0.1 to maintain fairness with other benchmark models.
For other feature encoders, such as text, image, and price, we experimented with different settings. The number of each encoder layer (Lmod) was tested across values of (2, 4, 8), and the number of attention heads across {1, 2, 4, 8, 16]. We also experimented with dropout rates of {0.1, 0.2, 0.3} in the hidden layers. The learning rate was tested across range of {0.001, 0.0005, 0.0001, 0.00005, 0.00001), while the L2 regularization penalty was tuned from {0.0001, 0.00005, 0.00001, 0.000005, 0.000001): ). A gamma value of (0.9, 0.75, 0.5) was set for learning rate decay.
For the baseline models (e.g., SASRec (Kang & McAuley, 2018), BERT4Rec (Sun et al., 2019)), we ensured that key settings such as batch size, the number of encoder blocks, and attention heads were aligned with our model for fair comparison. However, for other settings, we followed the recommended configurations in the original papers.
K. The Impact of ID Modules Across Different Datasets
This section examines the performance impact of removing ID modules in the MTSTRec model across three datasets: Food E-commerce, House-Hold E-commerce, and H&M (Trousers). Table 11 highlights notable performance differences between the full model and the version without ID modules. The results show that ID modules play crucial role in Food E-commerce and House-Hold E-commerce datasets, significantly boosting recommendation accuracy. In contrast, their removal in the H&M (Trousers) dataset unexpectedly improves performance.
Table 11. The impact of removing ID modules across e-commerce platforms.
Food E-commerce
House-Hold E-commerce
H&M (Tousers)
ABLATION STUDY
NDCG@S
MRRW5
NDCG
IRQ5
MRRWS
NDXG0S
HRW3
MRR
MISTREC
0,8800
0.8407
0.8086
0.8942
0.9067
0.8568
0.2307
0.3139
0.1871
W/O ID
0.7582
0.7506
0.6459
0,7913
0,8337
0,7184
0.2724
0.3676
0.2232
In the Food E-commerce and House-Hold E-commerce datasets, the removal of ID modules results in a noticeable decline in all metrics. For example, in the Food E-commerce dataset, NDCG@5 drops from 0,8800 to 0.7582, while HR @ 5 decreases from 0.8407 to 0.7506. Similarly, in the House-Hold E-commerce dataset, NDCG@5 decreases from 0.8942 to 0.7913, with HR @5 following a similar trend. One primary reason for this decline is the smaller number of unique products in these datasets, as shown in Table 10. With only 770 and 2,464 products, respectively, these datasets allow product IDs to serve as highly discriminative features, enabling the model to learn precise user-item interactions. The supervised learning framework further reinforces this dependency, as the model optimizes embeddings for individual IDs, which are particularly effective in smaller product spaces.
Conversely, in the H&M (Trousers) dataset, the removal of ID modules results in performance improvement, with NDCG@5 increasing from 0.2307 to 0.2724 and similar gains observed in HR@5 and MRR@S. @5. This improvement can be attributed to the larger and more diverse product space, which includes 11,150 unique items. The increased number of products introduces sparsity in user-product interactions, making embeddings less effective representing individual
18

---

MTSTRee: Multimodal Time-Aligned Shared Token Recommender
items. Moreover, the H&M (Trousers) dataset defines items with identical styles but different colors separate products, which complicates the embedding space and reduces the distinctiveness of product IDs.
These findings underscore the lataset-dependee role of ID modules. Product IDs are critical for achieving high recom- mendation accuracy in domains with fewer products and consistent user behavior. However, in datasets with larger product spaces and greater complexity, the reliance on IDs diminishes, requiring models to leverage other features. This highlights the importance of multimodality, as incorporating information from other modalities (e.g., images or text) provides richer insights, enabling models adapt better diverse datasets.
L. Configuration Details of the Shared Token
Time-aligned Shared Tokens (TST) (1:1), used in our MTSTRec model, pairs each time step in the sequence with a corresponding shared token in a 1:1 relationship. This means that each product in the sequence is aligned with single shared token, allowing features from different modalities of the same product to interact and share information at that specific time step. This design ensures that information sharing precise and time-aligned, leading to more accurate feature fusion.
TST (1:2) & TST (1:4) include configurations where each time step is associated with multiple shared tokens rather than just one. For example, in TST (1:2), each product in the sequence is paired with two shared tokens, and in TST (1:4), each product is paired with four shared tokens. These variants allow for more extensive information sharing at each time step.
Fusion Bottlenecks (all:4+1) is based on configuration from Google (Nagrani et al., 2021) originally designed for sequence fusion classification tasks involving image and speech data. In this setting, shared tokens are not tied to a specific time step but attend to the entire sequence, enabling broader information exchange across the sequence. For a fair comparison in our multimodal sequential recommendation task, we adapted this method to a Fusion Bottlenecks (all:20+1) configuration, matching the number of tokens used in our TST approach.
M. Parameter Size and Runtime Analysis
We report detailed statistics on the number of parameters, training time, and inference time for all models evaluated on the House-Hold E-commerce dataset. As shown in Table 12, MTSTRec incurs longer training time due to its multimodal architecture and the inclusion of the Time-aligned Shared Token (TST) module. Nevertheless, its inference time remains competitive, only marginally higher than that of other multimodal baselines. Despite incorporating additional modality- specific encoders and fusion layers, MTSTRec is significantly more compact than multimodal baselines such as SASReet, BERT4Rec* MMMLP, and MMMLP These results indicate that MTSTRec achieves a favorable balance between model size and effectiveness without compromising expressiveness or accuracy.
Table 12. Parameter size, training time, and inference time the House-Hold E-commerce dataset.
PARAMETER (MILLION)
TrainING TIME (MINUTE)
INFERENCE (SECOND)
SASREC BERT4REC
4.69
4.5
1.86
7.84
4.1
1.88
SASRECT 203.67
BERT4REC+
400.33
59
7.89
MMMLP
97.38
33
6.56
MMMLP
174.54
39
7.27
MISTREC
59.24
83
12.43
N. Output Token Design Choices
N.1. Use of Modality-Speciffi CLOZE Tokens (=.z)
During model development, we compared two strategies for summarizing each modality's 's sequence: (i) using the output of the last token, and (ii) appending a dedicated CLOZE token (Zez) at the end of each modality sequence. As shown in Table 13, the latter consistently outperformed the former on the House House-Hold E-commerce dataset. Introducing a Zez token
19

---

MTSTRee: Multimodal Time-Aligned Shared Token Recommender
per modality provides dedicated placeholder for the next-item prediction, enabling the model to aggregate relevant signals from each modality's perspective. This led to clearer and more informative multimodal representations for downstream prediction.
Table 13, Performance impact of modality modality-specifie CLOZE token (Zex) on the House-Hole E-commerce dataset.
MODEL
NDCG@5
HR@5
MRR@5
MTSTReC
0.8942 -035)
0.9067 (+0.0024)
0.8568 (+0.000 0041) 0.7553 (=0. 0043)
w/o CLOZB
0.8115 0023)
0 8.88275 (+0.0027)
N.2. Excluding the Shared Token (Zah.) from Final Output
During model development, we also investigated whether including the shared fusion token 2sh in the final prediction output would improve model performance. While 2sh plays a crucial role in aligning cross-modal features during the fusion, our experiments showed that naively incorporating it into the output layer degraded prediction accuracy. The comparative results summarized in Table 14.
Table 14. Performance impact of the shared token (Zsh) in the final output the House-Hold E-commerce dataset.
MODEL
NDCG@5
HR 55
MRR@5
MISTREC
0.8942 (20. 0035)
0.9067 (#0.0024)
0.8568 0041)
CONCATENATE Pah 0.8106 (=0.0089)
0.8692 (+0.0067)
0.7 (+0.0108)
O. Dataset Release Document
0.1. Introduction
We have released two datasets, Food E-commerce and House-Hold E-commerce, both collected with full user consent from e-commerce platforms. These datasets contain user interaction data from October 2023 to June 2024, including page views and purchases.
O.2. Dataset Overview
The Food E-commerce and House-Hold E-commerce offer rich source of user interaction data. Since each order can contain multiple purchased products, each sequence may have multiple correct answers (purchases). We provide a statistical summary of the raw data in Table 10. The statistics include key metrics such as #Sessions, representing the number of user sessions in the dataset; #Productss indicating the total number unique products; Avg. Session, which is the average length of user sessions; Avg. Purchase, the average number of products purchased per order; and fcctions, capturing the total count of product browsing records and purchase actions.
O.3. Feature Preprocessing
The style features of product images are extracted using the first two layers of the VGG-19 (Simonyan & Zisserman, 2015) network. We calculate Gram matrices (Gatys, 2015; Gatys al., 2016) from these layers capture the relationships between feature maps and then apply max-pooling to compress the style information. Each image is ultimately represented a 512 112-dimensional embedding that summarizes its style characteristics.
Image Style Embedding:
Text Embedding: For textual features, we process product titles and descriptions using Llama 3.1 (Dubey et al., 2024). This generates a 4096-dimensional embedding for each product. Additionally, we create five prompt-based text embeddings to highlight different aspects of the product information. These embeddings provide a comprehensive representation of the textual data. (see more details in Section 3.2.4)
0.4. File Location
The files containing the datasets and related resources are organized in structured directory format for easy access and navigation. Below is an overview of the file locations, using the Food E-commerce dataset as an example. The Holde-Hold
20

---

MTSTRee: Multimodal Time-Aligned Shared Token Recommender
E-commerce dataset follows the directory structure.
Food/browse.see petissssee_trree seq_train, browse_seq_val. browse_sed hwtpwe_seq_eee seq_test: These files contain the training, validation, and testing sequences of user interactions, including browsing records and purchase actions within Food E-commerce.
Food/sessionaate train, sessiondate. sessiondate and testing data within Food E-commerce. The corresponding purchase date for each session is provided to determine which recommended products can be offered.
stw.oioddtteeasssssiinteeeeeeee test: These files contain the purchase date of training, validation,
Food/Product Feature/PriceFeatuue:: This file contains the sale price feature of each product within Food E-commerce.
Food/Product haarrodduteFaattur/maaeeeeeee Stteellleemmeddingn.......... Food E-commerce.
This file contains the style embeddings of each product within
Food/Product Feature/Text Embedding TitleDescription, Text Embedding Basic Paraphrase, Text Embedding Basic_Tags, Text Embedding Basic Buess Text Embedding Rec Paraphrasp Text_ Eewedding-Reeeee. Rec. - Tags: These files contain the different text embeddings each product within Food E-commerce including Title and Description, Basic Paraphrase Prompt, Basic Tags Prompt, Basic Guess Prompt, Recommendation Paraphrase Prompt, and Recommendation Tags Prompt.
Food/Month. Food/Mooth_Proddcc20011111.1 Prodduct2201produuttistt productlist: This file contains the product index for the Food E-commerce platform in January 2024. It provides a snapshot of the products available at the beginning of the month, serving as a reference for predicting which items should be recommended hased on their corresponding purchase dates. The folder Month Product contains eight files, namely: 23 proouuctutt 2311 1311.producctis 2312 111opprodutllit, 2401 prodductttl,, prodpptou 2403 produccot .roduucou,,,,ist, and 2406_ producctu...
0.5. Dataset Example
The following examples illustrate the structure and content of the datasets used in this research. Each entry in the dataset consists of various encoded features that represent different aspects of user interactions, item properties, and contextual information. The examples below showcase typical data points, including encoded identifiers, interaction sequences, and embeddings extracted from pre-trained language models. These examples are representative of the format and type of data that researchers will work with when utilizing Food E-commerce and House-Hold E-commerce datasets for their studies.
browse seq
Content format (each line): [ (bbrowse.rroduet.ii (purchase (The product IDs start from
1.)
Example: [[1, 57, 35, 102, 435], [720, 102, 35]] ]
sessiondate
Content format (each line): purchase.datetine
Example: datetime, datetime (2024, 3, 16, 0, 0, 0, 0)
PriceFcature
Content format: every product sale price (normalization) (From the price the product with ID to the product with ID 1.)
Example: [0. 5035971223021583, 0. 0.5392953929533225.
0.625490196078414,
0. 62 941 01335224117577059
Image Style Embedding
Content format (each line): gram matrixes of product (dim=512) (The lines sorted based on the product index. Since our index starts from 1, the first line is filled with an all-zero embedding.)
Example: tensor ( [0.0033, 0. 0204,
00977,, 0. 0.0167))
Text Embedding
Content format (each line): Llama 3.1 embedding of a product (dim=4096) (The lines are sorted based on the product index. Since our index starts from 1, the first line is filled with an all-zero embedding.) Example: tensor (667, -0.1427, -3.3,47, -0. 9397]]
21

---

MTSTRee: Multimodal Time-Aligned Shared Token Recommender
2403_ #403_productlis Content format: product idx 1ist Example: [587, 588, 183, 184]
0.6. Conclusion
This technical appendix provides comprehensive overview of the datasets released part of research. The Food E-commerce and House-Hold E-commerce datasets serve valuable resources for advancing the study of multimodal recommendation systems. With their diverse interaction data and carefully curated features, these datasets offer a unique opportunity for researchers to develop and evaluate cutting-edge models. The provided file locations and preprocessing steps ensure that users can easily access and utilize these datasets for further research and development efforts
22


Reviews:
[{"review_id": "IN7vovJBqw", "reviewer": "Reviewer_W2Te", "summary": "The authors propose a sequential recommendation framework focusing on multi modal feature fusion. In the proposed models, the authors include feature sets like product IDs, images, text, and prices.The main contribution comes from authors proposing a new block named Time-aligned Shared Token Fusion module. Each modal has its own self-attention block and the TST module learns an element wise average pooling token (z^sh) to be concatenated together with the modal specific token z^mod to be fed into the next layer's input to learn the mod specific patterns for the next layer. This basically forced the model to add a fused cross modal feature input into each layer. The authors conducted offline analysis on 3 datasets, including performance comparison and ablation evaluations on importance of multi-modal features and impact of different ways of multi-modal fusion.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate the recognition of the practicality and clarity of our proposed method, the comprehensiveness of our experiments, and the potential impact of releasing two of our datasets to the public. We are especially grateful for your comment that “a not-complicated but working solution is way more important than an over-complicated algorithm design,” which aligns with our goal of proposing practical and effective methods for real-world recommendation scenarios.\n\nWe have carefully addressed the reviewer’s comments and questions as follows.\n\n\n**[W1]** Lack of online testing [Weaknesses]\n\n**[R1]** We completely agree with your comment on online testing. This work is part of an industry-academia collaboration, and the proposed MTSTRec model has so far been developed and evaluated using historical data. As the next step, an online A/B test is planned to assess the model’s performance in a live setting. This paper, if accepted, will serve as strong evidence to convince the top management of the company to go ahead with the online testing. We hope to share what we will learn with the research community soon. \n\n\n**[W2]** Model size vs. performance gain/parameter comparison [Weaknesses]\n\n**[R2]** We acknowledge that MTSTRec has more parameters than the baseline models that are based solely on historical interaction data, such as SASRec and BERT4Rec, due to its modality-specific encoders and fusion layers for handling multimodal inputs. The parameter sizes of the models used in our experiments are:\n* MTSTRec: 59.24M\n* SASRec: 4.69M\n* BERT4Rec: 7.84M\n* SASRec+: 203.67M\n* BERT4Rec+: 400.33M\n\nAlthough we were unable to obtain the exact parameter sizes for MMMLP and MMMLP+, MTSTRec is significantly smaller than SASRec+ and BERT4Rec+, which serve as enhanced multimodal baselines.\nAs shown in our ablation studies, the performance improvements of MTSTRec are not merely due to model size but largely result from the proposed TST fusion module and time-aligned multimodal design.\n\n\n**[Q3]** Why add a $z_{cz}$ token instead of using the last token? [Weaknesses]\n\n**[R3]** We have explored both approaches during development. Our experiments showed that relying solely on the last item token often obscured modality-specific contributions, particularly in short sequences or sessions with abrupt shifts, as it conflated signals across modalities. In contrast, adding a $z_{cz}$ token per modality acts as a dedicated placeholder for “the next item,” aggregating relevant signals from each modality’s perspective and yielding a clearer multimodal representation for prediction. Our testing confirmed that this approach consistently outperformed the last-token method across metrics like NDCG@20, enhancing accuracy without significant computational overhead.\nDue to space constraints, we did not elaborate on this comparison in the original paper. However, we recognize its importance and will include a detailed discussion, along with supporting experimental results, in Section 4 or an appendix of our revised paper to further substantiate this design choice. Thank you for highlighting this point.\n\n\n**[Q4]** Why not concatenate the shared token ($z_{sh}$) at the final output? [ Weaknesses]\n\n**[R4]** We evaluated both approaches—concatenating the shared tokens in the final output and excluding them—and found that preserving only the modality-specific CLOZE tokens yields better performance. The shared tokens effectively align cross-modal features during fusion, but adding them to the final output diluted modality-specific details essential for accurate predictions. We will provide more details on these findings in our revised paper.", "claims_and_evidence": "I found the claims of the authors convincing. However, I will be more convinced of the results if there are online performance evaluations of the proposed algorithm (instead of just offline evaluation). I totally understand that not all researchers(especially on recommendation domain) will have access to online experiments, but since the authors are using private data from AviviD Innovative Multimedia for experiment evaluation, it will be great if there is any online results can be shared.", "methods_and_evaluation_criteria": "The methods makes sense and is easy to follow.", "theoretical_claims": "I checked the correctness of the proposed method and they looked correct to the best of my knowledge.", "experimental_designs_or_analyses": "Comprehensive experiments are conducted on 3 offline datasets(1 is public right now and the author promise to make the other 2 public once the paper is published). The experiment design makes sense.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strength\n\n1.Sequential recommendation is a very well established domain and there are many real world applications that can make the work relevant to a large group of audiences.\n\n2. The proposed method makes sense and is actually straightforward and easy to follow. The innovation is not ground-breaking but more incremental on top of existing/well-established techniques. But for recommendation systems, sometimes a not-complicated but working solution is way more important than an over-complicated algorithm design.\n\n3. The experiments are well designed, comprehensive and include all the analysis and ablation I would like to know as a reader. The offline analysis shows significant gains of the proposed algorithm on top of the existing baselines.\n\n4. The paper is well written and easy to follow\n\n5. The authors promised to release 2 of the offline dataset to the public after the paper is published, which can benefit follow-up researches.\n\n\nWeakness or Questions\n\n1. My biggest concern is the lack of online test results. This is definitely not a reason to reject this paper but this definitely impacts on the confidence of whether this work will really work in real-life settings. In sequential recommendation, there can be a lot of times that offline evaluation results do not match online performance and an online LE can make the conclusion more convincing.\nIf the authors have access to online evaluation, it will be great if they can share some of the learnings.\n\n2. My second concern is that the proposed model will be much larger than many of the baselines. I.e., the proposed model uses an independent transformer block for each modal and has multiple layers on top of it vs baselines like bert4rec which comes with a much smaller model structure. How much of gain is coming from the model size(e.g. the effective size of the model) and how much is coming from the proposed methods?\n\nCan authors provide the parameter size of different models used in the experiment sections?\n\nSome nit questions(not concerns):\n\n3. Why do the authors add a close embedding token (z_cz) for each modality at the end of each sequence instead of directly leveraging the output of the last activity of the sequence? Does this design choice come with gain observed offline?\n\n4. Why the final output only concatenate all the modal specific tokens (z_mod) btu did not concatenate the shared token (z_sh)?", "other_comments_or_suggestions": "Please refer to Other Strengths And Weaknesses section. Thanks!", "questions_for_authors": "Please refer to Other Strengths And Weaknesses section. Thanks!", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lkPoa562sN", "reviewer": "Reviewer_RAjD", "summary": "This paper introduces MTSTRec, a multimodal sequential recommendation model that integrates textual, visual, and price information into a unified, time-aligned shared token representation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We greatly appreciate your recognition of the strengths of our work, particularly your acknowledgment of our conceptual motivation for time-aligned shared token (TST) fusion, the well-structured experiments and ablation studies, and the comprehensive supplementary material. \n\nBelow, we address each of your questions in detail.\n\n\n**[Q1]** Robustness and efficiency evaluations [Questions] [Experimental Designs Or Analyses 2]\n\n**[R1]** Thank you for the suggestion. Robustness to noisy inputs is indeed an important aspect of model reliability. While MTSTRec leverages time-aligned shared tokens to fuse modality-specific features—which may inherently help mitigate localized noise—we did not conduct perturbation experiments in this version due to time constraints.\n\n\n**[Q2]** Comparison of training/inference time [Questions] [Experimental Designs Or Analyses 1]\n\n**[R2]** The following table shows that MTSTRec requires more training time (83 minutes) due to its multimodal inputs (ID, text, image, prompt-text, price) and the TST module (Sec. 3.3.2), but its inference time (12.43 seconds) remains competitive—only slightly higher than SASRec+ (7.48 seconds). In return, MTSTRec achieves significantly better accuracy (NDCG@5 = 0.8942 vs. 0.8150 for SASRec+, Sec. 4.2), offering a strong performance–efficiency trade-off suitable for real-world use. We provided the time complexity analysis. See [R3] for Reviewer 2WPs.\n| Model      | Training Time (minutes) | Inference Time (seconds) |\n|----------------|-----------------------------|------------------------------|\n| MTSTRec        | 83                          | 12.43                        |\n| SASRec       | 0.083                          | 1.860                          |\n| BERT4Rec       | 4.1                         | 1.88                        |\n| SASRec+     | 66                          | 7.48                         |\n| BERT4Rec+   | 59                          | 7.89                         |\n\n\n\n**[Q3]** Visualize TST [Questions]\n\n**[R3]** The attention heatmaps from the MTSTRec model at layer 0 visualize the self-attention weights for a sequence of 21 tokens (18 product IDs padded with 2  zeros to reach the maximum length of 20) across five modalities: token (product ID), style (image), text, prompt-text, and sale price. Each 21x21 heatmap shows how much each token attends to others, with the x-axis and y-axis representing sequence positions (0 to 20). Yellow indicates high attention, and dark blue/purple indicates low attention. The heatmaps reveal modality-specific patterns: token focuses on item identity with sparse attention, style captures visual similarities with broader attention, text and prompt emphasize textual features with vertical stripes, and sale price shows scattered attention, leading to less importance in recommendation. https://anonymous.4open.science/r/MTST_ICML_rebuttal-0E5D/MTST_attention_heatmap.png\n\n\n**[Q4]** Potential of contrastive learning (e.g., CLIP-style loss) to improve TST fusion [Questions]\n\n**[R4]** Integrating a multimodal contrastive loss such as CLIP-style alignment is an intriguing direction, particularly for bridging text–image or text–style embeddings. In principle, a contrastive term might further align the modalities during TST fusion, potentially sharpening the shared token’s cross-modal representation. However, we have not explicitly explored a full-blown contrastive learning design so far, primarily because the supervised next-item prediction objective already forces alignment across modalities that correlate to the same product and also because contrastive training can introduce substantial additional computational overhead. That said, exploring a dual-objective setup—where TST’s latent space is refined by a CLIP-like objective—could be a compelling avenue for future research.\n\n\n**[Q5]** Missing LLM-based multimodal recommendation baselines [Questions] [Experimental Designs Or Analyses 3]\n\n**[R5]** We acknowledge recent progress in LLM-based recommenders such as LLM-Rec. While these approaches inspire our work, MTSTRec is designed for a multimodal sequential recommendation, integrating not only text but also images, prices, and item IDs via our proposed TST module. In contrast, LLM-Rec focuses on text-only scenarios with different modeling objectives. It is worth noting that LLMRec can be seen as analogous to our prompt encoder. Therefore, our ablation studies—comparing with early fusion, late fusion, and using only the prompt encoder—can be viewed as an indirect comparison with LLMRec.\n\n\n**[W1]** Missing related work on mid-fusion strategies [Relation To Broader Scientific Literature]\n\n**[R6]** Thank you for the helpful comments. We would like to clarify that [1] has already been cited in our paper. We agree that [2, 3] are relevant to our mid-fusion design and will include proper citations and discussion in the final version.", "claims_and_evidence": "The claims made in the paper are generally supported by the evidence.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria make sense for the problem at hand.", "theoretical_claims": "The paper does not introduce novel theoretical results but motivates the TST fusion conceptually.", "experimental_designs_or_analyses": "The experiments are well-structured, with ablation studies confirming the importance of each modality.\n\nHowever, robustness and efficiency evaluations are missing:\n- What is the model’s inference speed compared to SASRec?\n- How does the model handle noisy or missing modalities?\n\nThe comparison is limited.\n- LLM-powered recommenders (e.g., LLM-Rec, ACL 2024) should be considered as baselines.", "relation_to_broader_scientific_literature": "- The paper builds upon work in multimodal recommendation (e.g., MMMLP, VBPR) and Transformer-based sequential models (e.g., SASRec, BERT4Rec).\n- The idea of mid-fusion via shared tokens is related to bottleneck attention mechanisms in multimodal learning (e.g., [1]), and these relevant works should be cited.\n- The idea of mid-fusion also relates to Variational Information Bottleneck (e.g., [2, 3]), and these relevant works should be cited.\n\n[1] Nagrani, Arsha, et al. \"Attention bottlenecks for multimodal fusion.\" \n\n[2] Wei, Chunyu, et al. \"Contrastive graph structure learning via information bottleneck for recommendation.\" \n\n[3] Zhao, Wenkuan, et al. \"DVIB: Towards Robust Multimodal Recommender Systems via Variational Information Bottleneck Distillation.\"", "essential_references_not_discussed": "See `Relation To Broader Scientific Literature`.", "other_strengths_and_weaknesses": "See `Questions For Authors`.", "other_comments_or_suggestions": "See `Questions For Authors`.", "questions_for_authors": "1. Introduce random perturbations in images/text to see how robust MTSTRec is to input noise.\n2. Report the comparison of training/inference time.\n3. Can TST be visualized? Showing attention heatmaps of how tokens interact across modalities could strengthen the argument for TST effectiveness.\n4. Would contrastive learning improve TST fusion? Have you considered using multimodal contrastive loss (e.g., CLIP-style loss) to further improve alignment?\n5. Missing LLM-based multimodal recommendation baselines.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HgqqLMywcX", "reviewer": "Reviewer_2WPs", "summary": "This paper proposes a unified multimodal recommendation framework with a Temporally-aligned Shared Token (TST) fusion module to learn cross-modal interactions, ensuring time-consistent alignment and modality fusion. Comprehensive experiments are conducted to compare the framework with existing works and to validate the effectiveness of different modules.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We appreciate your recognition of our work’s clarity, the thoroughness of the experimental evaluation, and a well-written presentation with interesting ideas.\n\nBelow, we address each comment and concern in detail:\n\n\n[W1] Redundancy between TST and positional encoding; unclear role in time alignment [Weakness] [Questions]\n\n[R1] TST aligns features across modalities at each time step, complementing positional encoding’s intra-modality ordering. Together, they ensure robust temporal alignment and fusion. Omitting positional encoding sometimes yields decent results (e.g., NDCG@5: 0.8946 vs. 0.8942, NDCG@10: 0.9105 vs. 0.9086), due to TST’s implicit time handling. However, retaining it ensures fair Transformer-based comparisons and model stability.\n\n\n[W2] No visualized results or case studies to demonstrate the model’s effectiveness. [Weakness] \n\n[R2] Ｗe provided additional visualized results. See our response to Reviewer RAjD’s comment [R3].\n\n\n[W3] Time-complexity analysis [Weakness]\n\n[R3] MTSTRec’s complexity primarily arises from Transformer layers at $O(n^2 \\cdot d)$ per modality, yielding a naive upper bound of $O(m \\cdot n^2 \\cdot d)$. TST adds just one extra token per time step (plus a prediction token), so the overhead is small. With optimized implementations, MTSTRec remains only a minor constant overhead above standard Transformer-based sequential models. See [R2] in Reviewer RAjD.\n\n\n[W4] Comparisons with the latest SOTA methods like DiffMM, etc. [Weakness]\n\n[R4] Our method emphasizes temporal ordering, whereas DiffMM and PromptMM lack timestamps, complicating comparisons. FETTLE operates more as a plug-in than a standalone framework, though we may use its ideas later. Truly multimodal sequential recommenders remain rare; MMMLP (2023) is a notable example. For fairness, we also adapted SASRec and BERT4Rec to handle multimodal inputs.\n\n\n[W5] Unfair comparison due to different input sources across models [Weakness]\n\n[R5] We used prompt-text and price primarily to showcase TST’s mid-fusion scalability rather than any specialized engineering. For fairness, SASRec+ and BERT4Rec+ use matching text and image inputs. Ablations show MTSTRec (without prompt-text) still surpasses SASRec (e.g., 0.8574 vs. 0.8015 NDCG@5), proving TST’s advantage extends beyond additional modalities. Our core contribution is TST’s effective integration of diverse features, further boosted by extra signals.\n\n\n[W6] Clarification on the contribution of multimodal features vs. ID embeddings [Claims And Evidence]\n\n[R6] We acknowledge that ID embeddings are powerful predictors, yet our ablation studies show that removing other modalities—especially Text and Prompt Text—causes noticeable performance drops, as shown in Table 2 (leading to a significant drop in NDCG@5 from 0.8800 to 0.8574). This highlights the collective benefit of multimodal features. \n\n\n[W7] Lack of non-e-commerce datasets for generalizability [Experimental]\n\n[R7] Suitable non-e-commerce datasets for session-based, multimodal recommendations are hard to find, and time constraints limit broader tests. We are releasing our two private e-commerce datasets and plan to explore additional domains in future work.\n\n\n[W8] Ablation only on one dataset [Experimental]\n\n[R8] To address this concern, we conducted additional ablation studies on the House-Hold E-commerce dataset. The results show consistent performance drops when removing the TST module and fusion encoder, confirming the effectiveness and generalizability of our design across datasets.\n| Fusion Method                                  | NDCG@5 | NDCG@10 | HR@5  | HR@10 | MRR@5 | MRR@10 |\n|------------------------------------------------|--------|---------|-------|--------|--------|---------|\n| **MTSTRec**                                    | 0.8942 | 0.9086  | 0.9067 | 0.9358 | 0.8568 | 0.8607  |\n| w/o TST & Fusion Encoder (Late Fusion)         | 0.8773 | 0.8896  | 0.8839 | 0.9116 | 0.8392 | 0.8429  |\n| w/o Multimodal Encoder (Early Fusion)          | 0.8366 | 0.8516  | 0.8404 | 0.8738 | 0.7929 | 0.7974  |\n\n\n[W9] The method extends established Transformer and time-aware models, and additional related works could be considered for a broader context. [Relation To Literature]\n\n[R9] We have cited [1] and [3], and will add [2], [4], [5], and [6] to better position our method. While based on Transformer, our contribution lies in the TST module, which enables time-aligned multimodal fusion—a key difference from prior works [2–4] that lack such fine-grained alignment. Unlike time-aware methods [5,6] that rely on timestamps, we maintain temporal consistency via position encoding and TST alignment. Our use of prompt-enhanced text and thorough ablations further support the novelty.\n\n\n[W10] Over-reliance on appendix [Comments]\n\n[R10] We agree on the importance of self-containment. However, due to the 8-page limit, we had to move several implementation and analysis details to the Appendix.", "claims_and_evidence": "The authors claim that this work is a unified solution for multi-modal recommendation. However, the evidence presented in Table 2 appears to partially undermine this claim. It can be observed that removing the ID embedding leads to the most significant performance degradation while removing other modalities (e.g., style) results in nearly identical performances. For instance, on the Fresh-Food E-commerce dataset, the NDCG@5 for MTSTRec (0.8800±0.0023) and for MTSTRec_W/O_style (0.8784±0.0020) show negligible difference, suggesting that the multi-modal features may not contribute substantially to the model’s overall performance. The above observations raise significant doubts regarding the model’s effectiveness in multi-modal recommendation scenarios. The minimal performance impact observed when removing specific modalities (e.g., style) suggests that the model may not be fully leveraging the potential of multi-modal features. Instead, the heavy reliance on ID embeddings indicates that the model’s success is predominantly driven by single-modal (ID-based) information. I expect that the authors further clarify the main contributions.", "methods_and_evaluation_criteria": "The proposed method is built upon Transformer-like architectures with a TST fusion module. Since transformer has been applied in both academic research and industrial applications within the domain of Recommender Systems (RS), the soundness of the approach is well-established. For criteria, the authors use the commonly adopted metrics (HitRate, NDCG, and MRR) for evaluation, which are consistent with research works from literature.", "theoretical_claims": "N/A. This is an Application-driven ML paper, and no theoretical claim is presented.", "experimental_designs_or_analyses": "I have carefully checked the experimental designs, results, and analyses, and have two major concerns:\n1)\tThe datasets (Fresh-Food E-Commerce, House-Hold E-commerce, and H&M) are all E-Commerce datasets, which is limited in evaluating the model’s generalizability. Datasets in other domains (e.g., MovieLens, Last-FM, Yelp, etc.) should be considered for evaluation. It would greatly strengthen the paper if the above diverse datasets are used for evaluation.\n2)\tThe ablation studies are all conducted on the Fresh-Food E-commerce, which further limits the method’s generalizability. It remains unclear whether the observed performance improvements and the relative importance of each module hold true for the other two datasets.", "relation_to_broader_scientific_literature": "As discussed in Methods And Evaluation Criteria section, the proposed method is built upon Transformer-like architectures with an TST fusion module. The transformer was first proposed in [1]. Since then, many works ([2], [3], [4]) in RS adopted such an architecture. Also, time-aware recommendations are also well-explored [5], [6].\n\n[1] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. NeurIPS 2017.\n[2] de Souza Pereira Moreira G, Rabhi S, Lee J M, et al. Transformers4rec: Bridging the gap between nlp and sequential/session-based recommendation. RecSys 2021.\n[3] Sun F, Liu J, Wu J, et al. BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. CIKM 2019.\n[4] Li C, Xia L, Ren X, et al. Graph transformer for recommendation. SIGIR 2023.\n[5] Lei Wang, Chen Ma, Xian Wu, et al. Causally Debiased Time-aware Recommendation. WWW 2024.\n[6] Qi Zhang, Longbing Cao, et al. Neural time-aware sequential recommendation by jointly modeling preference dynamics and explicit feature couplings. IEEE TNNLS. 2021.", "essential_references_not_discussed": "N/A. The authors have provided a comprehensive literature review in the related work section.", "other_strengths_and_weaknesses": "Strengths:\nS1. The paper is overall well-written with interesting ideas.\nS2. Extensive experimental results are presented.\nS3. The presentation is clear and precise.\n\nWeaknesses:\nW1. The TST module, from my point-of-view, serves as a temporal global feature switch, facilitating interaction between features of different modalities by acting as an intermediate channel. However, the time alignment function of the TST module seems to be duplicated with the positional encoding. In Table 4, the experimental results of different shared token configurations inadvertently undermine the claimed significance of the time-aligned mechanism.\nW2. No visualized results or case studies to intuitively demonstrate the model’s effectiveness.\nW3. Neither time-complexity analysis nor the computational overhead is presented, raising questions in real-world applicability.\nW4. SASRec, BERT4Rec, and MMMLP are all published before 2024. I would like to see comparisons with the latest SOTA methods like DiffMM (https://dl.acm.org/doi/pdf/10.1145/3664647.3681498), PromptMM (https://dl.acm.org/doi/pdf/10.1145/3589334.3645359), and FETTLE (https://dl.acm.org/doi/pdf/10.1145/3626772.3657701).\nW5.  In page 15, the authors claim that the enhanced versions of SASRec and BERT4Rec integrate item ID, text, and image features. However, for MTSTRec, two additional sources are incorporated: prompt-text and price. The unfairness in input sources raises questions about experimental settings.", "other_comments_or_suggestions": "One major suggestion is that the main body of your submission should be self-contained. However, the authors heavily rely on Appendices for details, which makes it inconvenient for readers.", "questions_for_authors": "I would like to hear from authors how the TST module functions when aligning time-aware features, as in my opinion, it serves more like an intermediate channel. Will the MTSTRec experience a significant performance degradation when removing the positional encoding in Equation (4)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IaIYE8xZNf", "reviewer": "Reviewer_AnDS", "summary": "The paper introduces MTSTRec, a transformer-based multimodal recommendation framework that temporally aligns different modalities to improve sequential recommendations. Unlike existing methods, Unlike existing methods that perform either early or late fusion, MTSTRec employs a Time-aligned Shared Token (TST) module for intermediate fusion, ensuring better cross-modal alignment while preserving the unique contributions of each modality. Extensive experiments on multiple datasets demonstrate its superiority over baseline models, with ablation studies highlighting the importance of different modalities and the effectiveness of the TST module.\n\n## update after rebuttal\nSince the authors did not add new results I mentioned in my comments, I will keep the score unchanged.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We are especially grateful for the recognition of the strengths of our work, particularly the innovation of the time-aligned multimodal fusion module and its effectiveness demonstrated through our experiments. We also appreciate your kind comments on the clarity and structure of the writing, as well as the usefulness of the supplementary materials. \n\nBelow, we address each of your comments and suggestions in detail.\n\n\n**[W1]** Insufficient discussion of related work [Weaknesses] [Experimental Designs Or Analyses 3]\n\n**[R1]** Thank you for your helpful suggestion. The mentioned works ([1], [2], [3]) were included in our original related work section, but due to space limitations, we were unable to discuss them in depth. In the revised version, we will enhance this section by more clearly comparing these methods with ours and highlighting the unique aspects of MTSTRec, particularly its time-aligned mid-fusion design. We appreciate your feedback and will further elaborate on these references if space allows in future revisions.\n\n\n**[W2]** lack of multimodal sequential baselines [Weaknesses] [Claims And Evidence 2] [Essential References Not Discussed]\n\n**[R2]** We respectfully contend that the current experimental design adequately supports the evaluation of MTSTRec’s novelty and effectiveness within the defined scope. In Section 4.1 (Experimental Settings), we included baselines such as SASRec and BERT4Rec (single-modal sequential models), enhanced versions SASRec+ and BERT4Rec+ (early fusion with multimodal features), and MMMLP (a state-of-the-art late-fusion multimodal model). Additionally, MMMLP+ incorporates the same multimodal features as MTSTRec, ensuring a fair comparison across fusion strategies. These models, tested on three diverse datasets (Section 4.2), provide a solid benchmark for assessing MTSTRec’s improvements, as evidenced by its superior performance (e.g., NDCG@5 gains of 3.4%-43.7% over baselines).\n\n\n**[W3]** Insufficient discussion and comparison of multimodal fusion strategies, especially mid-fusion [Claims And Evidence 1] [Methods And Evaluation Criteria] [Other Comments Or Suggestions 2]\n\n**[R3]** Thank you for your feedback. We believe the current content adequately supports MTSTRec’s contributions. Section 2.2 outlines early (VBPR), mid (MM-Rec), and late (MMMLP) fusion approaches, aligning with the three fusion types. Our focus is on the novel Time-aligned Shared Token (TST) module, which distinguishes our mid-fusion by temporally aligning modalities—unlike prior mid-fusion works like MM-Rec. Additionally, Table 4 compares various mid-fusion methods (e.g., TST (1:1), TST (1:2), TST (1:4), and bottlenecks), demonstrating TST’s superiority.  \n\n\n**[Q1]** Impact of large language models on textual feature extraction and overall performance [Questions]\n\n**[R4]** Referring to Table 5, larger language models (LLMs) consistently outperform smaller text encoders by capturing more nuanced context, boosting scores on HR@5 and NDCG@5. Despite higher computational overhead, their richer embeddings better align user preferences with item attributes, closing a notable performance gap. (Llama 3.1 excelling in MTSTRec’s text extraction, with NDCG@5: 0.8754, HR@5: 0.8340, outperforming BERT (NDCG@5: 0.8585, HR@5: 0.108) and others. Its richer embeddings improve preference alignment by ~1.97% in NDCG@5)\n\n\n**[Q2]** Comparison with other LLM-based multimodal sequential recommendation models [Questions]\n\n**[R5]** We acknowledge recent progress in LLM-based recommenders such as LLM-Rec. While these approaches inspire our work, MTSTRec is designed for a multimodal sequential recommendation, integrating not only text but also images, prices, and item IDs via our proposed TST module. In contrast, LLM-Rec focuses on text-only scenarios with different modeling objectives. It is worth noting that LLMRec can be seen as analogous to our prompt encoder. Therefore, our ablation studies—comparing with early fusion, late fusion, and using only the prompt encoder—can be viewed as an indirect comparison with LLMRec.", "claims_and_evidence": "The authors provide evidence for the effectiveness of the proposed time-aligned fusion module through ablation studies on multiple datasets. However, two key issues remain:\n1.The discussion on multimodal fusion strategies is limited, particularly regarding mid-fusion, which is central to this work. A more thorough comparison with existing approaches would strengthen the contribution.\n2.The experiments lack baselines from multimodal sequential recommendation models, making it difficult to fully assess the model’s novelty and effectiveness. Including such baselines would provide a clearer evaluation of its improvements.", "methods_and_evaluation_criteria": "Methods: The TST module effectively integrates multimodal features while maintaining temporal consistency, and extensive ablation studies validate the impact of individual components.\nEvaluation Criteria: Multiple datasets are used for evaluation, including one public dataset and two new datasets. However, the three types of multimodal fusion mentioned by the authors should be represented by corresponding baselines.", "theoretical_claims": "The paper provides detailed formulations of the model and the calculation methods for evaluation metrics in both the main text and supplementary materials. However, the theoretical analysis and proof are somewhat limited, with a primary focus on experimental validation.", "experimental_designs_or_analyses": "1.Multiple datasets are used for evaluation, including one public dataset and two new datasets.\n2.The detailed ablation studies effectively demonstrate the impact of individual features and fusion mechanisms.\n3.The paper lacks a thorough comparison with more state-of-the-art multimodal recommendation methods, limiting the context for evaluating the model's novelty.", "relation_to_broader_scientific_literature": "1.Multimodal Fusion: It proposes a time-aligned fusion module, improving the integration of various modalities in recommendation systems, which advances multimodal learning research.\n2.LLM for Feature Extraction and Processing: The work demonstrates the use of large models for extracting text information and handling different feature types, contributing to more effective feature processing in recommendation systems.", "essential_references_not_discussed": "The paper lacks a sufficiently comprehensive discussion of relevant literature on multimodal fusion strategies, including but not limited to the following references, which were either not discussed or mentioned without being compared as baselines:\n[1] Zhou, Xin, and Zhiqi Shen. \"A tale of two graphs: Freezing and denoising graph structures for multimodal recommendation.\" Proceedings of the 31st ACM International Conference on Multimedia. 2023.\n[2] Zhong, Shanshan, et al. \"Mirror Gradient: Towards Robust Multimodal Recommender Systems via Exploring Flat Local Minima.\" Proceedings of the ACM on Web Conference 2024. 2024.\n[3] Jiang, Hao, et al. \"What aspect do you like: Multi-scale time-aware user interest modeling for micro-video recommendation.\" Proceedings of the 28th ACM International conference on Multimedia. 2020.", "other_strengths_and_weaknesses": "Strengths\n1.The time-aligned multimodal fusion module proposed in this paper is highly innovative, and experimental results demonstrate the effectiveness of the token.\n2.The paper provides a valuable analysis of multimodal sequential recommendation, offering insights that inspire further research into the role of multimodal fusion at different stages.\nWeaknesses\n1.The discussion of related work is insufficient, with a lack of thorough literature review. \n2.Additionally, the experimental comparisons with similar models are inadequate, leaving the innovation and effectiveness of the approach open to further validation.", "other_comments_or_suggestions": "1.The writing is clear and flows smoothly, making the paper easy to follow for readers.\n2.The discussion of related work is insufficient, and the experimental comparisons are lacking. More attention should be given to the discussion of multimodal fusion techniques.", "questions_for_authors": "1.How significant is the contribution of large models in extracting text information to the overall performance of the proposed model?\n2.How does the performance of other LLM-based multimodal sequential recommendation models compare to your model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The evaluation from four reviewers reached a borderline. After carefully checking the paper, the reviews, the rebuttal, and the author-reviewer discussions, I think the strong points slightly outweigh the weak points. Thus, it is a weak accept.
Decision: Accept (poster)