DEIM2020 B4-3

Purchase Prediction based on Recurrent Neural Networks
with an Emphasis on Recent User Activities
Quanyu PIAO† , Joo-Young LEE†† , and Tetsuya SAKAI†† Faculty of Science and Engineering, Waseda University
3–4–1 Okubo, Shinjuku-ku, Tokyo 169–8555, Japan†† Wider Planet, Inc.
KFAS Building 8F, 211, Teheran-ro, Gangnam-Gu, Seoul, Korea
E-mail: †quanyu.piao@ruri.waseda.jp, ††jooyoung@widerplanet.com, †††tetsuyasakai@acm.org
Abstract Existing purchase prediction models for E-commerce are often limited in accuracy when they overly
concentrate on speciﬁc user activities. Hence, we propose three methods of aggregating user activities in over a time
sequence in such a way that recent activities are emphasized compared to old ones. The compression of the original
data and reduction of training time is signiﬁcant, and the result shows that the models trained by the Fibonacci
Aggregation give the best comprehensive performance of the three aggregating methods.
Key words Recommendation System, Sequential Data, Machine Learning, Recurrent Neural Networks

Machine (FM) [3] [4] series are also widely used. However,

1 INTRODUCTION

because of the complexity of the user behavior data men-

Advertisement recommendation is an important part in
the E-Commerce ﬁeld. The quality of the recommendation
will highly aﬀect user behaviors.

tioned, all of these methods need careful data analysis andfeature engineering before the model training.

Whether the user will

Neural Network (NN) models have developed rapidly inclick and buy will give a tremendous impact on proﬁts ofrecent years. They can discriminate and select the features

the advertisers. The user behaviors include time informa-

automatically to prevent from complicated feature engineer-tion (timestamp), commodity information (item ID, cate-ing, and ﬁnally provide good prediction and recommenda-

gory, etc.), event (click, buy, etc.). The user behaviors havetion results [5]. Especially, the Recurrent Neural Networks

the following characteristics: Sequentiality, Uncertainty, and(RNN) have great adaptability for sequential data. We be-

Multidimensionality, that lead to challenges for the user be-

lieve that the RNN will also perform well in this task for user

havior prediction task and corresponding recommendations.

behaviors that are strictly sequential.

Sequentiality The events given by the users are strictly sequential so that the prediction models should pay more
attention to the time information.
Uncertainty The user behaviors are complex and changeable. Short and simple click may lead to a buy action;
however, it is also possible for a user buying nothing
after a long time session with many clicks.
Multidimensionality There are a huge number of commodities in the world with diverse categories. Also, users
may have diﬀerent behaviors at diﬀerent times such as
promotion time.
There are many models using classic machine learning
methods such as Random Forest [1] and Gradient Boosting
Decision Tree (GBDT) [2] to predict the user behavior. Recently, some simple prediction models such as Factorization

The present study is based on RNN to predict the buy
action intention of the users and explores the prediction performance in the case where the models are trained by theaggregated data.

2 RELATED WORKIn the RecSys 2015 challenge [6], Wu, et al. [5] proposed an
RNN model to predict the buy session by simply using thetime-stamp of the clickstream data to analyze the item andsession features. Also, Sheil, et al. [7] predicted purchasing
intent by considering the item price variance and doing skip
connections to combine the original input with every RNN
layers. Besides, Gligorijevic, et al. proposed a time-aware
approach [8] to model user behavior which can capture implicit signals of users’ conversion intents.

Pi = i,

3 METHODOLOGY

i = 1, 2, 3, . . .

In Fibonacci Aggregation, the Fibonacci number sequence
3.1 Data Aggregation

can be represented as:

The proposed approach focuses on data preprocessing.
F (i) = F (i − 1) + F (i − 2),

Our starting point is analogous to the way people thinki = 1, 2, 3, . . .

and remember when browsing products in online shopping.
Actually, people do not remember exactly when and whatitems they browsed or bought but only have a rough im-

So, the number of processed activities in every step will
be:


1,
Pi =
F (i),

pression. For example, a user may only remember that hebrowed spoons, chopsticks, and bows yesterday morning, andespecially a stainless steel spoon impressed him a lot. In
this example, the time information is no longer as accurate

i = 1 and 2
i = 3, 4, 5, . . .

Similarly, in the Exponent Aggregation, we choose the baseas 2, therefore the Pi will be:as the data record we have, and the correspondence rela-P i = 2 i−1 ,tionship between time and items has been blurred. Also,i = 1, 2, 3, . . .

the information about the products the users browsed is notcompletely remembered. The farther the behavior is, theless information the users remember. Based on this view,
we propose a data aggregation process that aggregating ascending numbers of activities to one activity in the data preprocessing. Firstly, the user behavior records are too hugeto handle. The aggregation of the original data can summa-ܲ଺ = 2
ܲହ = 5user records have quite diﬀerent lengths, after the aggregation, the overall record length will be signiﬁcantly reduced.
Hence, not only the users with a small number of activities
we can focus, but also can consider the long-activity users.
A session is full of click records by a user, and each record
including this session’s ID (SSID), clicked timestamp, item

Time, Item, Category
…
…
…
…
…

Time, Item, CategoryTim…

rize and compress the data. Secondly, predicting behaviors
for users who have short activity records is a challenge. As

Time, Item, Category

ܲସ = 4
ܲଷ = 3

ܲଶ = 2

ܲଵ = 1

Time, Item, Category
Tim…

Time, Item, Category
Tim…

Time, Item, Category
Tim…

Time, Item, Category
Tim

Time, Item, Category

Time, Item, Category
Tim

Time, Item, Category
Time, Item, Category
Time, Item, Category
Time, Item, Category
Time, Item, Category

Figure 1: Natural Aggregationand its category. We apply our aggregating methods to every session. The aggregating methods are detailed as follows.

3.1.2 Aggregation method in every step

We propose three aggregating methods: Natural Aggrega-

The features in the records may include timestamp, item

tion, Fibonacci Aggregation, and Exponent Aggregation.

ID and item category. In every aggregating step, for those

3.1.1 Aggregation deﬁnitions

numerical features, like timestamp, we get the average of theFirstly, we deﬁne the number of activities we aggregatedfeatures:within one session in every step Pi as:f eature new = avg(
Pi ,∑f eature original )i = 1, 2, 3, . . .
For those non-numerical features, like item ID and cate-

Then,

the

total

numbersofactivities

aggregated

Naggregated can be represented as:∑

step. Note that if some features are correlative, the new fea-

max(i)

N aggregated =gory, we choose the most clicked feature in this aggregatingtures should also be correlative. For example, a most clickedP i,

i = 1, 2, 3, . . .

i=1

Obviously, Naggregated should be smaller than the longest

item and its category should be:f eature newItem = mostClicked(f eature clickedItemList )session in the dataset:f eature newCategory = Category(f eature newItem )N aggregated ⩽ maxLength(session)
For Natural Aggregation showed in Figure 1, we aggregate
numbers of activities in natural number sequence, so Pi willbe:

3.2 Embedding

Table 3: RecSys 2015 dataset buy action sample

Since the advent of Word2Vec [9], embedding has been a(Timestamp column omitted)

common technology in Machine Learning. It can map the

SSID

Item ID

Price Quantity

high dimensional sparse vectors, like one-hot representation,

420374 214537888

01

to low-dimensional dense vectors. Embedding plays a vital

420374 214537850

01role not only in Natural Language Processing (NLP) tasks,
but also entity representation [10] in Knowledge Base tasks.

4.2 Data Preparation

We embed time information, item IDs, and tem categories
in the user behavior records to solve a Recommendation System prediction task.

We use the whole RecSys 2015 dataset. As most sessions
consist of less than 50 activities, we choose the activities
which length are shorter or equal to a suitable value in each

3.3 Long Short-Term Memory (LSTM)

aggregating method (Table 4). We applied padding to ses-

Recurrent Neural Networks (RNN) have a node structurethat can be cycled so that it can handle the sequential input.
And there are inverted sentences in some situations. The Bidirectional RNN (Bi-RNN) models can consider the context
in order and reverse to catch more accurate meanings of thesentences.

sions that are shorter than the value. However, the imbalance between buy and the not-buy session is still an unavoidable problem: only 5% of sessions ending with one or more
buy actions. This makes the purchase prediction, which is apositive sample, quite hard to implement directly. Thus, we
sampled nearly the same number of buy data from the click

However, original RNN is susceptible to the problem ofdata to get a balanced training dataset.

gradient disappearance or explosion, which will stop thetraining or let the network unable to convergence, due to thesimple cell structure. The emergence of Long Short-Term

Table 4: Activity length limit and shrunken percentage in each aggregating method

Memory (LSTM) attempts to alleviate this problem.

Activity

ShrunkenWe utilize an LSTM in our models. As for the Bi-LSTM,

Aggregation

we believe that the bi-directional model can get more user ac-

Natural36

99.60%

tion information from E-Commerce clickstream data to per-Fibonacci

33

99.46%

form better.

Exponent

31

99.35%

4 EXPERIMENTS

Length Limit Percentage

Our task is to predict whether the user will give a pur-4.1 Dataset

chase action or not. We process each property of click and

The RecSys 2015 Challenge dataset [6] published by YOO-

buy data as follows:

CHOOSE provides user clickstream data for training andtesting our prediction models.

It consists of 9.2 million

user sessions. The sessions are anonymous and consist of•

Click data: All properties are unchanged.

•

Buy data: Only session ID, timestamp and item ID area chronological sequence of events describing user behaviors. There are two kinds of user actions in the RecSys 2015
dataset: click and purchase. The click data including session

retained.
•course we retain this during session grouping by the ID).

ID, timestamp, the item user clicked and the item s category,
while the buy data including the session ID and timestamp,

Session ID is discarded in the training and prediction (of•the purchased item ID and its price and quantity.

Timestamps are split into month, day, day of week, hour,
minute, and second. All sessions happened in 2014, so
we discard the year property. And we believe that theTable 1: RecSys 2015 dataset overview
Total

Click

Buy

UniqueUniqueclickssessions

sessions

Item

Category

33,003,944 9,249,729

509,696

52,739339

Table 2: RecSys 2015 dataset click action sample
SSID

Timestamp

Item ID

minute and second features are not critical to the prediction so that we discard them when we are using thedata.
•

Category IDs are unchanged.

•

Target data: For every session in click data, if the ses-Categorysion ID exists in the buy data, this means the session is1

2014-04-07T10:51:09.277Z 214536502

0

a buy session. We assign number 1 to show that this is1

2014-04-07T10:54:09.868Z 214536500

0a buy session. Otherwise, set number 0 to show that thesession not ended with a buy action.

Finally, we group the session into one row, including ses-month: 4

sion ID, session features, and session buy. There are month,

day of week: 3

day: 5day, day of week, hour, minute, second, item ID, and item
category in the session feature column. The item ID andhour: 5

Embedding

L
S
T
M

Sigmoid

Output0.5

Prediction

item: 128
category: 32

item category ﬁeld are recorded as two lookup tables andFigure 2: Model structure with feature embedding

converted to indexes. Table 5 shows the sample after thewidthsbasic feature engineering. Then 6 features (except minuteand second features) are embedded into dense vectors.

compress the original data to aggregated data to save space
usage. Then, using aggregated data can reduce the training
time, especially for complex neural networks.

Table 5: Feature data sample
Featureis buyTable 8 and Table 9, which are the results from the bal-

7

4:2:3:6:38:53:389:0|4:2:3:6:39:5:847:00

anced and original dataset, have a similar shape. Except367

4:7:1:9:42:12:566:0|4:7:1:9:43:14:566:0

1

the results of Fibonacci Aggregation, in each table, the sim-

SSID

ple LSTM models trained by aggregated data have better
4.3 Activity Aggregation

performance in not-buy precision and buy recall, but worse

For the time data including month, day, day of week, hour,

performance in buy precision and not-buy recall than theminute, and second, ﬁrstly we convert to seconds from themodel trained by the unaggregated data. We consider thatyear 1970, then we calculate the average time in an aggre-

the simple models trained by the natural and exponent ag-

gation step as the new timestamp, then restore to the time

gregated data have more tendency to predict user buy action

features. As for the item ID and category, we choose theso that causing the better buy recall result. In the meantime,

most clicked item and its category in this aggregating step.

they lose not-buy recall performance and get a little better

There may remain activities after the aggregation. These

result in not-buy precision.

remaining activities will be processed in the same way men-

However, for the complex models, the results are com-tioned above. Table 4 shows the change of the dataset after

pletely opposite to the simple models’ results. We believe

activity aggregation.

that the models gain more ability to predict not-buy action,

4.4 Model

which causes the results: better performance in not-buy re-In this work, we construct an LSTM model to handle thecall and buy precision, worse performance in not-buy preci-

sequential data. We prepared two LSTM models which aresion and buy recall.

simple and complex. Then every model applied with Bi-

Finally, let’s concentrate on the results of the models

directional feature and set dropout rate as 0.5 during train-

trained by the Fibonacci aggregated data. Compared withing. All structures of the LSTM models as follows:

the other two aggregation methods, the changes are more stable, or, the shape of the results are similar with the models

Table 6: LSTM Model Structures

trained by the unaggregated data: in every evaluate target,

Hidden size Hidden layer

the result is reduced or similar (except buy recall in the sim-

Simple32

1

ple models), with the best ROC AUC values. We believe

Complex128

3that the models trained by the Fibonacci Aggregation givethe best comprehensive performance of the three aggregat-

The last output of the LSTM will be full connected [9] toing methods. With 97.5% - 98.5% of the best performancethe ﬁnal output layer, which is a neuron that provides the(the results trained by unaggregated data), it provides dataprobability of a buy session. We use a sigmoid cell to map

compression and huge training acceleration.

the output to [0, 1] space and set a threshold as 0.5, whichmeans the outputs bigger than 0.5 will be predicted as a buy

Table 7: Data compress rate and training time re-session. Figure 2 shows the overview of the model structure.

duction rate (Approximation)
Dataset4.5 Experiment Results
For the buy prediction task in E-Commerce, we evaluate

Simplethe predictions by measuring buy and not-buy session’s pre-Complexcision, recall, and the ROC AUC score.
Table 7 shows the approximate value of data compress rate
and training time reduction rate. Firstly, we can greatly

Training timecompressed rate reduction rate
20%

15%
50%

Table 8: Results of balanced test dataset (buy: not-buy = 1:1)
Not-buy

Buy

Not-buy

Precision Precision

Buy

RecallRecallROC AUC

Simple0.73300.73870.73630.7354

Simple with Natural Aggregation

0.66450.77310.82470.58920.73580.7070

Simple with Fibonacci Aggregation

0.70860.73840.75280.69270.7228

Simple with Exponent Aggregation

0.64190.7816

0.84920.53260.69090.7322Complex0.71780.74760.75850.7058Complex with Natural Aggregation

0.74140.6676

0.5994

0.79370.6965Complex with Fibonacci Aggregation

0.70290.72700.73900.68990.7145Complex with Exponent Aggregation

0.73240.68170.63520.77090.7031

Table 9: Results of original test dataset (buy: not-buy = 1:18)
Not-buy

Buy

Not-buy

Precision Precision

Buy

RecallRecallROC AUC

Simple0.13750.97950.73390.73430.7341

Simple with Natural Aggregation

0.10390.98290.82210.59060.7064

Simple with Fibonacci Aggregation

0.12490.97970.75030.69590.7231

Simple with Exponent Aggregation

0.09570.98420.85280.53280.6928Complex0.12950.98060.75820.70580.7320Complex with Natural Aggregation

0.14290.97140.59480.79410.6945Complex with Fibonacci Aggregation

0.12160.97860.73850.69150.7150Complex with Exponent Aggregation

0.13900.9734

0.63680.77130.7040real session length. The padding blocks may interfere

5 CONCLUSIONSIn this work, we proposed three kinds of data aggregation
methods: Natural, Fibonacci, and Exponent Aggregation.
Then we investigated and discussed the eﬀects made by theaggregating of the original data. Firstly, the result showswith the training process.
•Some of the features, such as item IDs, may need more
analysis to convert the original features to some moremeaningful features.

that the models trained by balanced and unbalanced original

We consider that we should pay more attention to data

dataset have similar result shape. Then, except Fibonacci

analysis and feature crafting. Besides, advanced and com-

Aggregation, for simple LSTM model, the models perform

plex neural networks is also a research point. Moreover, we

better in not-buy precision and buy recall, but worse per-should propose better solutions to handle more general andformance in buy precision and not-buy recall. However, forimbalanced E-Commerce data in the future.

complex LSTM model, the results are completely opposite.
On the other hand, the Fibonacci Aggregation performs sim-

ACKNOWLEDGMENTS

ilar with the results trained by unaggregated data. It gives

Special thanks to all of the members in Real Sakai Lab.

the best comprehensive performance of the three aggregat-

Thanks for their important suggestions and guidance. Alsoing methods. The method provides data compressing andspecial thanks to Nao Toyama, who always gives me emo-huge training acceleration with 97.5% - 98.5% of the best

tional support.

performance.
During the experiments, the biggest problem is data overﬁtting. We had to early stop the training process to get thebest training results. However, all of the ROC AUC scores
are not satisﬁed, which means that the ability of models is
unsatisfactory. The problems may be as follows:•

The models were not trained enough. Probably there aresome problems with the model and hyperparameters.

•

We did not apply dynamic RNN to deal with diﬀerent

References[1] Breiman, Leo. “Random Forests.” Machine Learning 45(2001): 5-32.
[2] Friedman, Jerome H.. “Greedy function approximation: A
gradient boosting machine.” (2001).
[3] Rendle, Steﬀen. “Factorization Machines.” 2010 IEEE International Conference on Data Mining (2010): 995-1000.
[4] Rendle, Steﬀen. “Factorization Machines with libFM.”
ACM TIST 3 (2012): 57:1-57:22.
[5] Wu, Zhenzhou et al. “Neural Modeling of Buying Behaviourfor E-Commerce from Clicking Patterns.” RecSys ’15 Challenge (2015).

[6] Ben-Shimon, David et al. “RecSys Challenge 2015 and the
YOOCHOOSE Dataset.” RecSys ’15 (2015).
[7] Sheil, Humphrey et al. “Predicting Purchasing Intent: Automatic Feature Learning using Recurrent Neural Networks.” ArXiv abs/1807.08207 (2018): n. pag.
[8] Gligorijevic, Djordje et al. “Time-Aware Prospective Modeling of Users for Online Display Advertising.” ArXiv
abs/1911.05100 (2019): n. pag.

[9] Mikolov, Tomas et al. “Eﬃcient Estimation of Word Representations in Vector Space.” CoRR abs/1301.3781 (2013):
n. pag.
[10] Liu Zhiyuan,Sun Maosong,Lin Yankai, et al. “Knowledge
Representation Learning: A Review[J].” Journal of Computer Research and Development, 2016, 53(2): 247-261.