Constraining Latent Space to Improve Deep Self-Supervised e-Commerce Products Embeddings for Downstream Tasks
Authors: Cristian Cardellino
Rafael Carrascosa
Venue:  (2021)
Keywords: representation learning
deep learning
self-supervised learning

Abstract: The representation of products in a e-commerce marketplace is a key aspect to be exploited when trying to improve the user experience on the site.  A well known example of the importance of a good product representation are tasks such as product search or product recommendation. There is however a multitude of lesser known tasks relevant to the business, examples are the detection of counterfeit items, the estimation of package sizes or the categorization of products, among others. It is in this setting that good vector representations of products that can be reused on different tasks are very valuable. Past years have seen a major increase in research in the area of latent representations for products in e-Commerce. Examples of this are models like Prod2Vec or Meta-Prod2Vec which leverage from the information of a user session in order to generate vectors of the products that can be used in product recommendations. This work proposes a novel deep encoder model for learning product embeddings to be applied in several downstream tasks. The model uses pairs of products that appear together in a browsing session of the users and adds a proximity constraint to the final latent space in order to project the embeddings of similar products close to each other. This has a regularization effect which gives better features representations to use across multiple downstream tasks, we explore such effect in our experimentation by assessing its impact on the performance of the tasks. Our experiments show effectiveness in transfer learning scenarios comparable to several industrial baselines.

Under review as a conference paper at ICLR 2021
CONSTRAINING LATENT SPACE TO IMPROVE DEEP SELF-SUPERVISER E-COMMERCE PRODUCTS EMBED- DINGS FOR DOWNSTREAM TASKS
Anonymous authors Paper under double-blind review
ABSTRACT
The representation of products in e-commerce marketplace is key aspect be exploited when trying to improve the user experience on the site. A well known example of the importance of good product representation are tasks such as prod- uct search or product recommendation. There is however multitude of lesser known tasks relevant to the business, examples are the detection of counterfeit items, the estimation of package sizes or the categorization of products, among others.
this setting that good vector representations of products that can be reused on different tasks very valuable. Past years have seen major increase in research in the area of latent representations for products in c-Commerce. Ex- amples of this are models like Prod2Vec Meta-Prod2Vee which leverage from the information of a user session order to generate vectors of the products that can be used in product recommendations. This work proposes novel deep coder model for learning product embeddings to be applied in several downstream tasks. The model uses pairs of products that appear together in browsing session of the users and adds a proximity constraint to the final latent space in order to project the embeddings of similar products close to each other. This has regu- larization effect which gives better features representations to use across multiple downstream tasks, we explore such effect in our experimentation by assessing its impact on the performance of the tasks. Our experiments show effectiveness in transfer learning scenarios comparable to several industrial baselines.
INTRODUCTION
The e-Commerce environment has been growing at a fast rate in recent years. As such, new tasks propose new challenges to be resolved. Some key tasks like product search and recommendation usually have large amounts of data available and dedicated teams to work on them. On the other hand, some lesser known but still valuable tasks have less quality annotated data available and the main goal is to resolve them with small investment. Examples of the latter are counterfei/fforbidden product detection, package size estimation, etc. For these scenarios, the use of complex systems is discouraged in favor of industry proven baselines like bag-of- Bag-of-words or fastText (Joulin et al. 2016)
In particular, with the advent of "Feature Stores" et al. 2017), industrial applications seeing rise in the adoption of organization-wizn representations of business entities (customers, products, etc.). These are needed in order speed up the process of building machine learning pipelines to enable both batch training and real-time predictions with as low effort as possible.
In the present work we explore the representation learning of marketplace products to apply in downstream tasks. More specifically we aim to train an encoder that that can transform products into embeddings to used as features a linear classifier for specific task, thus avoiding feature engineering for the task. The encoder model training is done is self-supervised fashion by leverag- ing browsing session data of users in our marketplace. Using product metadata and an architecture inspired the recent work of Grill et al. (2020), explore how the use of pairs of products in session can enable transfer learning into several downstream tasks. As we discuss further in Section 3 we extend on the work of Grill et al. (2020) with new objective function that combines their

---

Under review as a conference paper at ICLR 2021
original idea with cross entropy objective. Our experiments show that the added objective helps the model converge to better representations for our tasks.
We also show, through experimental evaluation, that the encoder model learns good representations that achieve comparable results with several strong baselines including fastText (Joulin et al. 2016),
Meta- Meta-Prod2Ver Vec (Vasile al. 2016), Text Convolutional Networks (Kim 2014) and BERT (Devlin
et al. 2018) in a set of downstream tasks that come from some of our industrial datasets.
This paper is structured follows: Section presents other works in the area of product represen- tation and also the works we take inspiration to design the encoder model and establish how our approach differs from the previous literature. Section 3 describes in detail all the components of our proposed architecture. Section 4 lists our experimental evaluation setup. Section 5 shows the results of our experimentation. Finally, in Section 6 we summarize our findings and delimit our line of future work.
2 BACKGROUND
Recent years have seen a dramatic increase of latent representations, which have proven to be more relevant in transfer learning scenarios in Computer Vision with the aid of large pre-trained models (Raina et al. 2007 Huh et al. 2016); and, more recently, with the aid of architectures for train- ing unsupervised language models like LSTMs (Merity et al.) (017)) or the attention mechanism (Vaswani et al. 2017), transfer learning has seen an explosion of applications in Natural Language Processing (Howard& & Ruder 2018, Devlin al., 2018, Radford al. 2018). For the case of the e-commerce environment, there is extensive research work in the area of latent representation for some of the main tasks.
In the area of recommender systems there is a very large body of work in which the idea is to use information of the user shopping session to generate latent representations of the products. The Veod2Vec algorithm (Grbovic et al. (215)) proposed the use of word2vec (Mikolov et al. 2013) in a sequence of product receipts coming from emails. The Meta-Prod2 Vec algorithm (Vasile et al.) 2016) extended upon Prod2 Vec by adding information on the metadata of user shopping session.
Using metadata of the products during a stream of user clicks is explored with the aid of parallel recurrent neural networks (Hidasi et al., 2016) where the authors use images and text of an product to expand in order to have richer features to model the products in the session.
Other works that uses more metadata, in this (Zheng et al. 2017), which consists of two parallel neural networks coupled in the last layers. One network learns user behaviour and the other learns product properties, based on the reviews written.
the user review of product, is DeepCoNN
There is also work in the area of modelling information on session-aware recommender systems [Twardowski] 2016), where the user information not present and the focus of the task is leaning towards using the session information to recommend products.
This extensive research of representation learning for marketplace products is heavily influenced with the end goal of recommendation. Many of them also leverage from the unsupervised informa- tion available such sessions, reviews, metadata, etc.
In this work the end goal of the representations is not recommendations, but different downstream tasks that we have available from challenges we face in our marketplace. For that we propose deep encoder architecture that follows the work presented in "Bootstrap Your Own Latent" (BYOL) (Grill et al., 2020) with the intended objective of learning embeddings of products of the same session close to each other in the latent space. However, our experiments showed that this was not enough to ensure the transfer of knowledge, as such we extended the learning objective of BYOL to have a cross entropy objective using the product category as target and we explore how the correct combination of each part of the objective function impacts on the quality of the final embeddings.
The main contributions of our paper are the following: 1) novel deep encoder architecture that can be trained on pairs of products found in user browsing sessions, 2) a study of how this architecture performs for downstream tasks compared to some strong proposed baselines, 3) extension to the BYOL architecture to a different domain from the proposed by Grill et al. (2020) and how it impacts on the final results.
2

---

Under review as a conference paper at ICLR 2021
3 METHOD
3.1 BROWSING SESSION DATA
The data needed to train our Product Embeddings are browsing sessions of different users in the marketplace. Each product user visits (i.e. check the details) is part of the browsing session of the user. A session ends when at least T minutes have passed without new visits.
More formally, given sequence of products the timestamp of the accessed product, we have that 1 T(p)) S. ' for a fixed time window T in minutes. Each product pj in the session is represented by two attributes: the title {p,)) and the category c(p)). The title of the product is written by the user of the marketplace. It contains brief description of the product with some information such as brand, model, measures, etc. It depends on the product itself, and the same product can appear with different titles. Some products have an associated id to identify that they are the same product sold by different users, but that is not always the case. As the language of our marketplace is Spanish, the titles are normalized by stripping accents, removing stopwords and punctuation, lowercase The category of the product is selected by the user from list of fixed categories. These make both attributes susceptible to noise as are driven by the user's 's actions (e.g. are spelling errors on the title missing categories for one-of-a-kind products).
(P1:--Ph)) Pn)) where p; is an product and T(p:) is
words and whitespace normalization.
We use the titles of these products as their appear in the sessions to build sentence piece tokenizer (SPM) (Kudo & Richardson 2018). This same tokenizer is applied to the titles reduce out-of- vocabulary related issues. The titles are finally represented to the model as sequence of sub-word tokens from the SPM.
We see all the possible combinations without replacement with two products between all the prod- ucts of a session. Each product pair is what we feed our model. The main concept here that we are interested in products that are part of the same session, regardless of the order they were accessed by the user. We also remove all duplicate products in single session (since is common for the user to access the same product more than once in the time frame).
3.2 PROPOSED MODEL
We are interested in finding an encoder to map the product title latent space that minimizes the distance of embeddings of a product in the same session while maximizes the distance between products of different sessions. Once have the encoder function, it is used on the product titles of downstream tasks where the session information is not available. We transfer knowledge from the sessions into the tasks and train linear model for the task.
More formally, given an encoding function fo we use the information of the browsing session to calculate the parameters 0. Then given a task dataset S = (111./1),, (Ing Mn)]. use fe to encode {f( .... (xx,,) and use those features to train linear classifier Iw on the weights w while freezing the weights
3.2.1 BOOTSTRAP YOUR OWN LATENT
To constraint the latent space in order to minimize the distance between products of the same session while maximizing the distance with products of other sessions we originally aim the idea of unsupervised visual representation learning methods such as SimCLR (Chen et al., (2200) and MoCo (He al. 2020). Both these methods rely on contrasting positive and negative examples in order to learn better representations, finding good negative examples in our environment was non-trivial. We then came across "Boostrap Your Own Latent" (BYOL) (Grill et al., 2020) which expanded the previous architectures with interesting concept that didn't 't require the explicit sampling of negative examples and we based our architecture on
10/ We need to point out that according to http://untiiledaa..cco understandee anding- self suppeupiiiise www.--uuprriieddddeeeee cont rast www.eeeeerrann.... html the reason why BYOL works is an implicit contrastive learning given by the use of batch normalization. We haven't ' done an extensive research but some preliminary experiments did show that removing batch normalization had a negative effect the overall performance of the representations.
3

---

Under review as a conference paper at ICLR 2021
product
790 eroooddding
roprosematton
propetioon
prodiation
(r(+)
Marros
Q
A
V
*9(2)
target
oxlegory
Bp
Les
The model's architecture is based BYOL (Grill et 2020 The model receives as input pair of products and pj. One product is projected through the online network until obtained the projector value qo(2). The other product is projected through the targer network until obtained the value 2/. The projection of p, is also used to obtain the predicted category Cp, of Pj with aid of the category predictor hg. The model is trained on the weights by minimizing the objective functions B. BYO. which is the distance between and sg(=') (where sg means stop-gradient) and
Figure 1:
the cross entropy loss LCB between the predicted category čp, and the category Cp5 of pj.
3.2.2 MODEL'S S ARCHITECTURE
Figure shows the model architecture. The core idea of the model to learn representation y that can be used for downstream tasks. As it is shown in the Figure, the base of our architecture is heavily inspired by the model proposed in Grill al. (2020).
In the original architectures of SimCLR, MoCo and BYOL, the input of the model is single image that is augmented with different views it. This the self self-supervision given by image itself. In our scenario, our main hypothesis is that different products of the same browsing session can be thought as different augmentations of the same session.
In our original set of experiments, the products were solely represented by their title. The generation of the representation depends on the encoder we apply to obtain a feature vector of the title of the product. In our setup this encoder is the Text Convolutional Neural Network of Kim) (2014) as experiments show that the nature of these networks function very well with the type of text data we have.
More formally, given a pair of products that are part of the same session: a key product p. and a query product Pj. the BYOL model consists two paths: the online network and the target network. The online network defined by a set of weights and is comprised of five stages: an embedding lookup tp. representation encoder fe (in our
Text CNN), a projector go. and a predictor qa-
The target network has a very similar architecture to that of the online network, except for the predictor qo. it also has different set of weights ç. The weights of the target network are optimized with the data, but are obtained by a slow exponential moving average of the online pa- rameters 0, given target decay rate [0, 1], after each training step the weights are updated: & + T + (1 - T 0. Both 0 and & are initialized to the same values when creating the networks.
After training the model, we interested in the encoder fo that gives us the feature vector y to use on downstream tasks. The rest of the model is discarded.
3.2.3 BYOL OBJECTIVE
The approach proposed by Grill et al. (2020) is to train the online network by minimizing the distance between the prediction do and the projection z' of the target network. As seen in Figure with the
4

---

Under review as a conference paper at ICLR 2021
of the coap-grrdient function "sg", the idea is to avoid optimizing over the weights of the target network. This is to avoid the collapse given by the trivial solution of output a constant value for the embedded space. The weights & of the target network obtained instead by slow-moving average the weights 0 of the online network.
From the key product ps, the online network outputs representation y fo(to(p;)) and projection 2 = (o()). The target network outputs from the query product Pj the target representation y' (p,)) and the target projection z' 9e (y))). In the final stage, the online network outputs prediction qo (=) of z'. The final step is to take the loss between the Lz-normalized versions of the predictions (z(2) and the target projections
(10(2); =²) [190 2 1|
LBYOL =
(1)
3.2.4 CATEGORY PREDICTION
In our original setup we explored the use of the BYOL optimization objective alone. However we found it difficult in some scenarios to really achieve knowledge transfer to some of the tasks had. As such, we decided to take advantage of some of the others meta-date available for the products. In particular, for this work data available at our marketplace.
using the product category which is one of the most common meta
Inspired by the idea of the original BYOL architecture. In this case, we add new predictor he that maps the representation of the key product ps the category Cp; of the query product pj: Ep = (ho o fo (p:)). Then optimize the cross entropy loss:
(Vasile et al. 2016) added extra optimization step to
ICc log
Σ El
3.2.5 FINAL OPTIMIZATION OBJECTIVE
Given that the losses in Eq. D and Eq. only refer one way of the product pair Pi, we can make the final loss of the model symmetric by swapping the key and the query products, and give them as input to the networks using Pj as the key and p; the query:
L = (LBYOL,, + LBYOL,, (LCE,, LCEp,)
(3)
3.2.6 IMPORTANCE OF EACH OBJECTIVE
The question that arises is how important each optimization objective is to the overall performance of the representations learnt. A first approximation is to sum them in Eq. However, experiments show that this is not the best case. Moreover, one hypothesis we originally had that the transfer of knowledge was solely obtained because added the cross entropy objective Leg- Our experimental results however show different story. To see the importance of each part of the loss in the final performance, we added an extra hyperparameter [0, I] that weights the final objective in a complementary fashion:
L = == + LBYOL,, + a)) + LCEp,)
(4)
Successive experiments showed that at larger values of a, with more focus the LBYOL ob- jective we had better representations for the downstream tasks. However, a takes the value of 1, completely cancelling the influence of L.CE, the representations cannot achieve good results.
5

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTAL EVALUATION
4.1 DATASETS
The experiments need two kind of datasets: one for the self-supervised training of the Product Embeddings and the others for the downstream tasks. For better validation of our hypothesis, we experimented with data coming from two countries where our marketplace is present. Each country has it own sets of session data and downstream tasks data, although the objective is the same, the data distribution not. For the scope of this work we present 4 binary classification tasks:
Counterfeit Products A dataset based on the users reported counterfeit objects that are being sold in marketplace.
Forbidden Products A dataset of products that are prohibited to commercialize in our marketplace.
Free Shipment Eligible A dataset with products sizes (height, length and width). The task is to de- termine the product eligible for free shipment, this happens when all of the dimensions are smaller than 70 cm.
Product Condition A binary classification to check if the product is new or used.
The tasks datasets were divided into train, test and validation with a 60/20/20 split. The validation set for hyperparameter tuning and the final results are taken from the test set. We are interested in how the model works when there is limited data availability. For this we run experiments on subsamples of the train data, using 100, 500, 1000, 5000, 10000, 50000 and 100000 randomly chosen instances. To assess the impact of selecting the data, we run experiments on 5 samples for each training size. We report the results of mean and standard deviation for each training sample on each task. These tasks represented by the title and the label of the task. The titles are pre pre-processed with the same text normalization as for the sessions datasct.
Note the data Due the sensitive nature of these datasets, we working on the anonymiza- tion and future release of them. We cannot guarantee the availability at reviewing time, but we are doing our best efforts order to have them available along the code to reproduce the experiments for the camera ready.
4.2 MODELS
The supersupe encoder feots used to extract the representations of each of the tasks products and we use those representations with logistic regression classifier. We use the validation data to select the a value (we assess the impact using 5 values of a: O, 0.1, 0.5, 0.9 and 1). Due to space limitations we leave the detailed hyperparameters for the reader
in Appendix A.
We compare the with the following baselines: regression, and Meta-Prod2Vee with K Nearest Neighbors. For more detailed explanation of how the baselines are applied please refer to Appendix A.2
fastText, Text CNN, Spanish BERT with logistic
The metric used for comparison of performance is the Area Under ROC (or AUCROC), which measures the performance of the classifier being independent of the threshold.
5 RESULTS AND DISCUSSIONS
5.1 IMPACT OF THE LOSSES IN THE FINAL REPRESENTATION
Figure 2 shows the learning curves that measure the impact of the selection of the a hyperparameter that controls the importance of each of the losses in Eq. Each graphs represents a task for each site. The x axis represents the size the training set and the y axis represents the the performance on the validation set measured by AUCROC. The graphs on the left represent Site A and the ones on the right represent of Site B. The color intensity and the style of the line represents the value of a. A higher value of is represented by darker shade of blue. As the pattern is clear, the better performance is for the value of a of 0.9 specially at very low training samples when the curve is more steep for cases of a equal to 0.9. However when reach the value
can observer, in almost all plots

---

Under review as a conference paper at ICLR 2021
Alpha Hyperparameter Selection
Sipmes Bigh
Bytte.
Cosendert Pod Dietecttonn Sie
Corsterigg Produt Detrctie
Figure 2: Impact weighting each loss differently in the performance of the models for the down- stream tasks. The darker the color, the higher the For the black line. the encoder is random (i.c. no self supervised training). The æ axis represents the amount of training data. The y represents the AUCROC at different training sizes.
of a equal to one the model collapses. For comparison sake we also plot the value a random encoder (i.e. not trained with the self-supervised task), represented by the black line. It is to notice that some cases, a lower value of a or a value of equal to 1, the model performs worse than random. This results support the hypothesis that both LBYor. and LoE are essential for the model to transfer knowledge, and showcase the importance LYYO which is the part of the objective function in charge of shortening the distance between products of the same session.
5.2 COMPARISON WITH BASELINES
Figure presents the comparison of our model (named "Product Embeddings" in the plot) to the baselines defined in previous sections. Similarly to the case of Fig. 2 the x axis represents the size of the training set and the y axis represents the performance measured by AUCROC. In this case, however, the performance is measured over the test set of each task. Each line of different color represents the model. The shade around the lines represents the confidence interval since each 5 different subsamples of each training size. The plots show clear advantage of
model was more complex models like TexICNN and Spanish BERT, specially as the amount of data available increases. However model does not fall behind, specially since the tuning of hyperparameters

---

Under review as a conference paper at ICLR 2021
Downstream Tasks Results Comparison
Shpmes Bigh 5t:
ptemett Bybt.
Garraffe Pfdd St
Counterit Podud Dessttonn
Sevel 187 Sahhol
CondacC Sleck
Product Conditon 2bt
Figure 3: Results of the experiments. Each graph represents the performance comparison in different downstream tasks. The different models are presented the different colors. The x axis represents the amount of training data. The y axis, depending on the task, represents results using different metrics. The axis depends on the task, represents the metric associated with the task.
was minimal. This is important since TexICNN requires GPU for training each task, and BERT require GPU at inference time. Our model was trained on GPU for the self-supervised task, the downstream task the model is only used for inference and does not require GPU to do On the other hand the model outperforms both Vec is particularly important since our model achieves better results by only encoding based on the title, without the need of generating one embedding per product nor any other metadata at inference time.
and specially fastText. The case of Meta-
CONCLUSIONS
In this work we presented novel deep encoder model trained on self-supervised fashion from a marketplace environment and used feature extractor for downstream tasks. We extended the work of Grill et al. (2020) to a new domain and proved that, with some adjustments, is a feasible solution worth exploring. The model shows good performance in comparison to some industrial baselines with advantages over many of them. Future work will include the exploration of different learning objectives with aid of other metadata present in the items.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
José Cañete, Gabriel Chaperon, Rodrigo Fuentes, and Jorge Pérez. Spanish pre-trained bert model and evaluation data. In to appear in PMIADC ICLR 2020, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020.
Jacob Devlin, Ming-Wet Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018.
Rong-En En Fan, Kai-Wei Chang. Cho-Jui Hsieh, Xiang-Rut Rui Wang, and Chih-Jen Lin. Liblinear: A library for large linear classification. J. Much. Learn. Res., 9:1871-1874, June 2008. ISSN 1532- 4435.
Mihajlo Grbovic, Vladan Radosavljevic, Nemanja Djuric, Narayan Bhamidipati, Jaikit Savla, Varun Bhagwan, and Doug Sharp. E-commerce in your inbox: Product recommendations at scale. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '15, pp. 1809-1818, New York, NY. USA, 2015. Association for Computing Machinery. ISBN 9781450336642.
doi: 00.11452783255.2788666. URL https://doi.
org/10 11e11114577388 /t////2777228888888888 2788627
Lean-Bastion Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning, 2020.
Balázs Hidasi, Massimo Quadrana, Alexandros Karatzoglou, and Domonkos Tikk. Parallel recurrent neural network architectures for feature-rich bession-based recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems, RecSys '16, pp. 241-248. New York, NY, USA, 2016. Association for Computing Machinery. ISBN 978 7781450340339 doi: 10.1145/
2959100.2995 29591697 URL. htttts///:////t htt///oi.org//.../// 1145/ 9591 00. 2 2959167
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification, 2018.
Minyoung Huh, Pulkit Agrawal, and Alexei A. Efros. What makes imagenet good for transfer learning?, 2016.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on In- ternational Conference on Machine Learning Volume 37, ICML' 15, pp. 448-456 JMLR.org, 2015.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint 1607.01677 2016.
Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1746- 1751, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/ D14-1181. URL.ttts:
www aclweb. org/ httpr///annhooloyolooo-o--
Taku Kudo and John Richardson. SentencePiece: A simple and language independent sub- word tokenizer and detokenizer for neural text processing. In Proceedingy of the 2018 Con- ference on Empirical Methods in Natural Language Processing: System Demonstrations, pp- 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: (11633/11/D182220
URL https::t//// www aclweb. org/ www.g//tttoggg///a...o.oo 2012
Li Erran Li, Eric Chen, Jeremy Hermann, Pusheng Zhang, and Luming Wang. Scaling machine learning service. volume 67 of Proceedings of Machine Learning Research, pp. 14-29, Microsoft NERD, Boston, USA, 11-12 Oct 2017. PMLR. URL http: /tte:://pae http::////ooceednng.....
m w 1...prres//v7/1177.2 html

---

Under review as a conference paper at ICLR 2021
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing Istm lan- guage models, 2017.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space, 2013.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning. 10, pp. 807-814, Madison, WI, USA, 2010. Omnipress. ISBN 3781605589077.
Pedregosa, G. Varoquaux, Gramfort, V. Michel, Thirion, O. Grisel, Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg. J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan guage models are unsupervised multitask learners. 2018. URL /tllitttps::///ccpppkee
Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer, and Andrew Y. Ng. Self-taught learn-
ing: Transfer learning from unlabeled data. In Proceedings of the 24th International Confer-
ence on Machine Learning. ICML '07, pp. 759-766, New York, NY, USA, 2007. Associa- tion for Computing Machinery. ISBN 9781 9781595937933. doi: 10.1145/1273496 1273592.. URL
https://doi. www.:0...r011..1 http://..111112222443113 1273592
Bartlomiej Twardowski. Modelling contextual information in session-aware recommender systems with neural networks. In Proceedings of the 10th ACM Conference Recommender Systems, RecSys 16. pp. 273-276, New York, NY. USA, 2016. Association for Computing Machin- ery. ISBN 978 9781450340359, doi: 01145/2999100.2959166
BtLhhtts: htt:://ooi.ooo org/10.
1145/2959100 2ttp/29955100.22551122 2959162]
Flavian Vasile, Elena Smimova, and Alexis Conneau. Meta-prod2vec: Product embeddings us- ing side-information for recommendation. In Proceedings of the 10th ACM Conference on Recommender Systems, RecSys '16, pp. 225-232, New York, NY. USA, 2016. Association for Computing Machinery. ISBN 978 4503445333
doi: 10.1145/29511002295911) URL
https::/doi.on ort/10..rg/1.... 11 1441 45 2055225911000 Twt.:59911..999999.. 2959160]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need. In Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 5998-6008. Curran Associates, Inc., 2017. URL nips. cc/ wwtpccpppppppeeeeeeeeeeeeeeeeeee wttppppe//iiaatttnniooo....... -o---o----eo-l----------eeeeee pdf
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue. Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's 's transformers: State- of-the-art natural language processing. ArXiv, 1 2019.
Lei Zheng, Vahid Noroozi, and Philip Yu. Joint deep modeling of users and items using re- views for recommendation. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, WSDM '17, pp. 425-434, New York, NY, USA, 2017, Associ- ation for Computing Machinery. ISBN 9781450346757. doi: 10.1145/3018661. 3011 URL
A EXPERIMENTAL SETUP HYPERPARAMETERS
A.1 ARCHITECTURE AND MODEL TRAINING
The selected architecture was an embedding lookup to for 30,000 tokens of the vocabulary of the sentence piece tokenizer and dimension 512.
10

---

Under review as a conference paper at ICLR 2021
The representation encoder fo is Text CNN with 4 kernels of size 2, 3, 4 and 5 tokens (as these ID CNN), a total of 128 filters for each kernel size with a global max pooling operation. The output dimension of the encoder is 512.
The projection encoder go is a multi-layer perceptron (MLP) that takes the 512 and expands it to a hidden layer of size 1025, followed by batch normalization loffe & Szegedy (2055) with a decay rate of 0.9, rectified linear unit Nair & Hinton (2010) and a final linear layer of size 128. The predictor qe uses the same architecture of the projector. Similarly, the category predictor he follows the same architecture previously defined, except for the final linear layer which will project to the cardinality of the set of product categories. The exponential moving average of the target networks is updated with an initial T of 0.996, and following the update defined in Grill et al. (2020).
As we have two datasets of sessions, one for each country, we trained two models. In both cases the models were trained by running one epoch over the whole dataset. The datasets contained 12,421,493 pairs of products for site A and 7,978,613 for site B. Those pairs of items were obtained from set of 300,000 users sessions for each site. To avoid a combinatorial explosion, we limited the pairs in each session by truncating each session to a maximum of 10 products. This however resulted in some problems when trying to directly compare to Meta-Prod2Vees as for the latter we use directly the full 300,000 sessions datasets. It is something to address in future releases. The batch size was of 4096. We used Stochastic Gradient Descent with learning rate of 0.45, a momentum of 0.9 and a weight decay of le-5. The learning rate was scaled to the batch size following the parameters given in Grill et al. (020), well as Cosine Anealing rating schedule as established by the BYOL work.
With the encoder model we trained a Logistic Regression classifier for maximum of 1000 iterations and the liblinear ver Fan et al. (2008) of the implementation in the Scikit-Learn Pedregosa et al. 1 package (we leave the rest of the parameters in it's default values).
A.2 BASELINES
The detailed baselines are the following:
fastText For classification tasks use fastText's text categorization Joulin et al. (2016) engine. This is the main baseline working for that kind of downstream tasks in production environ- ments.
Bag-of-Words LR We did some preliminary experiments using bag-of-words with linear clas- sifier (i.e. logistic regression) but we decided to discard it as consistently showed less performance than fastText.
Text CNN Both for the classification and regression tasks we are interested in how the encoder can perform trained from zero (including the embeddings) for the specified task. We use a Text CNN with the same parameters as the encoder fo tg. It is trained for maximum of 25 epochs over the whole dataset with an early stopping of of iterations without gaining in the validation data. For this case, each experiment in each training sample (as explained in the next session) is run times under different random seeds and we report the mean of the results.
BETO (Spanish BERT) We evaluate our model using the Spanish version of BERT Devlin et al. (2018), named BETO Cañete et al. (2020). since our dataset is in Spanish. Like for our model, we only use the encoded version of the data, i.e. the embedding for the [CLS]] token. We use the implementation of the Huggingface Library Wolf et al. (2019). We feed the encoded [CLS] one used to evaluate our self-supervised model (described in the previous section). The reason for showing our results using BETO and not multilingual BERT is because some preliminary results showed that BETO consistently had at least the same performance of BERT and generally outperformed it, which makes sense because it is trained specifically on Spanish text.
logistic regression classifier with the same characteristics as the
Meta-Prod2Vec + KNN The Meta-Prod2 algorithm was used for comparison purposes, al- though unlike the rest of the baselines, it depends on metadata that might not be present (c.g. the id of an item or the category). It was used alongside a K Nearest Neighbor classi- fier regressor, depending the task. The value of K was 5. To train the embeddings for
11

---

Under review as a conference paper at ICLR 2021
this algorithm used the full dataset sessions for each country, with the id of the item as the main vector and the title and category of the item as the metadata.
12


Reviews:
[{"review_id": "lmDI7SZTJx0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies representation learning in an e-commerce setting. In particular, the paper explores the use of the recently proposed Bootstrap Your Own Latent (BYOL) framework to learn product representations. Rather than using different views of the same entity (as is done in the image domain), different products within a single shopping session are used. Experiments show that the unsupervised BYOL alone does not perform well in this setting. To remedy this issue, the objective is modified by adding a supervised product category prediction loss. Experimental comparisons are made to several baselines on four downstream supervised learning tasks.\n\n### Strengths\n1. Much of the recent success in unsupervised representation learning has been in the CV and NLP domain. Studying the effectiveness of these techniques in other domains is an important practical research direction.\n2. Experiments empirically confirm the benefit of the proposed modification to the BYOL objective.\n\n### Weaknesses\n1. The study of the BYOL framework in a new domain is a welcome contribution (contribution 3). Unfortunately, the experiments do not effectively isolate and study this topic due to the number of factors which change between baselines (architecture, objective, training data, etc). Notably missing are benchmarks for an equivalent architecture (TextCNN) trained with a more standard representation learning setup using a contrastive loss. Without such comparisons it is not possible to determine whether the improved representation learning abilities of the BYOL framework in the CV domain transfer to the e-commerce domain, or if the improved performance is the result of architectural and other experimental differences between baselines.\n2. The paper does not justify the precise modification of the BYOL objective presented here. Would adding an explicit contrastive term, which would be advantageous in scenarios where supervised data is not available, have been less, equally, or more effective? Why predict $c_{p_j}$ from $p_i$ rather than $c_{p_i}$? What about predicting the categories of all products in the session in a multi-label manner, etc? The choices made in the paper should be explained and justified.\n3. Experimental details are not clearly described. For example, I could not find a description of the difference between Site A and Site B. The number of unique products and users is not specified. Are the products that appear in the supervised tasks a subset of the products that appear in the session level data, or are there \"cold-start\" items which the model did not see during pre-training? Is the BERT model fine-tuned for the specific down stream supervised task, or are only the classifier parameters updates? The Appendix leads me to believe the the latter, \"we only use the encoded version of the data\", in which case this baseline seems less representative of best practices.\n4. The Appendix mentions that the Meta-Prod2Vec baseline uses the full session dataset, that this caused \"some problems\" with direct comparison, and that it will be addressed in a \"future releases.\" The nature of \"some problems\" should be clarified and possibly mentioned in the main paper. If results are not comparable they should be removed or fixed before publication.\n5. The paper overstates contributions. The architecture used here (TextCNN) is quite standard, and does not represent a significant contribution alone (contributions 1 and 2). \n\n### Recommendation\nI vote for rejection. Although I believe understanding the effectiveness of BYOL in non-CV domains is an important research direction, I do not believe the current paper effectively addresses this topic due to issues with baselines and a lack of experimental details.\n\n### Additional Info\nThe paper contains grammar mistakes which I found made reading difficult at times. Most are relatively minor and did not obscure from the overall message and content of the paper, and therefore did not contribute substantially to my assessment. Below are several examples from Section 5.1.\n\n1. \"As we can **observer**, in almost all plots\" => \"As we can **observe**, in almost all plots\"\n2. \"the better performance is for the value of α of 0.9 **specially** at very low training samples\" => \"the better performance is for the value of α of 0.9 **especially** at very low training samples\"\n3. \"the curve is **more steep** for cases\" => \"the curve is **steeper** for cases\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "L5KzOMIHrAy", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors study the problem of representation learning of marketplace products to apply in downstream tasks. More specifically, the authors extend the work of BYOL with a new objective function by adding a cross-entropy objective. The main hypothesis of this paper is that different products of the same browsing session can be thought of as different augmentations of the same session. \n\nOverall, the paper is clear and easy to follow. However, I have a few concerns and questions for authors.\n\nConcerns:\nThe first concern is that the main hypothesis of this paper is not very clear to me. Why can different products of the same browsing session be thought of as different augmentations? I did not find clear explanations behind this hypothesis. This hypothesis is not validated in the experimental section, which leads that the hypothesis is not convincing to me. \n\nThe second concern is about novelty. The authors claim that the main contributions of the paper lie in their novel deep encoder architecture. However, the bi-encoder architecture is not novel, and the novelty of added cross-entropy is also limited.\n\nQuestions:\n1. Why can different products be thought of as augmentations to each other? Have you considered cases 1) complimentary products or 2) the users want to purchase several non-related products and browse them at the same time. Have you analyzed the similarities among products in the same browsing session?\n\n2. The sensitive nature of datasets can be understood. Have you considered conducting experiments on other public datasets for easy reproductions?\n\n3. The gap between the performance of the proposed model and the state-of-the-art models is large. The authors claim that one of the advantages is that the inference of their model does not rely on GPU. Why do the authors think that reliance on GPU is a limitation? Moreover, considering the inference efficiency, how does the proposed model compare to other baselines?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "CMNKGz4Y-V7", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, I vote for weak rejection. The idea of being able to learn good latent space embedding with less expensive resources is very beneficial and can have a lot of easy to use applications in e-commerce problem space. The input used in the paper of product title and category are usually widely available for such datasets. The concerns are detailed in the cons section and hopefully the authors can address my concerns in the rebuttal period.\n \n######################################################################\n\nPros: \n(1) The paper has good clarity and is mostly understandable. Interesting and useful enhancement to existing self-supervised learning work of BYOL for application in the new problem space of e-commerce. \n(2) The optimization objectives are well defined and the hypothesis of adding the category-based loss to original BYOL loss is well supported by evaluation and experiments presented in the paper.\n(3) The proposed method has good value and is supported by the authors’ extensive comparison experiments done with industry standard baselines like Meta-Prod2vec, fastText which are commonly used for such tasks of learning product embeddings.\n(4) Detailed explanation of the proposed approach and the related work. Extensive experiments across two datasets.\n \n \n######################################################################\n\nCons: \n(1) The authors use AUCROC as a measure of performance for their comparisons with other baselines, since it is threshold invariant. However, some of their downstream tasks are classification problems (counterfeit and forbidden product detection) which are typically more susceptible to class imbalance and minimizing false positives might be more important. Can the authors provide some more metrics which gives more insight into the performance of the model in such cases or provide an explanation as to why it might be unnecessary?\n(2) The BYOL model used by authors as a base architecture has a significant difference from authors, in that the BYOL tries to learn the representation on the same image (online and target projections are for the same image). But the authors use item pairs from items interacted within the session. But given the scope of the e-commerce listing type, users can interact with a lot of unrelated or complimentary items within the session. To overcome this, they had to add the category constraint, and their experiments prove that without it, the model becomes random. But what happens when the training sample has a lot of unrelated/complimentary items as +ve pairs? One suggestion for downstream classification could be to evaluate the model’s performance on the category prediction. Can the authors provide more clarification on product assortment in general and within the session? (eg: if the marketplace data has products from diverse categories in the same session)\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2ugYaoPUl5M", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a way of product representation learning via BYOL framework. Background introduced, model explained, experiments conducted. I don't think this paper is good enough for acceptance. See detailed comments:\n\n1. The writing. The model part is not clearly written, and neither is the experiments set-up.\n\n2. The novelty. It seems it's a derivative work from BYOL for this particular product embedding application.\n\n3. The sampling method. It is well known that sampling via browse data suffers from bias caused by the search engine. For example, two products are browsed together only if the search engine decides to show them to the shoppers together. The search engine quality will largely impact the quality of the sampled data.\n\n4. Missing important literatures and comparisons. The browse product pairs are essentially the degree-2 connection of products on the query-product bipartite graph. There are vast amount of work for learning representations on such graph.\n\n5. The experiment section is not convincing. By applying the embedding to specific problems doesn't justify the method's superiority.\n\n6. The results about different losses. Results are illustrated yet no discussions or insights are provided. What's the motivation behind?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviews are somewhat below the acceptance threshold. The main concerns are in terms of lack of novelty, and that some of the paper's main claims are unsupported. Many of the criticisms are quite focused on specific details, but these seem significant enough to have been deal-breakers for this submission.
Decision: Reject