A Flexible Large-Scale Similar Product Identification System in
E-commerce
Zhen Zuo, Lixi Wang, Michinari Momma, Wenbo Wang, Yikai Ni, Jianfeng Lin, Yi Sun{zhenzuo,lixiwang,michi,wenbowan,yika,ljianfen,yisun}@amazon.com
AmazonSeattle, WA

ABSTRACT
Identifying similar products is a common pain-point in the worldof E-commerce search and discovery. The key challenges lie in two
aspects: 1) The definition of similarity varies across different applications, such as near identical products sold by different vendors, products that are substitutable to each other for customers
with common interests, personalized products visually similar interms of design pattern, style or color. It is difficult to build a general solution for all scenarios. 2) Computing pairwise similarities
among billions of products are resource demanding which makes
it challenging in large-scale data processing. To provide a flexible
and consolidated solution at scale, this paper presents an all-in-one
system, Product Similarity Service (PSS), which leverages the stateof-the-art Deep Neural Networks and distributed computing technology to serve diverse Amazon-scale product similarity computations. The experimental results show that PSS can return highly
relevant similar products in both verification and ranking tasks,and has good computing efficiency and system scalability on large
data volume.

CCS CONCEPTS• Information systems → Recommender systems; Data mining; Computing platforms;

KEYWORDS
Similar product search systems, recommendation systems, deeplearning, metric learning, multi-task learning, embedding.
ACM Reference Format:
Zhen Zuo, Lixi Wang, Michinari Momma, Wenbo Wang, Yikai Ni, Jianfeng
Lin, Yi Sun. 2020. A Flexible Large-Scale Similar Product Identification System in E-commerce. In Proceedings of The 26th ACM SIGKDD Conference on
Knowledge Discovery and Data Mining (KDD’20 IRS Workshop). ACM, New
York, NY, USA, 9 pages.

1

INTRODUCTION

E-commerce reshapes the new era of customer shopping experience with more and more customers choosing online shopping.
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than
the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee. Request permissions from permissions@acm.org.
KDD’20 IRS Workshop, August 23–27, 2020, San Diego, CA, USA© 2020 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-6201-6/20/8.

Meanwhile, a dramatic rise is observed in the number of products sold on e-commerce websites. Identifying similarity among
a tremendous number of products becomes a critical problem in alarge variety of online business scenarios. For example, Alibaba
built a visual search system to find similar products given customer’s image queries [28]; Pinterest applied image search in theircontent-based product recommendations [27]. In Amazon, there
is also a trend to apply similarity identification to diverse popular
emerging applications. Fig. 1 shows three applications, where (a)recommends substitute products for out-of-stock items based on
their images, titles and descriptions with matched key attributes(e.g. the same phone type), (b) personalizes fashion products whichlook similar but vary in some aspects (e.g. different designs), and(c) searches near identical items sold by different vendors. Allof them need to explore product similarities among Amazon’s
increasingly growing repository but with distinct business goals
and requirements.
As each business application has its own definition of similarity, most of the similarity computing systems in the industry are
tailored with their focus on a certain type of application domain.
It remains challenging to consolidate them into a generic solution
which is flexible enough to serve diverse similarity-based applications, while at the same time, salable enough to process datain large volume. Specifically, Amazon applications launched withsimilarity features usually need to deal with billions of products
from various categories such as fashion, electronic devices, andfurniture, etc. Building independent computing systems for different applications is inefficient from both model development andresource utilization perspectives. Instead, we present a generaland flexible all-in-one similarity product identification platform,
Product Similarity Service (PSS), to compute the similarity as aservice for a variety of customized similarity use cases at Amazon
scale. The following key challenges are addressed in our system.
Flexible Similarity Definition for Various Applications.
Variant Input Product Information: For different applications, thesystem needs to provide the flexibility for input product information selection. For example, for visual similarity recommendations,product image is the key information to use, while for substituteproduct identifications, image, title, and description could all be
important clues for similar product identification.
Different Optimization Goals: The system needs to be flexible
enough to optimize the output goals of different applications. For
example, in visual search, the goal is to return products whichhave similar visual patterns with the query image in a ranking
order. While in substitute products recommendation, the goal is
to identify products which can be replaceable by each other whencustomers make purchase decisions.

Figure 1: Similar product identification applications at Amazon: for a query product, PSS is able to return similar productswith different application definitions.
Key Attributes Matching: The system needs to have the flexibility
to remove unqualified similar items using post filters. For example,
for substitute phone case recommendations, if a customer is exploring iPhone X case, the post filter should be able to remove phonecases that are not compatible to iPhone X. Otherwise, it will cause
a very disappointing customer experience.

mainly two types of signals for similarity computing: productcontextual information, and customer behavioral information.
Content based similarity identification systems have been
adopted throughout industry, they aim to return items with highly
similar contextual information (e.g. image, video, description,
attribute, etc.) to the query item. Among which, visual searchis one prominent application trend: e.g. Alibaba Pailitao Visual
Search [28], Pinterest Visual Search [27], Microsoft’s Visual
Search [9], and they all focus on learning image representations
optimizing for image retrieval problem. Another highly developed
direction is content based recommendation: e.g. [5], whichleverages video contextual information to recommend customers
videos with similar category/search tokens, etc.
Behavior based recommendation systems are also widely applied, They aim to recommend items viewed/purchased by users
with similar preferences. Youtube [13] learn from customers’ cowatched behavior for video recommendation. Amzon [2] learn
from re-purchase behavior for Buy It Again recommendation.
PSS supports the hybrid of contextual and behavior based signals in similarity modeling, and provides enough flexibility tochoose any combination for different applications (e.g. product image, purchase behavior, combination of image & title & behavior,etc.). Moreover, PSS also allows customer to give their own definition of similarity for different applications, e.g. identical products
searching, visually similar products recommendation, etc.

Scalable and Efficient Similarity Computing.
High scalability and efficiency are needed for a similarity computing system to handle Amazon-scale data. Firstly, the systemneeds to be able to scale to process enormous and continuously
growing product sets and publish refreshed similarity results with
controllable time frames. Secondly, the system needs to manage
the computing resource (e.g., CPU/GPU and memory usage) to
optimize for cost as it involves image and text embedding computation from deep neural networks and large item indexing forsimilarity search. Thirdly, the system should be capable of identifying and caching the pre-computed result to make it reusable
across different applications without repetitive computations.
In PSS, we utilized a hybrid modeling method with both productcontextual and customer behavior information and made it configurable for application-specific goals. We leveraged the state-ofthe-art cloud computing technologies to design a distributed similarity computing solution for scaling. We evaluated our system oncomprehensive modeling and system experiments across multiple
applications. We demonstrated that PSS is able to support productbased similarity identification tasks with high quality model andsystem performance at the Amazon scale.
In the rest of the paper, Section 2 compares the related work to
our system, Section 3 describes our similarity framework and approach, Section 4 demonstrates the system architecture in details,
Section 5 evaluates the system performance through experiments,
and Section 6 gives the conclusions and future works.

2.2

Deep Metric Learning

Another related research domain is deep metric learning, which
has been shown to achieve impressive performance for identifying
similar items at multiple application areas: visual search [27, 28],
recommendation [5, 8, 22, 23], face verification / recognition [17,20, 21], etc. Based on the relations of items, there are mainly two
types of supervision signals for similarity: pairs of items marked
as positive or negative [1, 20], and triplets of items (anchor item,
positive item, negative item) [7, 14, 18, 19, 21]. The pairwise supervision can be trained via deep neural networks like SiameseNets[3, 4, 11], which is usually relatively easy to train, and pairwise
training data is also easier to collect at industry. In contrast, thetriplets supervision can be trained by Tripletnet [7, 17, 25], which
has stronger capability to preserve ranking information, but it

2 RELATED WORKS
2.1 Similarity Identification Systems
Most of the industry similarity identification systems focus onproviding highly specialized solutions optimized for very specificapplication scenarios. They are not flexible enough to support
multiple applications. In general, for any application, there are2usually needs more careful training strategies (e.g. triplet selection)
to get good results[17]. Both pair-wise and triplet supervisions areused to optimize for the item embedding, where similar items areclose together, while dissimilar items are pushed far away fromeach other in the embedding space.
In this work, we leverage multi-task SiameseNet to train item
embedding with pairwise datasets, which provides flexibility tosupport application requirements.

3

PSS FRAMEWORK

Figure 3: Compound feature generation using both productcontent features and customer behavior features.

similar / dissimilar) relationship between products. Meanwhile, we
leverage multi-task learning to cover various similarity tasks (e.g.,human labeled similarity data, click-through similarity data). Thenat the query stage, the product embeddings can be extracted fromthe last layer of SiameseNets.

Figure 2: PSS overall framework.
As shown in Fig. 2, PSS similarity identification framework has
three main components: a) Product Embedding: At the training
stage, a product embedding learning model is optimized to represent each product with a feature vector. The embedding should
be encoded with product’s contextual & customer behavior information, and be optimized for specific application based similarity
goals. At the query stage, each query item is passed through thetrained embedding extraction model, and represented as an embedding vector. b) Similarity Indexing & Search: At the training
stage, all product embeddings are built into index artifacts basedon their similarity correlations. Then at the query stage, multiple
queries are sent in parallel to retrieve topK ranked similar items
efficiently. c) Post-filtering: For any application that require further
refinement on similarity results, PSS provides a general attributebased post-filtering module, which only keeps the items whose
attribute values match to those in the query item. Finally, the product similarity datasets are uploaded to the data warehouse, from
where downstream applications (e.g. search, recommendation) canfurther consume.

3.1

3.1.1 Feature Extraction from Multiple Contexts. Fig. 3 shows
the process of extracting initial product compound feature frommultiple sources of raw product information. There are mainly
two types of information: product content (e.g. image, title) andcustomer behavior responses (e.g. co-purchase, co-view).
As shown on the left branch of Fig. 3, for product content
information, contextual feature for each input content domain isextracted. For example: image feature (e.g. extract via AlexNet[12]), title / description feature (e.g. extract via Word2Vec [16]),
etc. Each feature extraction model is trained to focus on a specificcontent domain. E.g. AlexNet image embedding, which is trainedon ImageNet [6] for image recognition; Word2Vec embedding,which is trained on Google News corpus for reconstructing word
linguistic contexts. As shown on the right branch of Fig. 3, for
product level customer behavior information extraction, an Amazon internal service is leveraged to extract product-level behavior
embedding, which is a graph based solution.
As shown at the bottom of Fig. 3, after extracting featuresfrom both product content and customer behavior responses, eachfeature is normalized and concatenated together to get the productCompound Embedding.

Product Embedding Learning

The module (a) in Fig. 2 is the first step in the workflow and generates a vectorized embedding for a product, which should concisely
encode key product information required for similarity identification. However, the definition of similarity at different applications
may vary a lot. For example, similar shoes usually have similarheel type, but this similarity criteria is not suitable for the otherproducts (e.g. electronic devices, furniture). One common solution
is to carefully select suitable input signals and similarity criteria
for each application. However, this is neither scalable nor efficient.
To build a flexible product embedding learning framework, we
use common product information applicable for multiple producttypes (e.g. image, title, customer behavior responses, etc.) as input
signals. We leverage SiameseNet to learn from the pairwise (i.e.,

3.1.2 Multi-Task SiameseNets for Product Embedding Learning.
The initial product compound feature has a good coverage of all
types of product contextual information, however, it’s not directly
optimized for similarity identification tasks. Thus, as shown in Fig.
4, we build a SiameseNet [3, 11] on top of the compound feature
to learn a more discriminate product embedding from the pairwise
similarity dataset. Furthermore, to enable the flexibility of optimizing for multiple similarity training goals (for example, customer
behavior similarity goal, human labeled dataset similarity goal), weleverage multi-task learning.
33.1.3 Pairwise Similarity Data Mining. The major challenge oftraining a high quality product embedding from SiameseNet is tofind hard positive/negative training pairs [24]. A naive way is torandomly select products from the same category as positive pairs,
and products from different categories as negative pairs. However,there are lots of false positives which do not meet the application
requirements (e.g. pairing a sport style t-shirt and a hip-hop stylet-shirt as positive might not be appropriate), meanwhile the crosscategory negatives are very easy to train, and the convergence is
too quick without contributing to the learning.
In order to make the model training efficient meanwhile effective for any give task, firstly, we leverage general similarity
customer behavior data sources (e.g. view-to-purchase) to mine
positive and hard negative training pairs. Secondly, we leverage application focused data sources (e.g. human labeled similar fashion
items) to min positive/negative training pairs. For any given similarity data source, we further conduct cleaning steps to collect highquality positive & negative training samples. More specifically, forpositive pair candidates, we remove the pairs which don’t havehigh similarity (e.g. cosine similarity < 0.6) on either image / title/ product behavior embeddings. For negative training candidates,
we remove the pairs which have very high similarity (e.g. cosine
similarity > 0.8) on all embeddings.

Figure 4: SiameseNets product embedding learning.

More specifically, in the training stage (Fig. 4 left), pairs ofsimilarity training data are forwarded in two branches of networks
with shared weights. Then at the network’s last hidden layer, theoutputs from the two branches are merged and their L2 distanceis computed for similarity prediction. Finally, the pairwise loss
(e.g. contrastive loss) of each optimization task is computed, andaggregation of weighted losses for all the tasks is the final loss. We
use back-propagation to optimize the network. In the query stage(Fig. 4 right), we directly consume the learned network to extract
the product embedding for each item.
Mathematically, for an item i, its compound feature fi passes the
L-layer fully connected (FC) networks F F N (.), and each FC layer
is then followed with a batch normalization [10] layer which makes
the optimization more robust. Finally, the product embedding canbe represented as the last hidden layer of the network:
hiL= F F(N ((fi )))= σ σ fi W (1) + b (1) · · · W (L ) + b (L )3.2

Similar Products Indexing & Search(1)where W (1) ,W (2) , · · · , W (L ) are weight matrices andb (1) , · · · , b (L ) are biased terms. σ (.) is the non-linear activation
function (in this paper, we use Relu [12]),A high quality product embedding is able to pull similar items
closer while push dissimilar items away from each other in theembedding space. Thus, the pairwise objective function for a task
t is:
(
)
∑
1
Lt = |M
Lti j yi j , hi , h j|(i, j ) ∈M∑
(2)
Lall =
α t ∗ Lt

Figure 5: Distributed similarity indexing and query
After each product is represented with embedding, similarity
indexes (Fig. 2 module b) is built, which enables similarity searchin an accurate and scalable manner. k-Nearest Neighbor (kNN) isa natural and popular way to do similarity indexing and querying.
The similarity indexing step targets on organizing and recording
all the items in the search space (the set of products from which
we want to find similarities) with a indexing structure, which iscalled Index Artifacts, and it can support fast search afterwards.
Popular indexing algorithms include HNSW [15], K-D tree [26].
In PSS, we select HNSW, since it provides a good balance between
accurate similarity indexing and fast searching. The querying steptargets on retrieving similar items for each query from the built
index artifacts. The main challenges are: 1) How to efficiently build(t ) ∈T

where Lti j is the loss between SiameseNet left branch output hi andright branch output h j for task t. yi j is the pairwise ground truthlabel (similar/dissimilar), M is the set of training pairs, and |M| isthe training dataset size. T is the set of all tasks. In this paper,
we use contrastive loss for each task, and Lall is the weighted
aggregated loss for all tasks. Note F F N (.) is optimized with thelearning of (2) and once model training is done, F F N (.) can beused to generate product embeddings.
4

Figure 6: Effects of configuring post-filter refinement logic
index for a large-scale search space; 2) How to do similarity searchat scale.
To enable a Amazon scale similar product computing, we developed a distributed KNN solution illustrated in Fig. 5. To build theindex given very large dataset which cannot fit into a single machine, we divide the dataset into multiple partitions. Each partitionis small enough, so it can be loaded into the memory of an index
node and used to build a partitioned index. A set of the indices built
from the partitioned dataset is considered as an index collection. It
is referenced together to represent the entire search space for given
query. Similarly, query data is also partitioned and distributed toa cluster of query nodes. Each query node searches in a partitionindex for the query partition and return the sorted top K similar
items for each query in the partition. Finally, all partition resultsfrom the same index collection are pushed to a merge node where
it heap sorts final topk most similar items from all candidate lists
and upload them to destination.

3.3

Figure 7: Similarity computing as a service
user requests to deliver application specific similarity results. To
process data at large scale, distributed executors are orchestrated
hierarchically at each step to perform the computations onpartitioned data concurrently. To flexibly integrate different
product embedding models to the internal search engine, adedicated model pipeline is maintained to automate the model
learning and the hosting process.

4.1

Post-filtering Refinement

As shown in Fig. 2 module (c), the last process is post-filteringrefinement. For some business use cases, additional constraints
are required to make the similarity identification valid. For example, two T-shirts with identical image but different sizes are forcompletely different customers. In order to further enhance performance, PSS provides the flexibility of adding application specificconstraints through configuring a list of key attributes for similarity identification refinement. Any similarity candidates withunmatched attribute values will be removed in the post-filtering
stage. Fig. 6 shows an example of attribute based post-filtering: (a)shows similar items without any filtering; (b) shows similar itemswith the same color as the query item; (c) shows similar itemswith the same dress sewing style as the query item. In production,different application have enough flexibility to choose its own listof attributes.

4

Cache-based Product Embedding

As mentioned in Section 3, Product Embedding is served as a preprocess step of similarity computing to represent each product asa vector using a pre-trained product embedding model as specified
in the workflow configs. For example, the user can choose torepresent a product with an image embedding, a compound embedding with both the image and title information, or an application
specific embedding trained with SiameseNet.
Currently, a variety of heterogeneous Deep Learning (DL)-basedmodels are hosted in our system, implemented on a common model
framework with their own training and inference logics. A lightweighted elastic training solution is implemented based on AWS
Sagemaker 1 to launch the training on demand. When the training
is done, a model artifact will be pushed to a separate model hosting
pipeline where the model is containerized into a docker image
2 and published to an embedding model repository. Later the
search engine launches the model instance from its docker image
to generate the embedding for each product. With such a model
pipeline, it enables the model to be continuously deployed andflexibly integrated to the similarity service. When code changes
related to the model packages are pushed, the images with newer
version will be deployed without any changes on the search engine
side.

SYSTEM ARCHITECTURE

This section demonstrates the system architecture of PSS. As
shown in Figure 7, PSS allows the users to invoke a step-based
workflow for similarity computing through a service call so thatboth similarity training and query flows can be instantiated with

1 https://aws.amazon.com/sagemaker
2 https://docs.docker.com/

5For efficiency consideration, a cache mechanism is leveraged to
avoid duplicate calculations in the embedding extraction which is
expensive especially for the DL-based models. More specifically, acache table is associated with model identifier with its embedding
stored as a key-value pair. A cache handler is implanted in the basemodel layer to apply to all models. At embedding extraction step,
it first attempts to read the embeddings from the cache table andreturns immediately if hits the cache, otherwise it performs theactual embedding extraction and writes the new embedding to thecache table. Note that it is assumed that the product embedding
always remains invariant for the same model. However, the only
exception is when the model is updated and the inference functionality changes which is not very common in the production stage.
In that scenario, we will clean up the corresponding cache table.

4.23) System performance: we test system scalability on large-scale
similarity computing tasks. At last, we visualize the similarity
identification results deployed for multiple productions.

5.1

Distributed Similarity Computing

4.2.1 Batch Framework. The similarity search engine is built
on a distributed computing framework integrated with the
AWS Batch3 infrastructure which enables efficient and dynamic
resources provisioning and job scheduling for large scale batchprocessing. Under this computing framework, the resource pool
is divided by batch queues configured with virtual computing
environments such as number of vCPUs, scaling strategies, etc.;
a Batch scheduler dispatches the job executions across queues inpriority order on optimal computing resources. As the service
is invoked, a master job is submitted to the batch queue. Whenthe requested computing resource is provisioned, a step-based
workflow is initiated based on the user’s request, following apredefined workflow graph as shown in Fig. 7.

5.1.2 Data Sources for Model Training. We introduce two resources for training: customer view-to-purchase (V2P) data, andhuman expertise labeled (Labeled) data. V2P Data: It has hugevolume of candidates, but the data quality is relatively noisy. For
positive product pairs, we select the most recent viewed products,
and the purchased products right afterwards. For hard negative
pairs, we select the product viewed N (e.g. N=20) slots before thepurchased product. For both positive/negative pairs, we further dodata cleaning (refer to Section 3.1.3) to remove behavior data irrelevant to similarity. In this experiment, we randomly select 100K
positive pairs, and 500K negative pairs for V2P-based similarity
task training. Labeled Data: it’s an accurate data source, but usually
only contains limited number of data since it’s labor intensive tocollect. For positive pairs, we collect domain experts labeled substitute products (e.g. substitutable apparel products should have
similar image, title, color, material, etc.). For negative pairs, we
randomly sampled product pairs with 5x volume of positive pairs.
For model training, we split 4K positive human Labeled pairs
with 20K negative pairs for human label based similarity task. The
remaining human labeled pairs are for model evaluation, which
will be explained further in Section 5.2.

4.2.2 Distributed Executors. Each step in the master workflow
is scaled by step executors. A primary executor is initiated to index
the source partitions in a manifest file and then distribute the partitions to an array of secondary executors for parallel processing.
Each secondary executor is assigned a set of partitions according
to the manifest and is dedicated to process them locally. Both the
manifest and the partition size can be tuned for an appropriate concurrency level to achieve the overall scaling performance. Multilevel manifests are used for extreme large-scale processing.
This distribution solution applies to all the major steps. As
mentioned above, embedding executors compute the product vectors on the partitioned product set for similarity computing; filter
executors refine the similarity candidate partitions. The similarity
training and query are implemented in a similar way. In training,data is split to fit into a single indexing executor node. In querying,
the search executors first perform the topK query on partitionindexes distributively and then a merger collects and merges the
all the partition results from the same query for a final topK.

5

Offline Evaluation Setups

5.1.1 Input Contextual Information. We mainly experiment on
the Compound feature (Section 3.1.1), which is the integration of
three input contextual information: 1) Image: we use AlexNet[12]to extract initial image embedding, 2) Title: we use Word2Vec [16]to extract word-level embedding, then pool all word embeddings to
get a title-level embedding, 3) ProdBehavior: we use an internal service to learn product embedding based on customer behavior (e.g.
co-view, co-purchase). We further reduce each feature’s dimension
to 128-dim for computation efficiency. In Section 5.2 & 5.3, unlessexplicitly written, the default SiameseNet input is the Compoundfeature that integrates image, title, and ProdBehavior features.

5.2

Product Similarity Verification

To evaluate the learned product embedding quality (Section 3.1),
we conduct experiments on pairwise similarity verification tasks:
model performance on predict similar/dissimilar labels for pairsof products. For evaluation, we select 5K positive pairs labeled byhuman expertise (Section 5.1.2), and randomly select 25K negative
product pairs. The evaluation metrics are recall at precision 80%,90%, 95%.
5.2.1 Effects of using different input contextual information. To
quantify the effects of introducing multiple product context information (Section 3.1.1), we test different combinations of input contextual information. Comparing on R@P80% metric, compoundfeature (concatenating image, title and ProdBehavior) gets 43.32%,
while image only gets 29.61%, ProdBehavior only is 32.44%, andTitle only is 7.36%. Thus, different input signals are complementary
to each other, and aggregating all contextual input information canbring to the best performance on this similarity verification task.

EXPERIMENTS & ANALYSIS

In this section we analyze experimental results from three aspects: 1) Offline-evaluation: we describe offline experiment setups,
then test two tasks, pairwise product similarity verification, andsimilarity ranking. 2) Online-evaluation: we show the similarity
identification performance in online A/B test on Amazon products.
3 https://aws.amazon.com/batch6

5.2.2 Effects of Multi-Task Learning. To understand the effectsof supporting configurable multiple sources of similarity training signals, all the numbers in Table 1 are tested on different
combination of multi-task learning (Section 3.1.2) with the same
SiameseNets structure: four layer fully connected with hidden
nodes number: 256+256+256+256. As shown in Table 1 second
line, if we only leverage the most general product similarity signal
V2P, we’re able to return some high precision substitute productswith reasonable recall, which can be a good option when no extra
application specific training signal is available. As shown in line 3,
when only training with a small set of human expertise labeleddata (e.g. 4k human labeled positive pairs), we can get slightly
better results with V2P. As shown in line 4, when integrating
multiple optimization signals with multi-task learning, the optimal
results can be achieved via learning complementary information.

Discounted Cumulative Gain (NDCG) and Mean Average Precision(MAP) for performance evaluation. NDCG measures the quality ofranking. Mathematically, the formula of DCG at rank position k∑p
2r eli −1
is defined as: DCG k = i =1 log, where i is the position
2 (i +1)
in the recommended list and reli ∈ {0, 1} stands for whether
the ith recommendation is labeled positive. NDCG is the ratio
between DCG and the largest DCG possible. MAP measures theaverage accuracy of a ranking algorithm. A query product’s AP
is the area under the Precision-Recall curve of that query: AP =∫1
∑K
P(r )dr ≈P(r )(r (k) − r (k − 1)), where P(r ) is the
k =10
precision at recall level r and r (k) stands for the recall at top k
recommendations with r (0) = 0. In practice, AP is approximatedby a finite sum. We choose K = 30 in our calculation. MAP is themean of AP over all the queries. For the SiameseNet model, we use
256 + 256 + 256 + 256 structure.

Table 1: Comparison of different optimization tasks

5.3.1 Comparison on Different Product Embedding. Comparing
first 3 rows of Table 3, image feature relatively have high performance as a single contextual information source, since image isusually more easily to be captured by customers. Comparing line
4 with first 3 lines , aggregating more product contextual information (all vs image/ProdBehavior/Word2Vec only) can capture more
complementary signals, and get better similarity ranking results.
Comparing line 4 & 5, we can observe product embedding learning
through SiameseNets can encode extra information for similarity
identification, and generate better ranking results.

Data Augmentation
Compound Feature
View-to-Purchase(V2P)
Labeled 4K
V2P + Labeled 4K

R%@P80%
43.32
48.53
52.87
90.46

R%@P90%
32.31
18.06
19.92
78.78

R%@P95%
22.99
8.62
7.93
68.59

5.2.3 Effects of Different SiameseNet Structures. We conduct
this experiment to understand the effect of using different SiameseNet structures (Section 3.1.2). All the models tested in Table 2 aretrained with two tasks: V2P + Labeled 4K. From the fist two linesof Table 2, we can clearly see the product embedding learned from
SiameseNet is much better than the original compound feature(refer to Section 3.1.1). Comparing line 2-5, we can clearly see
adding more layers will progressively lead to higher performance
than shallower one when there is enough training data, but theextra gain might be relatively small when the network is almost
saturated with the given training data (line 4 vs 5). Comparing
the line 4 & 6, or line 5th & 7th, we can observe deeper network
performs slightly better than wider net.

Table 3: Ranking results on different product embeddings
Prod. Embeddings
Image
ProdBehavior
Title
CompoundSiameseNet (ours)

5.3

R%@P80%
43.32
79.33
83.27
88.95
90.46
85.53
89.18

R%@P90%
32.31
53.00
64.11
74.42
78.78
66.88
77.03

MAP
0.44480.34330.37290.48990.52675.3.2 Performance on Different Product Categories. As shownin Table 4, for different categories, customers show strong signal
preferences, which further justify the importance of providing
flexible similarity signals for multiple applications. For example,
for fashion categories (e.g. apparel, shoes) and home category, theimage is the dominant signal. While for other general categories,
e.g. PC, toy, aggregating multiple sources of contextual information can better help customers with multiple preference. We also
observe the SiameseNet based product embedding can overall dobetter similarity ranking across multiple product categories.

Table 2: Comparison of different SiameseNet structures
SiameseNet Struct.
Compound Feature
256
256 + 256
256 + 256 + 256
256 + 256 + 256 + 256
512 + 256
512 + 256 + 256

NDCG@1 NDCG@10 NDCG@20
0.35850.46610.47780.27120.35930.37120.29740.39070.4031
0.39280.50930.52550.41430.55380.5720R%@P95%
22.99
33.33
44.04
61.39
68.59
50.39
64.31

5.4

Online A/B Test

To evaluate the effectiveness of our models in real world, we
conduct the online A/B test for typical similarity applications in
E-commerce. To measure the actual customer engagement differences between the control (A) and treatment (B) groups, the lift in
Purchased Units are collected for the applications in Amazon.com.
Note that we only report the purchased units lift due to the page
limits as other important metrics such as products purchased profits lift, etc. are highly aligned.

Similarity Ranking

To evaluate the performance on ranking task, we test on a ranking
dataset collected from click-through data in substitute recommendation. It has 15K query products, and in-total 150K products insearching scope. For ranking evaluation, we generate topK substitutes for each query product. Similar to [13], We use Normalized7

Table 4: Average precision for click-through data
Category
Image
ProdBehavior
Title
CompoundSiameseNet (Ours)

pc
0.35670.31680.3514
0.46640.4476apparel
0.46620.13280.17070.2771
0.5687jewelry0.32230.73670.63730.76240.6006luggage0.46020.42710.44440.55700.5392home
0.73030.20870.25520.40510.6901

office product0.31520.58070.64350.68650.4688shoes
0.41270.15700.21710.33320.4328kitchen0.29050.53170.61180.67980.5080All
0.44480.34330.37290.48990.5267

Table 5: Performance of Distributed Similarity Computing

5.4.1 Substitutes Recommendation for OOS Products. This recommendation application displays the top similar substitutes forcustomers when their selected products are out-of-stock (OOS).
We choose this application because it has good coverage on mostof the product types (e.g. apparel, furniture, electronic devices,etc.). For the treatment group, the substitute data is learned fromproduct image and behavior information, and constrained on key
attributes matching (e.g. material, color, etc.) through PSS. For
the control group, the shown recommendation is the voting ofbest results from all in-production OOS recommendations methods. This experiment results in a significant lift on number ofpurchased items (+6.58%, p=.03) in the treatment vs. control during
four weeks’ experiment period.

Data Size
1MM
10MM
50MM
100MM

Indexing
Speedup Cost Save
4.65
78.48%
4.40
77.26%
5.33
81.22%
3.52
71.60%

Query
Speedup Cost Save
41.63
33.14%
48.42
46.16%
70.33
61.90%
77.82
63.92%

allocated to a distributed executor is in proportion to the size of thedata partition assigned to it.
Table 5 demonstrates the performance and efficiency improvement that our distributed similarity computing can achieve whenbenchmark against a non-distributed solution. From the result, we
can see that both distributed indexing and query constantly outperforms the corresponding non-distributed solution. Although
distribution introduces overhead from both resource provisioning
and partition merging, it is able to speed up 4.5X on average inindex-building with 5 partitions on the training dataset, and 59.6X
in searching for top 1000 similar items with 500 partitions on thequery dataset while still as good as the non-distributed with a
99.5% overlap of results. With current setting, the indexing canachieve its optimal performance when the dataset is divided into
10MM items per partition; in query, the performance improvesas the data size becomes larger, distribution on smaller partitions
does not gain as much as the bigger ones as the above-mentioned
overhead dominates over the actual searching time. We also compare the costs in both solutions based on the total vCPU andmemory allocation to the executors. Although more executors arelaunched in the distributed version, each executor tends to run
much faster with less resource which substantially improves theoverall resource utilization. As a result, we are able to save as muchas 77.14% and 51.28% of resource on average in distributed indexing
and query respectively.

5.4.2 Similar Fashion Product Recommendation. This application shown on personalized Amazon Fashion page is aimed toencourage customer to explore similar new fashion products basedon their selections. For the treatment group, the similar fashion
items are learned from items with both similar images and title /description through PSS. For the control group, it’s the voting ofbest results from all in-production personalized recommendations
for fashion products. This experiment results in a lift on purchased
units (+1.02%, p-value = 0.04) in the treatment vs. control during
four weeks’ experiment period.

5.5

toy
0.2692
0.4691
0.52810.5982
0.4659Distributed Computing Performance

To evaluate the scalability of our PSS service, we compare thesimilarity computing performance of our distributed solution with
the non-distributed one. To setup the environment, the computing
queue is configured with a cluster of virtual computing nodes
hosting on an EC2 host pool.
Four groups of tests are prepared with varied input size ranging
from 1MM to 100MM of products. Each input dataset is a list of 128dim image embeddings of products extracted from several related
categories and used as both indexing and query input within thetest group. For each group, similarity indices are first built using
the given dataset and then the top 1000 nearest neighbors aresearched from the indices built for each item in the set. This flow is
performed in both non-distributed and distributed scenarios. In the
non-distributed case, single executor is launched and provisioned
with sufficient resources to process the entire dataset, specificallythe memory which is considered as the bottleneck in a traditional
KNN solution; in the distributed case, we scale up to 500 partitionsin query while restrict the concurrency level in indexing to 5
partitions for each dataset. This is to minimize the overhead in
merging topK from multiple index models. The amount of resource

5.6

Visualizations

As shown in Figure 8, our PSS can support flexible similarity definition for multiple applications via configuring input contextual
information, and application-related optimization goals. For example, in the duplicated product identification application (Figure 8top), image and title information are consumed as inputs, and the
optimization goal is to identify products with near-identical product image / title as similar. The visual similar recommendation andsubstitute product search are the visualization for the productions
discussed in Section 5.4.
8

Proceedings of the IEEE International Conference on Computer Vision. 1983–1991.
[5] Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks
for youtube recommendations. In Proceedings of the 10th ACM conference onrecommender systems. ACM, 191–198.
[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009.
Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on
computer vision and pattern recognition. Ieee, 248–255.
[7] Elad Hoffer and Nir Ailon. 2015. Deep metric learning using triplet network. In
International Workshop on Similarity-Based Pattern Recognition. Springer, 84–92.
[8] Cheng-Kang Hsieh, Longqi Yang, Yin Cui, Tsung-Yi Lin, Serge Belongie, andDeborah Estrin. 2017. Collaborative metric learning. In Proceedings of the 26th
international conference on world wide web. 193–201.
[9] Houdong Hu, Yan Wang, Linjun Yang, Pavel Komlev, Li Huang, Xi Chen, Jiapei
Huang, Ye Wu, Meenaz Merchant, and Arun Sacheti. 2018. Web-scale responsive
visual search at bing. In ACM SIGKDD. 359–367.
[10] Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating
deep network training by reducing internal covariate shift. arXiv preprint
arXiv:1502.03167 (2015).
[11] Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. 2015. Siamese neuralnetworks for one-shot image recognition. In ICML.
[12] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet
classification with deep convolutional neural networks. In Advances in neural
information processing systems. 1097–1105.
[13] Joonseok Lee, Sami Abu-El-Haija, Balakrishnan Varadarajan, and Apostol
Natsev. 2018. Collaborative deep metric learning for video understanding. In
ACM SIGKDD. 481–490.
[14] Rui Lu, Kailun Wu, Zhiyao Duan, and Changshui Zhang. 2017.
Deepranking: Triplet MatchNet for music metric learning. In 2017 IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 121–125.
[15] Yury A Malkov and Dmitry A Yashunin. 2018. Efficient and robust approximate
nearest neighbor search using hierarchical navigable small world graphs. IEEE
TPAMI (2018).
[16] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficientestimation of word representations in vector space. arXiv:1301.3781 (2013).
[17] Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A
unified embedding for face recognition and clustering. In Proceedings of the IEEE
conference on computer vision and pattern recognition. 815–823.
[18] Devashish Shankar, Sujay Narumanchi, HA Ananya, Pramod Kompalli, andKrishnendu Chaudhury. 2017.
Deep learning based large scale visual
recommendation and search for e-commerce. arXiv preprint arXiv:1703.02344(2017).
[19] Kihyuk Sohn. 2016. Improved deep metric learning with multi-class n-pair loss
objective. In Advances in neural information processing systems. 1857–1865.
[20] Yi Sun, Xiaogang Wang, and Xiaoou Tang. 2014.
Deep learning face
representation from predicting 10,000 classes. In Proceedings of the IEEE
conference on computer vision and pattern recognition. 1891–1898.
[21] Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato, and Lior Wolf. 2014.
Deepface: Closing the gap to human-level performance in face verification. In
Proceedings of the IEEE conference on computer vision and pattern recognition.
1701–1708.
[22] Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. 2018. Latent relational metriclearning via memory-based attention for collaborative ranking. In Proceedingsof the 2018 World Wide Web Conference. 729–739.
[23] Aaron Van den Oord, Sander Dieleman, and Benjamin Schrauwen. 2013.
Deep content-based music recommendation. In Advances in neural information
processing systems. 2643–2651.
[24] Jiang Wang, Yang Song, Thomas Leung, Chuck Rosenberg, Jingbin Wang, James
Philbin, Bo Chen, and Ying Wu. 2014. Learning fine-grained image similarity
with deep ranking. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition. 1386–1393.
[25] Kilian Q Weinberger and Lawrence K Saul. 2009. Distance metric learningfor large margin nearest neighbor classification. Journal of Machine Learning
Research 10, Feb (2009), 207–244.
[26] wikipedia. [n. d.]. k-d tree. https://en.wikipedia.org/wiki/K-d_tree.
[27] Andrew Zhai, Hao-Yu Wu, Eric Tzeng, Dong Huk Park, and Charles Rosenberg.
2019. Learning a Unified Embedding for Visual Search at Pinterest. In ACM
SIGKDD. 2412–2420.
[28] Yanhao Zhang, Pan Pan, Yun Zheng, Kang Zhao, Yingya Zhang, Xiaofeng Ren,and Rong Jin. 2018. Visual search at alibaba. In ACM SIGKDD. 993–1001.

Figure 8: Visualizations of different similarity applications.

6

CONCLUSIONS & FUTURE WORKS

This paper presents PSS, a flexible and scalable similarity identification system, which supports different product similarity computation applications. With configurable SiameseNet-based models, PSS is capable of generating various high quality applicationspecific product embeddings and is scalable of efficient similarity computations at Amazon scale owning to its cloud-based distributed system framework. In the future, we will extend our solution to identify similarity among entities with richer informationsuch as video or audio. We will support different types of productembedding such as triplet model or graph-based models to fit moreproduction use cases. We are also considering making PSS a public
service available for research communities to use.

REFERENCES[1] Sean Bell and Kavita Bala. 2015. Learning visual similarity for product design
with convolutional neural networks. ACM TOG 34, 4 (2015), 1–10.
[2] Rahul Bhagat, Srevatsan Muralidharan, Alex Lobzhanidze, and Shankar
Vishwanath. 2018. Buy It Again: Modeling Repeat Purchase Recommendations.
In ACM SIGKDD. 62–70.
[3] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah.
1994. Signature verification using a” siamese” time delay neural network. In
Advances in neural information processing systems. 737–744.
[4] Dahjung Chung, Khalid Tahboub, and Edward J Delp. 2017.
A two
stream siamese convolutional neural network for person re-identification. In
9