An Extensible Multi-modal Multi-task Object Dataset with Materials
Authors: Trevor Scott Standley
Ruohan Gao
Dawn Chen
Jiajun Wu
Silvio Savarese
Venue: ICLR 2023 poster (2023)
Keywords: Multi-task
multi-modal
dataset
materials
weak supervision

Abstract: We present EMMa, an Extensible, Multimodal dataset of Amazon product listings that contains rich Material annotations. It contains more than 2.8 million objects, each with image(s), listing text, mass, price, product ratings, and position in Amazon’s product-category taxonomy. We also design a comprehensive taxonomy of 182 physical materials (e.g., Plastic → Thermoplastic → Acrylic). Objects areannotated with one or more materials from this taxonomy. With the numerous attributes available for each object, we develop a Smart Labeling framework to quickly add new binary labels to all objects with very little manual labeling effort, making the dataset extensible. Each object attribute in our dataset can be included in either the model inputs or outputs, leading to combinatorial possibilities in task configurations. For example, we can train a model to predict the object category from the listing text, or the mass and price from the product listing image. EMMa offers a new benchmark for multi-task learning in computer vision and NLP, and allows practitioners to efficiently add new tasks and object attributes at scale.

Published as a conference paper at ICLR 2023
AN EXTENSIBLE MULTIMODAL MULTI-TASK OBJECT DATASET WITH MATERIALS
Trevor Standley' Istand @cs.stanford.ccu
Ruohan Gao'
Dawn Chen
Jiajun Wu
Silvio Savarese
Stanford University
Google Inc.
SSaeesforce Research
ABSTRACT
We present EMMa, an Extensible, Multimodal dataset of Amazon product listings that contains rich Material annotations. It contains more than 2.8 million objects, each with image(s), listing text, mass, price, product ratings, and position in Amazon's 's product-category taxonomy. We also design a comprchensive laxonomy of 182 physical materials (e.g., Plastic - Thermoplastic - Acrylic). Objects are annotated with one or more materials from this taxonomy. With the numerous attributes available for each object, we develop a Smart Labeling framework quickly add new binary labels to all objects with very little manual labeling effort, making the dataset extensible. Each object attribute in our dataset can be included in either the model inputs or outputs, leading to combinatorial possibilities in task configurations. For example, we can train a model to predict the object category from the listing text, or the mass and price from the product listing image. EMMa offers a new benchmark for multi-task learning in computer vision and NLP, and allows practitioners to efficiently add new tasks and object attributes at scale.
INTRODUCTION
Perhaps the biggest problem faced by machine learning practitioners today is that of producing labeled datasets for their specific needs. Manually labeling large amounts of data is time-consuming and costly (Deng et al. 2009; Lin et al., 22044; Kuznetsova et al.] 2020). Furthermore, it is often not possible to communicate how numerous ambiguous corner cases should be handled (e.g.. is hole puncher "sharp"?) to the human annotators we typically rely on to produce these labels.
Could solve this problem with the aid of machine learning? We hypothesized that could accurately add new properties to every instance in a remi-automated fashion given a rich dataset with substantial information about every instance.
Consequently, we developed EMMa, large, object-centric, multimodal, and multi-task dataset. We show that EMMa can be easily extended to contain any number of object labels using Smart Labeling technique we developed for large multi-task and multimodal datasets. Multi-task datasets contain labels for more than one attribute for each instance, whereas multimodal datasets contain data from more than one modality, such as images, text, audio, and tabular data. Derived from Amazon product listings, EMMa contains images, text, and a number of useful attributes, such as materials, mass, price, product category, and product ratings. Each attribute can be used as either a model input or model output. Models trained on these attributes could be useful to roboticists, recycling facilities, consumers, marketers, retailers, and product developers.
Furthermore, we believe that EMMa will make a great multi-task benchmark for both computer vision and NLP. EMMa has many diverse CV tasks, such material prediction, mass prediction, and taxonomic classification. Currently, most multi-task learning for NLP is done on corpora in which each input sentence labeled only for single task. In contrast, EMMa offers unique tasks, such as predicting both product ratings and product price from the same product listing text.
One important contribution of this work is that EMMa lists each object's material composition. No existing materials dataset has more than a few dozen material types; furthermore, there are no other large materials datasets that object-centric. This prevents models from being able to learn

---

Published as a conference paper at ICLR 2023
Die-cast Mercedes Benz
Extensibility
Sharp
No
Transparent
Yes
Deformable
No
Articulating
Yes
Electronic
No
Images
Your property
?
Price
$ $4999
Material
Plastic, Metal 2.16 lbs
Label ~1k Objects
Weight
Ratings
* streingo
Categories
Toys & Games Vehicles,
Smart Labeling
About this item
ve cut retal body with pagtc parts
plodk
Propagate to all 2.8 Million
(ull Mulllnnagn dlaaitn With al cod spring puspansion
Over 2.8 Million Objects
syslani
We introduce EMMa, an object-centric, multimodal, multi-task dataset of Amazon product listings that contains over 2.8 million objects. Each object EMMa is accompanied by images, listing text, mass, price, product ratings, position in Amazon's 's cratgoyy taxonomy, etc. We also introduce Smart Labeling technique that allows practitioners casily extend the entire dataset with binary properties of interest (e.g.. sharpness, transparency, deformability, etc.) with only hours of labeling effort.
Figure I:
important distinctions about the materials from which object is made. In contrast, each object in EMMa is annotated with one more materials from a hand-curated taxonomy of 182 material types (see Figure
Armed with dataset containing such rich annotations, we developed a technique for adding high- quality binary object properties to EMMa with minimal manual labeling effort, by leveraging all available data for each object. Our technique employs active learning and powerful object embed- ding. We envision practitioners adding the labels themselves for their own use cases, obviating the substantial work typically required to obtain and curate high-quality data labels from crowdsourcing services such as Amazon Mechanical Turk.
Our main contributions are threefold. First, present EMMa, large-scale, multimodal, multi-task object dataset that contains more than 2.8 million objects. Second, our dataset labeled in accordance with hand-curated taxonomy of 182 material types, which is much larger compared with existing material datasets. Third, we propose a Smart Labeling pipeline that allows practitioners to easily add new binary labels of interest to the whole dataset with only hours of labeling effort.
2 RELATED WORK
Multi-Task Learning Datasets Many large computer vision multi-task datasets, such as Taskon-
omy (Zamir et al. 2018), 3D Scene Graph (Armeni et al. 2019), and Omnidata (Eftekhar et al..
2021 ofter large number of tasks for each image. However, unlike EMMa, these datasets are not
object-centre and focus on pixel-level prediction tasks. Other multi-task datasets, such as COCO Lin et al. 2014), NYUv2 (Nathan Silberman & Fergus 2012). and Cityscapes (Cordts al. 2016 relatively small and have only a few tasks, while others are either artificial, such as MulIMNIST (Sabour al. 2017). focused restricted domain like human faces (Liu et al.) 2015).
Learning from Amazon Data Our work not the first to use Amazon data in a machine learning context. For example, the ABO dataset (Collins et al. 2022) provides listings for 150k objects, about 8k of which have 3D models. Unfortunately, nearly half of the 50k objects are cell phone cases, and the data provided for most objects is in the raw form provided by Amazon. Likewise, image2mass (Standley et al. 2017) curates a dataset of Amazon listings for the purpose predicting an object's weight given its image, but the processed dataset is only for a single task. The UCSD Amazon review data (Ni et al. 2019) is quite large, containing raw data for 15.5 million products, with a focus on product reviews. We incorporate raw data from the image2mass and UCSD datasets into our dataset, which also contains new data we collected from Amazon. As far as we know, we are the first to take advantage of the bulk of the information in Amazon product listings for machine learning.
2

---

Published as a conference paper at ICLR 2023
Auto Labeling Our work is also related to algorithms for label propagation (Gong et al. 2016 Wang et al. 2013; Luo et al.) 2013; Iseen et al. 2019; Douze et al.] 2018). While they focus on distances between instances graphs,
use deep learning predictions from multiple data sources to fill in missing information. There are some similarities here with Tesla's auto-labeling (Elluswamy & Karpathi 202T), where labels are automatically produced corrected using other data in the dataset. Other projects, such as Snorkel (Ratner et al. 2017) and Snorkel McTaL (Ratner et al. (2018), seek to alleviate the data labeling burden by building on top of modeler-created sources of weak supervision. We take inspiration from them in the form of our keyword features (See Section
Material Classification Datasets The largest existing material datasets are OpenSurfaces (Bell) al. 2013). which has 20k images with per-pixel labels from 36 material categories, and the Materials in Context Database (MINC) (Bell et al., 2015), which has 436k images with 23 categories. Other datasets with materials are relatively small but focus on a variety of domains, such as fabric,
photography, and objects (Sharan et al., 20144; Gao et al., 2021 2022 Collins et al., 2022 Bell et al., 2014] Wang et al., 2016;; Kampouris et al. 2016 Sumon et al. 2022).
J. In contrast, we have over 2.8 capture materials that are not necessarily visible on the surface of an object, such as copper inside a wire.
million Amazon product fistings with labels from a taxonomy of 182 materials. Morcover,
Attribute datasets The MIT-States and UT-Zappos (Isola et al. 2015 Yu & Grauman 2014 2017)
datasets include object attributes, but are relatively small and focus on crowdsourcing the labels.
3
THE EMMA DATASET
EMMa is large-scale, multi-task, multimodal, dataset of over 2.8 million objects. See Figure 1 First we diseuss the data sources and our data cleaning process. Then, we present the diverse data attributes accompanying each object in our dataset. Finally, we summarize some statistics of our dataset.
3.1 DATA SOURCES
We collected data from three sources. We started with the Amazon Selling Partner API (ASPA), from which we collected about million items. Because of the QPS limit, this took about three months. We augmented this data with other publicly available datasets. We used the raw image2mass dataset, which has about 3 million listings, and the UCSD product rating dataset, which has about 15 million listings. In all three data sources, each entry is keyed by the Amazon Standard Identification Number (ASIN). Unfortunately, each data source was incomplete. For example, the UCSD data often lacked weight, materials, and size information; the ASPA data lacked product descriptions, features, categories and ratings; and the image2mass data lacked descriptions, categories, and ratings. Moreover, each entry from the image2mass and ASPA sources had only single image. To overcome this limitation, we join these datasets. This ensures that each entry has as all the available information.
This resulted in approximately 18 million entries, which were aggressively filtered through several steps (detailed in the Appendix). The entire filtering process resulted in a dataset with 2,883,698 instances. This partitioned into 2,806,806 training instances, 26,535 validation instances, and 26,941 test instances, and 23, 33,416 calibration instances. The test and validation instances were chosen from the set of instances that have all attributes filled-in and have more than one image and more than one review. Furthermore, more aggressive duplicate-deteccioo was applied to these sets to ensure they don't 't contain objects that are similar to objects in the training set. Product listings from the image2mass Amazon test set were excluded from the training set so that we can compare with their results below.
3.2 DATA ATTRIBUTES
EMMa contains the following attributes:
Images Every product listing contains one more product images. There 7,389,213 images total, an average of 2.56 images per listing (though the distribution is skewed). We use the original resolution (500x500) in our models. In this work, never treat images as output; however,
3

---

Published as a conference paper at ICLR 2023
think that the dataset could be useful for image generation. We also provide representation of the images in each listing by taking the vector sum of the ConvNeXT-LLare Liu et al. 2022) features for all images in the listing.
Text-Every Every product listing contains a title. Titles can be surprisingly descriptive, often stating materials, object categories, or uses. 88% of our listings contain one more bullet points, i.e., Product Features. Most of the textual information about a product is contained in these features. 63% of our listings contain a description. Descriptions can describe the product, the brand, or even repeat information already contained in the features. Together, the title, features, and description are concatenated to create the input for our text models. We provide a representation from an all -mpnet- b (Hugging Facee [22TT) of this text as part of the dataset. We think that this dataset could also be useful for captioning by using the text as an output, although we don't t explore this in the present study.
Weight (Mass)-77)) of our final dataset contains the product's weight.
Price -6% of our final dataset contains price information. Prices are missing for some products because some listings remain on the website even though the products are no longer available for purchase, and the price will be unavailable in such cases. When multiple data sources contained different prices for the same product, the median price was used.
Materials- -30% of our product listings contain materials. The raw data for materials is simply a list of seller-entered strings. Sellers can put anything into the materials entry, and so it had to be heavily cleaned and processed. We developed taxonomy of 182 material types shown in Figure 2 We used heuristics to match seller-entered materials strings to nodes in the hierarchy when possible in order to create the final materials labels. Items can be made out of multiple materials. Very often, it is not known which leaf node an object has, but rather only an internal node. For example, we may know that an object is made of plastic, but not which kind of plastic. Special attention was given to the quality of the materials label. 11% percent of the entries in the calibration set were manually labeled, focusing on the most difficult instances. We use this calibration set to help train and calibrate the probabilities of our materials models. Furthermore, we applied the smart-labeling process (Section 5) to 19 of the material nodes (nylon plastic, nylon fabric, cardboard, cardstock, diamond, liquid, electronics, food, gold, genuine leather, lace, marble, faux-fur, cherry, leather, fabric, wood, metal, and plastic) to refine the the labels for every object in the dataset.
Category-7000 70% of our final dataset contains product category, which specifies the position of product in Amazon's 's product taxonomy, e.g., Home & Kitchen -> Kitchen & Dining - Cookware + Pots& & Pans + Woks & Stir-Fry Pans. The hierarchy implied by the data contains 11,592 nodes at a maximum depth of 6. Even though Product Type and Category are both semantic classifications of the objects, they have surprisingly little overlap. Preliminary experiments show that models trained to predict one don't 't benefit much from the inclusion of the other as an input.
Size information Most of the final dataset includes some form of size information, such item dimensions or package dimensions. We used this data as an input for training models to predict mass as well models to fill in missing data. However, we do not predict size information task.
20 Sole
feam
Seel
coda
Wood
Plastic
Other Wood
Softwood
Hardwood
Elaastmmn
Fibergllass
Thermosetting Plastic
Thermoplastic
-ther -5thrrTS TSP
SSllicne
Resin
Polyresin
-eooprenee
-Melamine
Polyurethane
Other
Nylon Plastic
PET
Polycarbonal
Polypropylene
ABS
EVA
Polyethylene
Tritan
Acrylic
-PLA
TPU
-Engineered Wood -Ma Manufactures Wood
-Cork
-Laminate
Plywood
MDF
-Other Softwood
Rediwood
-edar
Pine
Other Other Hardwood
Basswood
Rubberwwoo
Walnut
-Ma Maaaggyy
-Birch
Map
-heeyy
Rosewood
-Beechwood
-Ok
Figure 2: Our materials taxonomy. Nodes were chosen to capture most of what Amazon sellers were writing in the materials field. The plastic and wood branches are enlarged for legibility, See the Appendix for our materials
taxonomy design philosophy.
4

---

Published as a conference paper at ICLR 2023
Reviews and ratings -55% of the listings have least extracted review. Each review comes with product rating. We predict these ratings as a task and use all-mpnet-base-v2 to extract features from reviews to as input.
3.3 DATASET STATISTICS
Figure 3 shows the diversity of the dataset. The price distribution shows good spread of both inexpensive and expensive items. About 9% of objects in EMMa cost more than $200. The objects also have wide variety of masses, with many objects weighing less than tenth of a pound and many weighing more than 100 pounds. Moreover, the dataset contains objects with diverse set of materials. 78 nodes the materials taxonomy have more than 10k instances labeled with them, and 91 % of the nodes have more than 500 instances labeled with them. On average, each object has 4.9 material labels. Furthermore, 1,049 category nodes have more than 1,000 listings. This subset the data could be used to compare with ImageNet Ik. Finally, it's rare for category to be dominated by a subcategory. On average, a node's s largest child contains only 53% of the product listings within node.
Count
Count
:0c
20c
000
10
01.0
Price
Mass
Clothing Slices & Jevelty
Home Kitchen
Spons Outdoors
Touls & Home Improvement
Cell Phoness Accessurien
Kitchen Dining
Dear
Grang Intionn
Shoes
Jevelry
HP
Basic Cases
Maui
Purpa
Ocdival Reeraddor
Clothing
Bedding
Bath
Cyclig
Toys Gemes
Pate Laon Ganden
Arb Duala Covn
Novelty
canes
Jewelty
Clotting
Electronics
Replacement Parts
Pot Supalies
Camean
Dogs
Shoes
Girls
Office Products
Radty
Bays
Vieeo
Figure (Top left) A histogram FMMa's prices. (Top right) A histogram EMMa's masses (log scale). (Bottom) A treemap showing the diversity of the categories in EMMa.
MODELS AND SYNTHETIC LABELS
Although our final dataset contains all the attributes mentioned in Section 3.2 the raw source data did not have every attribute for every object. For example, only about 875k of the listings had material information. This missing information complicates multi-task learning and reduces the amount of available supervision. However, missing attributes can often be accurately inferred from the available attributes, especially the product listing text. For each of the possible missing attributes except for size (price, mass, materials, categories, and ratings distribution), we trained a model to predict that attribute given all other available attributes. We these models to fill in the missing information.
5

---

Published as a conference paper at ICLR 2023
Price (MnRC) 1
Mass (MnRE)
Materials (F1)
Categury (Ace) f
Ratings (KL-d)
SINGLE IMAGE GEN I
0.649
0.640
60.9
82.6
0.155
SINGLE IMAGE GEN 2
0,653
0.649
72.7
84,5
0,153
TEXT GEN
0.710
0.693
79.8
91.6
0,148
TEXT GEN
0.704
0.695
85.3
91.9
0.147
EVERYTHING GEN
0.736
0.773
80.0
92.1
0.138
EVERYTHING GEN 2
0.743
0.788
86.0
91.9
0.136
The performance of our multi-task text and image models compared with the performance of the models that take all available data. MnRE (Standley al. 2017). F1, Acc, KL-d denote the minimum ratio error, Fl score, accuracy, Diveerggeeee, respectively. lower better, f higher better. Since we use the Everything models to fill in missing data, the performance we see here on the test set is the accuracy we should expect for the synthetic labels. See the Appendix for discussion about metrics and losses.
Table 1:
These models were trained using late-fusion strategy. We fine-tuned a computer vision model (ConvNeXT base pretrained on ImageNet 22k (Deng et al., (009)) to simultaneously predict price, mass, materials, categories, and ratings from the product images. Likewise, we fine-tuned an Transformer model (all-mpnet-base-v2) to predict the same attributes from the product listing text. We then extracted and standardized feature embeddings from these two models (both are vectors of length 768). Finally, we trained task-specific models that take in these standardized embeddings as well as all the other available attributes. Early experiments showed that this strategy resulted little to no loss in prediction quality compared to end-to-end training while being much more tractable. Moreover, this strategy produced useful embeddings and allowed faster model iteration. Each task has specific decoder architecture that was manually designed. See the code for details. See the Appendix for technical details about model training and task losses.
Assuming that the predicted labels are at least somewhat accurate, models and make even better predictions. In turn, with better predictions we can train better models. This is form of sspectation-maxmizzet on the original data as generation 1, and the models trained on the dataset with missing attributes filled in generation 2. Preliminary experiments showed diminishing returns after generation 2.
use them to retrain the
(Dempster et al. 1977). We refer to the models trained
Table T shows the performance of all of our models on the test set. As expected, models that take images, text, and the other attributes into account do better than the models that take just a single image or just the product listing text. Furthermore, we see a general improvement for most tasks when working with filled in data (Gen 2) on all three types of models. Finally, we note that the text models do a surprisingly good job capturing the relevant information. This is particularly surprising in the case of mass, which is a physical property that is usually not present in the listing text.
We use these models to produce synthetic labels to fill in the missing data. For the specific case of the materials task, the Amazon listing may only specify that an object is made out of mid-level material in the taxonomy (e.g., metal). These models can predict probability distribution over the types of metal, i.e., steel [0.4], aluminum [0.6], etc. This allows us to fill in probabilistic estimates for the types of metal, leading to training label that is full distribution over the 182 materials in our hierarchy. We use an iterative algorithm to enforce that these probabilities make sense, i.c., the probability of parent must be between that of its highest child and the sum of its children's probabilities. The parent's probability need not be equal to the sum of its children's 's probabilities because product can contain more than one material subtype (e.g., chess board can be made of both cedar and pine). Our iterative algorithm also fixes inconsistencies between smart-rated material labels and other labels.
For mass prediction, we compare our image model to the the model from (Standley et al. 2017) in Table 2 Our model outperforms theirs on their publicly available test set despite lacking a "geometry module," though we are using more recent backbone network (ConyNeXT-Baal vs 2x Xception
Our Amazon Test Set
IMAGE2MASS Amazon Test Set
IMAGE2MASS OURS
0.672
0.649
0.709
Table 2: The performance of our image model for mass predictions compared with the model from IM- AGE2MASS (Standley et al. 2017).

---

Published as a conference paper at ICLR 2023
(Chollet 2016)) and we have larger dataset. Our model doesn't need to be isolated pure white background and is predicting all of our other tasks in addition to mass. According to the study in the image2mass paper, this model performs better than humans.
Filling in missing information not only helps to produce better models, but also improves the object embedding we train in S If an attribute such as mass were to be missing in a bottleneck model, we would have to imput a fixed value (such as zero) and the model wouldn's tt be able to distinguish that from an object with that fixed mass.
EXTENSIBILITY, SMART LABELING, AND OBJECT EMBEDDING
One of the major problems in modern machine learning is collecting sufficient high-quality data to train a desired model. A common process in academia involves the use of a crowdsourcing platform such as Amazon Mechanical Turk. This is costly and comes with whole host of other problems, not least of which is the difficulty of communicating exactly how one's 's data should be labeled. In order to overcome both the cost and communication problems, we outline our Smart Labeling process. This process leverages active learning as well as two key properties of our dataset, the ability to use all data about each object to create powerful but compact object embedding, and the text itself.
Object Embedding Leveraging all known and filled-in attributes for every object, developed powerful object embedding. This embedding is a vector of length 2048 that captures most of what we know about product listing and is trained using simple network with single hidden bottleneck layer. The input is standardized vector containing text and vision embeddings in addition to mass, price, materials, categories, and ratings (Figure iillustrates Dillutaeil our architecture). The output was trained to match the input using L2 distance and achieves an average standardized L2 of 0.0054, which implies that each input feature is reconstructed to within 7.3% of its variance on average. More details in the Appendix.
Smart Labeling Framework We built Smart Labeling tool (see Figure for screen- shot) and developed a process for quickly gener- ating high-quality and high-impact training data for adding a new object property to our dataset.
Features (optonal) with an
To add task specifl features (eig taak reliteeelidd words)
Done adding reatures?
training set with number posttiva and negaiive ecemples inject important yyoes objects neglected sclive learning
The goal of the framework is train an accu- rate linear logistic regression model. Predictions from that model can be used to label the rest of the dataset.
Discaver a neglacted class objedss?
Found 50 posilives and 50 negatives?
The discriminatory power of any linear model is limited by the input features. This is why it is so important to train on top of a powerful object embedding like the one we developed. But as powerful as the embedding is, any fixed set of features can be limitation. Therefore, in ad- dition to the embedding, we support using any number of custom, task-specific features that can be added to increase discriminatory power as input. These task-specific features can be added using the Smart Labeling tool and can simply be derived from keywords related to the desired property (see below for details). The Smart Labeling tool can be used in one of sev- eral modes: features (for adding task-spec features), word-search (for seeding the training set with small number of positive and negative examples), active-learning (for fine-tuning the decision boundary), correction (for finding and correcting errors in the model), review (for finding and correcting labeling errors made in the previous steps). A flowchart of this process is presented as Figure
Active Leaning
To fine-tune the labeling model's decision boundery
Tool is soking aboud distinsiooss too suble to maiter?
conrect postives false negatives should be targeted)
Errors found?
Correction
Good precis on and coverage?
correct labeling emors made previous steps
Resiew
No errors found?
Figure A flowchart showing progression of the Smart Labeling process. This chart should be treated as guideline and need not be followed exactly.
7

---

Published as a conference paper at ICLR 2023
As an example, let's say that we want to add the is container property to the dataset. First, features mode to add as many container-lik like words as possible, which we use as binary text ext-magthing features. Each feature is positive iff the listing text contains the corresponding word. They can be added or modified at any time during the Smart Labeling process. They are not necessary, but it often helps start with some. We'll add cup, mug, box, container, holds, basket, leak, bottle, jar, bag, flask. Conceptually, these can be regular expressions, or even arbitrary functions that take an object as input and produce a real-valued output somewhat related the property (like in Snorkel), but we' ve only tried simple text matching strings.
Then, we use word search mode to create the initial training set. We can start with the word "basket". The system will search through listing texts for the word "basket" and show matching product images to the user in a random order. The user can quickly mark all objects that are actually containers as positive instances (left-click) and all the clearly non-container objects as negative instances (right- click). The user can also middle-click on any product image be taken to the Amazon product listing page. In this phase, it's best to skip any object for which it is not easy to determine the label. More matching products can be brought up by clicking to the next page. New search terms can be entered until the initial training set is large enough. Ideally, at least 50 positives and 50 negatives are collected, but we don't 't need more examples because active learning mode is more efficient for improving accuracy. Nevertheless, we can return to word search mode any time if, for example, we notice that the model is failing to surface a certain class of objects for labeling.
Next, we move to active learning mode, for which we use uncertainty sampling (Lewis & Gale 1994). A logistic regression model is trained on the existing labeled data, and predictions are obtained for a sample of unlabeled candidate objects. The objects with predicted probabilities closest to 0.5 are then presented to the user. The user can then label any or all of these and advance to the next page, at which point the logistic regression model will be retrained and a new candidate set will be chosen and presented to the user. After labeling several pages, it's common to start encountering corner cases, such as, "is phone case a container?" helps to record the decision to ensure consistency. This phase tends to take the most time per object because the system quickly hones in on the difficult cases. When most objects presented by the tool either cannot or need not be labeled (because their labels don't 't matter for the use case), it's time to move on to correction mode.
mode can be used to improve either precision or recall. This mode works just like active learning mode, except that instead of presenting the user with objects that have probability near 0.5, it presents users with objects that have currently predicted probabilities within a user-specifiable range. If the range is 0.5 1.0, then the system will present the user with objects that are currently classified as positives. In this case, the user should focus on correcting false positives and can ignore correct difficult items. Correcting even few false positives can have a large positive impact model performance. On the other hand, setting the probability range to 0.0 - 0.5 allows a user to correct false negatives. In practice, many of the properties a user might want to add to the dataset are rare (Bommannavar et al., 2014). In this case, it might be more useful to set the range to be something like 0.1 0.5, because otherwise the false negatives might be difficult to find.
Correction
Finally, review mode lets the user review the data they have already labeled. In review mode, 20-fold cross-validation is used to find how likely each example is to be mislabeled. Instances are presented to the user in order of descending score. This allows user to quickly find a (hopefully) small number of labeling errors that they may have made and correct them. Since we are using active learning, labeling errors have large effect prediction quality. Users should return to active learning mode they find labeling errors.
When precision and positive rate are sufficiently high for the intended use case, we can use the model to predict a probabilistic label for every object in the dataset.
Smart Labeling Evaluation We added five custom properties to the dataset using Smart Labeling: Sharp, Transparent, Deformable, Articulating, and Electronic. See the Appendix for property definitions, manual labeling criteria, and the text-matching features we used. See Figure 5 for some qualitative examples of products labeled by our models. We also created a test set for each class. Unlike the training set, the test set was sampled uniformly from the set of all products. Sample products that could not be manually labeled with confidence were discarded. Table shows performance metrics for each property. F1 and accuracy scores in the 90s were achieved on all five
8

---

Published as a conference paper at ICLR 2023
# Train
# Train
# Test
Test
# Errors
Precision
Recall
F1
Acc
SHARP
1331
633
1504
25
4
92.0
92.0
92.0
99.7
TRANSPARENT 2217
877
706
46
6
93.5
93.5
93.5
99.1
DEFORMABLE
636
350
253
150
9
96.7
97.3
97.0
96.4
ARTICULATING
1399
403
18
94.6
94.0
BLECTRONIC 525
248
301
63
5
96.8
95.2
96.0
98.3
Table 3: Performance of using the Smart Labeling technique to add five properties to EMMa. The sizes of the test and training sets are shown, as well as the number of positive examples in each set (# Train +, etc.). The numbers of errors, precision, recall FI, and accuracy are shown.
Sharp
Transparent
Deformable
Articulating
Electronics
Figure Some qualitative examples products with highly positive inferred labels. More examples can be found in the Appendix, along with examples of base dataset attributes, such as heavy and expensive.
tasks. We see that rare properties can also be easily annotated. The uncertainty sampling approach helps to balance the number of positive and negative examples given to the model.
Smart Labeling Advantages By labeling their own data, a practitioner overcomes communication friction, can leverage their own domain-specific expertise, and can tailor the labels their specific use case. For example, a roboticist could annotate which objects their particular robot could likely pick up. As another example, whether certain objects should be considered "sharp" depends on the specific use case: hole puncher may be considered sharp if deciding which objects can cut string and not sharp if deciding which objects can be wielded safely by a robot. Often, these corner cases are hard to predict priori and are best handled when they appear in the labeling process. Such corner cases are likely to appear in the Smart Labeling process due the use of uncertainty sampling.
Another advantage of the Smart Labeling technique is that the predicted probabilistic labels are more powerful than standard binary labels. Annotation errors are likely to be less certain, which means that any model learning from these annotations won't 't devote many resources to match them.
Our framework for creating new labels efficiently can be useful in other situations as well. For example, used similar process with image embeddings detect and filter out unhelpful product listing images, such as those of the packaging or of warranty labels.
In order to allow EMMa to expand and improve, we will host not only the core dataset, including our manually added properties, but also any properties that the community develops for EMMa and wants to share. This way, both the number of object properties and the ease of adding properties will grow over time and EMMa will become increasingly useful to machine learning practitioners.
CONCLUSION
FMMa is an object-centric, multimodal, and multi-task dataset of over 2.8 million Amazon product listings with rich annotations. EMMa's objects are annotated with material labels from compre- hensive taxonomy of 182 material types. We also propose a Smart Labeling framework that allows practitioners to accurately add new binary properties with only hours of effort.
ETHICS STATEMENT
We are committed to addressing potential ethical concerns in our dataset. We note that Amazon actively enforces its community standards product listings and reviews. We rely this to ensure

---

Published as a conference paper at ICLR 2023
dataset does not contain overt racism, foul language, nudity, illegal products, and harassment. We also employ Smart Labeling to mark images with human models or products sexual or religious nature, they can be filtered when necessary. In addition, smart-labeling empowers practitioners to clean and re-balance the dataset for their own ethical purposes. For example, practitioners can use Smart Labeling to annotate the gender of human models and then use stratified sampling re-balance the distributions during training to ensure that no gender is underrepresented.
ACKNOWLEDGMENTS
The work is in part supported by ONR MURI N00014-22-1-2740, NSF CCRI #2120095, NSF RI #221 1258, AFOSR YIP FA9550-23-1-0127, the Stanford Institute for Adobe, Amazon, and Meta.
AI (HAI),
REFERENCES
Iro Armeni, Zhi-Yang He, Joung Gwak, Amir R Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3d scene graph: A structure for unified semantics, 3d space, and camera. Proceedings of the IEEE International Conference on Computer Vision, pp. 5664-5673, 2019.
Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. OpenSurfaces: A richly annotated catalog of surface appearance. ACM Trans. on Gruphics (SIGGRAPH), 32(4), 2013.
Sean Bell, Kavita Bala, and Noah Snavely. Intrinsic images in the wild. ACM Trans. on Graphics (SIGGRAPH), 33(4), 2014.
Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. Material recognition in the wild with the materials in context database. Computer Vision and Pattern Recognition (CVPR), 2015.
Praveen Bommannavar, Alek Kolc7, and Anand Rajaraman, Recall estimation for rare topic retrieval from large corpuses. In Jimmy Lin, Jian Pei, Xiaohua Hu, Wo Chang, Raghumath Nambiar, Charu C. Aggarwal, Nick Cercone, Vasant G. Honavar, Jun Huan, Bamshad Mobasher, and Saumyadipta Pyne (eds.), 2014 IEEE International Conference on Big Data (IEEE BigData 20/4), Washington, DC, USA, October 27-30, 2014, pp. 825 834. IEEE Computer Society, 2014. doi: 10.1 109/ BigData.2014.70000
URL wttp::////oooo///...oo/.// o0g/10..110000.... /ttp///....o.11111.1//////aaaaa 201 4.114..00 7004312
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing deep multitask networks. CoRR, abs/1711.022577, 2017. /tll:ttpp//aaxxv.cc///////
17117111111 C2257
François Chollet. Xception: Deep learning with depthwise separable convolutions, 2016. URL
http://arxiv. http://aaxiv..orgab///1l... 02357 cite arxiv:1610.02357.
Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, Matthieu Guillaumin, and Jitendra Malik. Abo: Dataset and benchmarks for real-world 3d object understanding. CVPR, 2022.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B, 39:1-38, 1977. URL http::/weet /wwp://wwbb.... mit edu/ 6. 435/ www wtwp/Dempster77.p.g
Deng, Dong, R. Socher, L. Li, Li, and I.. Fei-Fei. ImageNet: Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Matthijs Douze, Arthur Szlam, Bharath Hariharan, and Hervé Jégou. Low-shot learning with large-scale diffusion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3349-3358, 2018.
10

---

Published as a conference paper at ICLR 2023
Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-tast mid-level vision datasets from 3d scans. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10786-10796, 2021.
Ashok Elluswamy and Andre Karpathi. Tesla ai day, 2021.
URL https:
50z4FweCy4M
Ruohan Gao. Yen- Yu Chang, Shivani Mall, Li Fei-Fei, and Jiajun Wu. Objectfolder: dataset of objects with implicit visual, auditory, and tactile representations. In CoRL, 2021.
Ruohan Gao, Zilin Si, Yen-Yu Chang, Samuel Clarke, Jeannette Bohg, Li Fei-Fei, Wenzhen Yuan, and Jiajun Wu. Objectfolder 2.0: A multisensory object dataset for sim2real transfer. In CVPR, 2022.
Chen Gong, Dacheng Tao, Wei Liu, Liu Liu, and Jie Yang. Label propagation via teaching-to- learn and learning-to-teach. IEEE transactions neural networks and learning systems, 28(6): 1452-1465, 2016.
HuggingFace.
all-mpnet-hase-v2, 2021.
URI.
https://hugginggfccs co/
m11-mpnet tasppeet-base--e v2
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi- supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5070-5079, 2019.
Phillip Isola, Joseph J. Lim, and Edward H. Adelson. Discovering states and transformations in image collections. In CVPR, 2015.
Christos Kampouris, Stefanos Zafeiriou, Abhijeet Ghosh, and Sotiris Malassiotis. Fine-grained material classification using miero-geometry and reflectance. Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.). Computer Vision - ECCV 20C6, pp. 778-792, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46454-11
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision, 128(7):1956-1981, 2020.
David D. Lewis and William A. Gale. A sequential algorithm for training text classitiers. In Pro- ceedings of the 17th Anmual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '94, pp. 3 3-12, Berlin, Heidelberg, 1994. Springer- Verlag. ISBN 19881X889X..
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Zhuang Liu, Hanzi Mao, Chao-Yurue Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Yong Luo, Tongliang Liu, Dacheng Tao, and Chao Xu. Multiview matrix completion for multilabel image classification. IEEE Transactions on Image Processing, 24()):2355-2368 2015.
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rghd images. In ECCV, 2012.
Jianmo Ni. Jiacheng Li, and Julian McAuley, Justifying recommendations using distantly-labeld reviews and fine-grained aspects. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 188-197, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10.18653/v 1/D19- 1018. URL arg/D19-1011
11

---

Published as a conference paper at ICLR 2023
Alex Ratner, Braden Hancock, Jared Dunnmon, Roger Goldman, and Christopher Ré. Snorkel metal: Weak supervision for multi-task learning. In Proceedings of the Second Workshop on Data Management for End-To- To-End Machine Learning. DEEM' 18, New York. NY, USA, 2018. Association for Computing Machinery. ISBN 978 9781450358286 doi: 10.1 00.1145/3209888.2209888
htthhhtss:///oi... www.oi.orr//oo 10.
3209898
Alexander Ratner.
Stephen H. Bach, Henry Ehrenberg. Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. Proc. VLDB Endow., 11
(3)22692882, nov 2017. ISSN 2150-8097. doi: 10.14778/115799431557979 URL https?
//6ol..rg/10011177//1551111 3157797
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. Advances in neural information processing systems, 30, 2017.
Lavanya Sharan, Ruth Rosenholtz, and Edward H. Adelson. Accuracy and speed of material categorization in real-world images. Journal of Vision, 14(10), 2014.
Trevor Standley, Ozan Sener, Dawn Chen, and Silvio Savarese. image2mass: Estimating the mass of an object from its image. In Sergey Levine, Vincent Vanhoucke, and Ken Gold- berg (eds.), Proceedings of the Isl Annual Conference on Robot Learning, volume of Proceedings of Machine Learning Research, pp. 324-333. PMLR, 13-15 Nov 2017. URL httpps:/pproceedings m
Borhan Uddin Sumon, Damien Muselet, Sixiang Xu, and Alain Trémeau. Multi-view learning for material classification. Journal of Imaging, 8(7), 2022. ISSN 2313-433X, doi: 10.3390/
jimaging8070186. URL/ttps:: wwww//wwwwwww. mdpi. com 2-13 4031433333//////
Bo Wang. Zhuowen Tu, and John K Tsotsos. class lubti-label classification. Proceedings of the IEEE international conference on computer vision, pp. 425-432, 2013.
Dynamic label propagation for semi-supervised multi-
Ting-Chum Wang, Jun- Yan Zhu, Ebi Hiroaki, Manmohan Chandraker, Alexei A. Efros, and Ravi Ramamoorthi. A 4d light-field dataset and enn architectures for material recognition. In Bastian Leibe, Jini Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision - ECCV 2016, pp. 121-138, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46487-9.
A. Yu and K. Grauman. Fine-Graing Visual Comparisons with Local Learning, In Computer Vision and Pattern Recognition (CVPR), June 2014.
Aron Yu and Kristen Grauman. Semantic jitter: Dense supervision for visual comparisons via synthetic images. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Amir R. 7amir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2018.
A APPENDIX
Filtering and Data Cleaning After merging based on ASIN, we had approximately 18 million objects. However, the data still contained many duplicates, consisting of the same product being sold under different listings. This was especially true for the UCSD data. When duplicates or near-duplicates were detected, we only kept the entry with the most attributes filled in.
Duplicate detection was accomplished using the pretrained NLP embeddings and the pretrained ImageNet embeddings. Any two entries for which either of the two embedding types were too close, were removed.
Next, we discarded any entries without titles or without images. Furthermore, any entry lacking mass, materials, and category was discarded (if one was present, the entry was kept).
12

---

Published as a conference paper at ICLR 2023
Then, we built model to detect entries for which the listing text did not match the images and removed those entries. Finally, some entries had only generic images that don't show the product and other entries had physical form, such downloadable software. We used an early version of Smart Labeling (Section 5) to find and discard such problematic entries.
Unfortunately, the data sources that a seller will occasionally replace the product listing information for one product with the information of another they hope to sell, perhaps piggybacking the positive reviews of the original product. We used heuristics to detect this and ensure that merged entries with the same ASIN were indeed for the same product. Otherwise the entry was discarded.
collected at different times (from 2017 2021) and found
F
Figure 6: screenshot of our Smart Labeling tool's UI during the lableing of hypothetical class, is container. Red entries are manually marked not in the class. Green entries manually marked in the class. Unmarked entries yet be labeled remain unlabeled.
Labeling Criteria The following are the labeling criteria we used for the Smart Labeled properties from section 5) We tried to include numerous categories with information that isn' 't obviously contained within the Amazon data.
Sharp Objects are considered sharp when they are used to cut or poke holes into things. Objects with internal sharp parts, such craft hole punchers not considered sharp. We used the following keyword features, "needle, knife, blade, razer, scissors, pin, sharp, dull, knitting, serrated, saw, sharpener" for the sharp labeling process.
Transparent-Objectt are considered transparent if they can be seen through to the other side or to objects placed inside of them. For example, typical watch is not considered transparent because you can only see the watch face through the glass, but picture frame is considered transparent because you can put picture inside it. Translucent objects are not considered transparent. As is often the case for such classes, there's a continuous gradient from transparent to translucent where we had to use our own judgement.
general, details must visible through the object for be considered transparent. We used the following keyword features, "needle, knife, blade, razer, scissors, pin, sharp, dull, knitting, serrated, saw, sharpener" for the transparent labeling process.
Defornable- Objects are considered deformable if typical use-case force can change their shape. Fabries are considered deformable, as are molding clays, gels, bendable wires/springs etc. We used the following keyword features, "shirt, cloth, fabric, clay, bendy, silicone, rigid, hard, stiff, cotton, polyester, flex, paste, soft" for the deformable labeling process.
objects are rigid bodies connected in movable way. Things with hinges or doors are articulating. As are connected wheels or other items that have parts that rotate. Chains and linked jewelry is also considered articulating. We used the following keyword features. "glasses, box, hinge, door, chain, telescoping, adjustable, book, fold, folding, folio, twist" for the articulating labeling process
Articulating-AAticuulle
13

---

Published as a conference paper at ICLR 2023
Blectronic-Anythinng Anything with electronic components wiring is considered electronic. Lamps considered electronic, so are cables that send signals or power. Batteries or anything with a battery is considered electronic. So is anything with an AC plug. Automatic camera lenses and accessories that use power are also considered electronic. Finally Auto-parts are considered electronic if they use electricity, however, these were very difficult to rate with accuracy because we do not have auto-parts expertise, so many of these were left blank and not considered in either the train or test sets. We used the following keyword features, "power, battery, electr, automatic, smart, case, led, usb, tool, mechanical, windup, charge, automatic, quartz, Ow, watt" for the electronics labeling process.
Packaging not considered part the object, and is not considered for labeling. Generally, an object is considered positive for property if any part of an object has the property. However, sometimes when an object overwhelmingly does not have property, but has it on a small part not. For example, when shoes/clothes have zippers we still do not consider them articulating.
Losses: For the price and mass tasks, we use ALDE as in (Standley et al. 2017). For materials, we use the mean the binary cross-entropy loss over 182 material types. When data item labeled with an internal node in the materials taxonomy, the descendants of that node are excluded from the mean calculation so that models aren't penalized for being more specific than the label. For example, if an object is labeled as plastic, predictions of thermoplastic or acrylic are not penalized. For categories, we use a cross entropy loss at each level of the taxonomy. We take a weighted mean of the loss at every level of the taxonomy. In other words, it's more important for the prediction to match at higher levels in the taxonomy. Furthermore, we only consider predictions made for the correct parent at any given level. In other words, if an object is in the Home & Kitchen - Cookware + All Pans category, we do not consider the model predictions for Automotive > Replacement Parts level and we do consider the predictions for Home & Kitchen - Bath + Towels for level 3. For ratings, we use KL-divergence and weight each product listing by the log of the number of reviews it has.
For the Price and Mass tasks, we report the Min Ratio Error (MnRE) defined in (Standley et al. 2017). MrRE= min( ( }) where is the prediction and t is the ground truth. This metric is nice because perfect predictions result in a score of 1.0 and the worst possible predictions have a score 0.0. For materials, we report the average score of an object is the average Fl score of each material classifier. For categories, we report the accuracy of each predicted class averaged first over level in the hierarchy and then over instances. For Ratings, simply report the loss KLL-Diveggenee), though note that the Pearson's 's R² between the average rating and the predicted average for an object also improves during training.
Metrics:
score of each object, where the F1
Training Details All models were trained using Pytorch and the Adam optimizer. All models used batch size of 512. All gradients were clipped to 32. Task losses were combined using weighted average. The weights are in Table 4 Text and vision models use an encoder-decoder architecture. For both, the encoder is frozen during the first epoch. Text and vision models were trained on 8xTitan RTX. Each took about three days to train. Everything models were trained on a IxTitan RTX wokstation in about 12 hours. Each model type has its own learning rate schedule and weight-decay, which are listed in Table 5
Price
Mass
Materials
Categories
Ratings
5.0
5.0
140.0
1.0
1.0
Table 4: The loss weights used for Text and Vision models. This should not be confused with task importance
because losses have different scales, Weights initally chosen using GradNorm (Chen et al. (001)) and then hand tweaked from there.
Other Useful Data Fields fields:
In addition to the fields described Section 3.2
we extract the following
Product Type-Ewee product listing contains a Product Type designation because it is required to start listing. The product type is used by the seller UI determine what information to ask about.
14

---

Published as a conference paper at ICLR 2023
Tex(TaAll
VisionTeAll
AllToPrice
AllToMass
AllToMaterials
AlfToCategories
AllTonatings
Weight Decay
0.01
0.001
0.1
0.1
0.03
0.01
0.02
Initial learning Rate
0.001
0.002
0.002
0.002
0.02
0.02
0.0002
Epochs
50
50
100
100
100
100
100
Table 5: Manually tuned hyperparameters for our models. Initial learning rates were reduced by a factor of 100 during the course of training.
Examples include PILLOW and WIRELESS ACCESSORY, There are 1,296 unique classes, but their counts are heavily skewed. Product Type does not seem to be exposed in the buyer user interface.
Package Weight-859% of our final dataset contains the package weight, which includes not only the weight of the product, but also the weight of the packaging. We feed this information to the models that we use to fill in missing attributes. It's particularly informative for filling in a product's weight. Sometimes the Package Weight listed is less than the listed Weight, which indicates an issue with one or both labels.
Object Embedding Figure Segure[shows the architecture for creating object embedding. The input a 4,542-dimensions vector composed of the image and text embeddings mentioned in Section 4 the original pre-trained image and text embeddings mentioned in Section as well as price, mass, materials, ratings, 128-bit embedding for the product type, and a 128-bit embedding for the product category. Our object embedding is the standardized activation of the bottleneck layer. We use I.2 loss to train our network. We use a batch size of 512 and learning rate of 3e-3 to 3c-7, approximately halving every 4 epochs. We train for 50 epochs. We achieve final loss of 0.0054, which implies that each input feature is reconstructed to within 7.3% of its variance on average.
Using this embedding, we can reconstruct the category with 99.8% accuracy, the materials to an Fl of 98.45. The mass to a Pearson's 's R² of 0.998 correlation with the ground truth, the price to a Pearson's 's Ra of 0.960.
Synthetic label Sanity Check In order to sanity check our synthetic labels, we randomly chose 100 products from the training set that initially lacked category labels. We examined the synthetic labels and found that 82 of those labels were either perfect or very good descriptions of the product. We repeated this experiment with random sample of objects that included categories from Amazon. We found that 86 of those had good descriptions.
Materials Taxonomy Design Choices The taxonomy was designed in data-driven way. Many entries in the raw datasets might not be considered materials. For example, 'plush' was present in thousands of products" materials entries. We decided to include materials like these for number of reasons. First, many of these imply their parents, i.e., microfiber implies fabric.
Second, philosophically. most materials we think about are not pure elements, compounds or mixtures. Leather and wood, for example, are about the configuration of molecules, not the base molecules themselves.
Finally, we can envision uses for knowing about these types of "materials". For example the weave of expensive than others, that has implications for price and
fabric. Some weaves are denser mass, and may even be a useful signal for custom features.
15

---

Published as a conference paper at ICLR 2023
Input (4542)
Output (L2 loss) (4542)
ImageNet Trained ConvNext- Base Features (1024)
ImageNet Trained ConvNext-Base Features (1024)
Pretrained all-mpnot- base -2 (788)
Pretrained all mpnett base .2
Features (768)
Multi-takk Finetuned
Linear Layer
Bottleneck (2048)
Linear Layer
Multi-task Finetuned
ConvNext--Sma Features (768)
(768)
Multi-task Finetuned all-mpnet-bse--22 Festures (768)
Multi-task Finetuned all-mpnet-base-2 Features (758)
Finetured all-nipnet-base-22 Review Features
Multi-ask Finetuned
all-mpnat-base-vz Review Features (768)
Categories Embedding (128)
Categories Embedding (128)
Type Embedding (128)
Type Embedding (128)
Materials Vector (182)
Materials Vector (182)
Price, Mass, and Review Distrabution (8)
Mass, Review Distrabution (8)
Figure 7: Our architecture for creating the powerful object embedding.
16

---

Published as a conference paper at ICLR 2023
Sharp
Transparent
Deformable
Articulating
Electronics
Expensive
Heavy
Metal
Wood
Skillet
Figure 8: A qualitative look at our dataset. The top S rows have high probabilities having the labeled custom property. Expensive objects are those that cost more than 300 USD. Heavy items are more than 100 pounds. Metal and Wood are material attributes. Finally, selection from the Home & Kitchen Cookware + All Pans Skillets category.
17

---

Published as a conference paper at ICLR 2023
Products Accent Decoretive Assosrto Bowh
DharmaObjects Tibetan Copper Offering
Category
Mandala Set 5.5" Diameter Title
Visa the OlarmaOheess Store
30 ratinnss
$6900
Price
vprime
Color
Brand
DhamaCbjjets
Material
Copper
Materials
Item Dimensions LoWhll
5x iicches
Size Information Mass
Item Weight
1,28 Founds
Traditional Tibetan Buddhisz offoring
sadhana,
An essential element of the tracitional Theean Buddhist altar layoue
Product Images
imarge
Features
Agoonrr grelimhary
Bosic riual Icom for mony of the Tibetan meditarion sochonas, Indudding the
from copper:
Hade in Indla, product of Tibeton refupess
Product Description
prebminaar
Description
Product Information
Customer reviews
Dimensions
Size information again
Weight
pounds
again
star
69%
Rating Distribution
star
14%
Item model number
8541854151 12
Customer Reviews
dk #r 4.1 out of stors
30 ratings
star
0%
Ulei bellers Hank
21,523220 in Home & Kitchen (bee top 200 in Home Kithhen)
star
15%
Reviews
*a lrak dir Good product fist shipping love thanks Reviowed in the United States on April 21, 2021 Premium
insd Manufacturer
No
5.5
like
Figure 9: The anatomy of product listing.
18


Reviews:
[{"review_id": "wiDu3kf090G", "reviewer": "Reviewer_AKRZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a novel multi-modal dataset based on Amazon items, with image, free text and structured fields as input or output.\nThey further propose a novel material taxonomy and propose an interactive labeling process to allow the community to add new labels to the dataset.", "summary_of_the_review": "In total the novel dataset represents a large database of object centric and related image, text and categorical data.\nThe idea of publically extending the labels is also interesting, however the question of quality control, also for the imputation conducted by the authros themselves, remains open\nAfter reading the rebuttal I better understand the semi-automatic labeling procedure and evaluation.\nWhile I am still interested in the ablation of the 3 steps (word search, active learning, correction), I think the dataset is a fruitful contributon to the community.", "strength_and_weaknesses": "Strength:\nThe dataset contains more than 2m objects collected over several month.\nThe proposed interactive labeling strategy is simple and intuitive and can be used to add various new labels creating a continously growing database for many tasks.\n\nWeaknesses:\nIt would be interesting to split the contributions/workload of the different steps word search, active learning, correction and review in a sperate ablation study to better understand the impact of each step.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aAwF-r6iNxK", "reviewer": "Reviewer_NcVD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a large-scale product dataset containing a lot of object properties such as listing text, mass, price, product ratings, and 182 material classes. The detailed collecting and implementation processes are given. Besides, a human-in-the-loop annotation system is proposed to actively add new attributes to the objects of the dataset. Given this labeling system, the authors show interesting additional annotation processes to add new attributes and verify the annotation quality quantitatively.", "summary_of_the_review": "Overall, this work proposes a large-scale dataset that would be useful for future works. Besides, the labeling process makes sense and achieves reasonable quality as shown in the paper. However, a lot of details and explanations are missing to make this dataset paper more solid. I suggest a careful revision to improve the paper's quality. It is OK for me to accept this paper.", "strength_and_weaknesses": "Pros:\n+ An extensive large-scale dataset that would be useful for many directions and studies.\n\n+ The data selection and annotation processes are described detailedly which would help the readers to use this dataset and enlarge it.\n\n+ Considering the \"materials that are not necessarily visible on the surface of an object, such as copper inside a wire\", this is very interesting and would be inspiring for future works.\n\nCons:\n- Lacking discussion about the object attribute datasets and methods (MIT-States, UT-Zap, etc.) though not all these works are related to products and are typically designed for common objects (but have some material attributes).\n\n- Lacking discussion about compositional zero-shot learning (CZSL) which is very related to this work, especially for adding new attributes to objects (attribute-object compositions).\n\n- It would be better to give a detailed bias analysis of the dataset itself and the annotation process. Because the authors use a lot of automatically labeling methods adopting DNN models. Moreover, the effect of longtail distribution is another important factor to analyze for both training and downstream applications.\n\n- A list of data license and availability analysis?\n\n- Though the proposed smart labeling system makes sense, the whole method designs are mainly common practices.\n\n- It would be more solid to give a comprehensive analysis of the data quality verification as many labels are got not done manually (a clear ratio of pseudo labels).\n\n- Lacking enough details of the models and experiments such as hyperparameters: \n\nHowever, missing attributes can be accurately inferred from the available attributes: how accurate?\n\nWhen precision and positive rate are sufficiently high: how much?\n\n- \"Even though Product Type and Category are both semantic classifications of the objects, they have surprisingly little overlap. Preliminary experiments show that models trained to predict one don’t benefit much from the inclusion of the other as an input\": why?\n\n- typo: we use uncertainty sampling(Lewis & Gale, 1994)： sampling (...)\n\n- fig 7: explain what attribute is being labeled.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "I2H5pMrIsN", "reviewer": "Reviewer_8smg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a new multi-modal dataset, EMMa, which contains 2M+ Amazon product listings with rich annotations of attributes, including images, prices, materials, ratings, etc. In particular, it contains object-centric annotations for 182 material types. Finally, it proposes a Smart Labeling pipeline that allows users to add new annotations with reduced effort.", "summary_of_the_review": "In summary, I think this is a valuable dataset with rich annotations of attributes, which can be potentially used for multi-modal or multi-task research. But I am not sure about the potential use cases and their impacts for now.", "strength_and_weaknesses": "(+) The proposed dataset contains detailed object-centric rich annotations of attributes, including detailed material types, which seem not available in existing datasets.\n\n(-) Some example use cases of this dataset could be introduced in the paper.\n\n(-) Attributes can be independent from each other. In this case, the idea of predicting the missing attributes based on the existing attributes may not be reasonable.\n\n(-) I am not sure if the proposed Smart Labeling pipeline will apply to other datasets for a bigger impact, because it seems some of its components are designed specifically for the EMMa dataset.\n\n(-) Some details of the Smart Labeling pipeline are unclear to me. Why is the feature vector 4,542-dimensional though feature embeddings from image and text are both 768-dimensional?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster