MuG: A Multimodal Classification Benchmark on Game Data with Tabular, Textual, and Visual Fields
Authors: Jiaying Lu
Yongchen Qian
Shifan Zhao
Yuanzhe Xi
Carl Yang
Venue: EMNLP 2023 Findings (2023)
Keywords: multimodal classification benchmark
resources and evaluation
multimodal graph neural network

Abstract: Previous research has demonstrated the advantages of integrating data from multiple sources over traditional unimodal data, leading to the emergence of numerous novel multimodal applications. We propose a multimodal classification benchmark MuG with eight datasets that allows researchers to evaluate and improve their models. These datasets are collected from four various genres of games that cover tabular, textual, and visual modalities. We conduct multi-aspect data analysis to provide insights into the benchmark, including label balance ratios, percentages of missing features, distributions of data within each modality, and the correlations between labels and input modalities.  We further present experimental results obtained by several state-of-the-art unimodal classifiers and multimodal classifiers, which demonstrate the challenging and multimodal-dependent properties of the benchmark. MuG is released at https://github.com/lujiaying/MUG-Bench with the data, tutorials, and implemented baselines.

MUG: A Multimodal Classification Benchmark on Game Data with Tabular, Textual, and Visual Fields
Jiaying Lu" and Yongchen Qian" and Shifan Zhao and Yuanzhe Xi and Carl Yang
Emory University [jiaying.lu, yongchen.qian shifan.zhar zhao, yuanzhe.xi, j.carlyang) @emegyyedee
name: Pidgey
Abstract
ability_1: Keen Eye
ability_ 2: Tangled Feet
Previous research has demonstrated the advan- tages of integrating data from multiple sources over traditional unimodal data, leading to the emergence of numerous novel multimodal ap- plications. We propose multimodal classi- fication benchmark MUG with eight datasets that allows researchers evaluate and im- prove their models. These datasets are col- lected from four various genres of games that cover tabular, textual, and visual modalities. We conduct multi-aspeet data analysis to pro- vide insights into the benchmark, including la- bel balance ratios, percentages of missing fea- tures, distributions of data within each modal- ity, and the correlations between labels and input modalities. We further present experi- mental results obtained by several state-of-th art unimodal classifiers and multimodal classi- fiers, which demonstrate the challenging and multimodal-dest -depeneed properties of the bench- mark. MUG is released https: //gi thub. com/luj mom/lujjan i DyyUUGGGGGGG- /MUG-Benchh with the data, tu- torials, implemented baselines.
generation: height m:: 0.3
weight_kgs 1.8
catch_rate: rate: 70
growth_ rate: Medium Slow
type (ground truth: Flying)
name: Clawfury Adept
text: Give all other friendly
characters attack this turn
attack: 2
health: 3
cost: 2
rarity; Common
type: Minion
cardClass =? (ground truth: Druid)
Illustration of data examples from MUG. Note that inputs cover tabular, textual and visual modalities. and task is multiclass classification.
Figure I:
ties, textual descriptions, and teasing pictures, thus enhaneing user search experiences and recommen- dation outcomes. Therefore, accurate classification models for tuble-text-image input are desired.
1 Introduction
The world surrounding is multimodal. Real- world data is often stored in well-structured databases that contain tabular fields, with textual and visual fields co-occurring. Numerous auto- mated classification systems have been deployed on these multimodal data to provide efficient and scalable services. For instance, medical decision support systems (Soenksen et al., 2022) utilize pa- tients' electronic health record data that contains tabular inputs (e.g., ages, genders, races), textual inputs (e.g., notes, prescriptions, written reports), and visual inputs (e.g., x-rays, magnetic resonance imaging, ct-scans) to help precise disease predic- tion. Similarly, e-commerce product classification systems (Erickson et al., 2022) categorize prod- uets based on their categorical/numerical quanti-
Deep neural networks have shown significant progress in multimodal learning tasks, such CLIP (Radford al., 2021) for image-text trieval and Fuse-Transformer (Shi et al., 2021) for subular-with-tes classification. This progress has been made with large-scale datasets provided to train the data-eager models. So far, there ex- ist many datasets (Ovalle et al., 2017; Wu et al., 2021; Shi et al., 2021; Qu et al., 2022; Lin et al., 2020; Lee et al., 2019; Kautzky et al., 2020; Srini- two modali-
vasan et al., 2021) that cover ties. However, the progress in tabular-text-image multimodal learning lags due the lack of avail- able resources. In this paper, we provide a mul- timodal benchmark, namely MUG, that contains eight datasets for researchers to examine their al-
"These authors contributed equally to this work

---

gorithms' multimodal perception ability. MUG contains data samples with tabular, textual, and vi- sual fields that are collected from various genres of games. We have made necessary cleaning, transfor- mations, and modifications to the original data to make MUG easy to use. We further conduct com- prehensive data analysis to demonstrate the diverse and multimodal--dependen properties of MUG.
modalities, there are bunch of datasets from the medical domain (Lin al., 2020; Lee et al., 2019; Kautzky et al., 2020; Gupta et al., 2020). Other than the mentioned table, text, and image modali- ties, multimodal learning has also been conducted in time-series, speech, and video modalities (Zhang et al., 2022, 2020; Li et al., 2020).
MUG can enable future studies of many mul- timodal tasks, and we focus on the multimodal classification task this paper. For the primary classification evaluation, we incorporate two state- of-the-art (SOTA) unimodal classifiers for each of the three input modalities, resulting in total of six, along with two SOTA multimodal classifiers. We also propose novel baseline model MUGNET based the graph attention network (Veličković al., 2018). In addition capturing the in- teractions among the three input modalities, our MUGNET takes the sample-wise similarity into account, yielding a compatible performance to ex- isting multimodal classifiers. We further conduct efficiency evaluations to reflect the practical re- quirements of many machine learning systems.
2.2 Multimodal Classifiers for Tabular, Textual, and Visual Fields
Fusion is the core technology for multimodal classi- fication problems, which integrates data from each input modality and utilizes fused representations for downstream tasks (classification, regression, retrieval, etc.). Based on the stage of fusion (Iv et al., 2021), existing methods can be divided into early, late, or hybrid fusion. Early fusion mod- els (Sun et al., 2019; Shi et al., 2021; Zhu et al., 2021) usually fuse raw data extracted features before they are fed into the learnable classifier, while late fusion models (Erickson et al., 2020; Soenksen et al., 2022; Lu et al., 2022) employ sep- arate learnable encoders for all input modalities and then fuse these learned representations into the learnable classifier. Hybrid fusion models are more flexible, allowing for modality fusion to oc- different stages simultaneously (Qingyun et al., 2021; Li et al., 2021). Although existing works have demonstrated remarkable capability in modeling feature interactions, they ignore sig- nals of sample proximity, such as the tendency for within group to exhibit similar behavior or share common interests. In response, we propose our ap- proach, MUGNET, which dynamically constructs graphs based sample similarity and effectively combines graphical representation learning with multimodal fusion. Our approach draws inspira- tion from pioneering graph neural networks (Guo et al., 2021; Wang et al., 2021; Georgantas and Richiardi, 2022), which have achieved success in various classification tasks.
2 Related Works
2.1 Multimodal Classification Datasets with Tabular, Textual, and Visual Fields
Machine learning models in real-world applications need to deal with multimodal data that contains both tabular, textual, and visual fields. Due to pri- vacy or license issues, there exist very few datasets that cover these three modalities. To the best of our knowledge, PetFinder' is one of the few publicly available datasets. HAIM-MIIIC-MM (Soenksen et al., 2022) is multimodal healthcare dataset containing tabular, textual, image, and time-series fields. However, only credentialed users can access HAIM-MIMIC-MM.. On the other hand, there exist many datasets that cover two modalities (out of table, text, and image modalities). The most com- mon datasets are the ones with both textual and visual features (MM-IMDB (Ovalle et al., 2017). V-SNLI (Vu et al., 2018), MultiOFF (Suryawan- shi al., 2020), WIT (Srinivasan et al., 2021), MELINDA (Wu et al., 2021), etc.). Meanwhile, (Shi et al., 2021: Xu al., 2019; Qu al., 2022) provide benchmark datasets for table and text modalities, For the combination of table and image
3 MUG: the benchmark
We create and release MUG with eight datasets for multimodal classification with tabular, text, and image fields the community for future studies. Raw data and examples of how to appropriately load the data are provided in https: //g thub. www.lllwjiyggg/iieoo Bench. MuG is under the
'PetFinder: https: : //www/ kaggle, e..gleecooo waggle.com//ee......oooooo.ooo.o tions/ petfinder- adoption iedoptioo-predicteee wwtpeiiiiio/oeeeeeeeeeeeeepppoee

---

"CC BY-NC-SA 4.0" license?, and is designated to use for research purposes.
3.2 Creation Process
To create MUG, we first identify the categorical columns that can serve as the prediction targets. The reasons for choosing these targets are elabo- rated in Appendix B.1. We obtain a total of eight datasets from the four games, including pkm and pkm from Pokémon; hs ac, _s_as, as, hs mr, and hs issss; lol so from LoL.; asg_sq from CS:GO. Then, we conduct necessary data cleaning and verification to ensure the quality of MUG. To allevi- ate the class imbalance issue in some datasets (e.g-, one class may contain less than 10 samples), we re-group sparse classes into one new class that con- tains enough samples for training and evaluation. For missing values target categorical columns, we manually assign special None_Tyy as one new class of the dataset. For missing values of input columns, we keep them blank to allow classi- fication models decide the best imputation strate- gies. Moreover, we also anonymize columns that cause data leakage (e.g., the id column in hs_as transformed to anonymous id column).
3.1 Data Sources
To collect multiple and large-scale datasets that support multimodal automated machine learning, we collected data from four games: Pokémon, Hearthstone, League of Legends, and Counter- Strike: Global Offensive. We deliberately chose these video games as they have distinet video game genres (e.g,, role-playing, card, multiplayer online battle arena, and shooting). All of these datasets were gathered from publicly accessible web con- tent by October 2022, and there are no licensing issues associated with them. They do not contain any user-specific private information. In particular,
Pokémon is a video game centered around fic- tional creatures called "Pocket Monsters" that trainers capture and train to battle each other. Pokémon is owned by Nintendo Co., Ltd., Crea- tures Inc., and Game Freak Inc. Pokémon data is collected from https: https://bubbapedi.. bulbagarden. net/kiki under the "CC BY-NC- SA 2.5" license.
After the abovementioned preprocessing, we split the dataset into training, validation, and test- ing sets with an 80/5/15 ratio. Each dataset com- promises between 1K and 10K samples, associ- ated with tabular, textual, and visual features. An overview of these datasets is shown Table 1. This broad range sample sizes and diverse data types ensures the representation of a wide varicty of instances, allowing for robust model training and evaluation across different data modalities.
HcarthStone is an online collectible card game developed by Blizzard Entertainment, Inc., featur- ing strategic gameplay where players build decks and compete against each other using variety of spells, minions, and abilities. Hearthstone data is collected from https: //her thserrtsttoonjeje com/ under the "CCO" license.
3.3 Benchmark Analysis
League of Legends (LoL) is a multiplayer online battle arena (MOBA) video game developed by Riot Games, Inc. where teams of players compete in fast-paced matches, utilizing unique champi- ons with distinct abilities achieve victory. LoL. data is collected from https://lolskinshhp. Counter-Strike: Global Offensive (CS:GO) is multiplayer first-person shooter video game developed by Valve Corporation and Hidden Path Entertainment, Inc., where players join teams to compete in objective-based matches involving tactical gameplay and precise shoot- ing. CS:GO data is collected from https: //wo:://www.. csgodatabase, com/.
The MUG benchmark is curated to meet the fol- lowing list of criteria:
(i) Publicly available data and baseline models can facilitate reproducible experiments and accelerate the development of advanced models.
(ii) Diversity should be preserved in the bench- mark. We do not want the benchmark to have bias toward certain data class distribution. The benchmark with high variety of datasets aids the research community in examining the robustness of models.
(iii) Multimodall pected for each dataset. Datasets that are too easy to be classified by single modality are not suit- able, since they would hide the gap between the multimodal perceptron abilities models.
classification is ex-
*CC BY-NC-SA 4.0: https://crraativeeommns. sicenses/by-nc-sa/A.
We conduct a rich set of analyses to verify

---

Dataset
Game
Pred. Target
#Row
#Class
#Feat (tab/Ixt/img)
pkm t1
Pokémon
Primary Type
719/45/133
18
23 (17/5/1)
pkm_t
Pokémon
Secondary Type
719/45/133
19
23 (17/5/1)
hs_ac hs_as
HearthStone
All card's Category All card's Set
8569/536/1609
14
18 (12/5/1)
HearthStone
8566/533/160
38
18 (12/5/1)
h8 mr
HcarthStone
Minion card's Race
5421/338/1017
16
13 (7/5/1)
hs_st
HearthStone
Spell card's School
2175/170/508
8
11 (5/5/1)
[ol_sc
LoL
Skin Category
1000/64/188
11 (3/7/1)
csg_sq
CS:GO
Skin Quality
766/49/141
6
7 (5/1/1)
Table 1: The statistics of the eight datasets in MUG.
Valid
Missing
Shunnon Equiubiitty
0.94
0.93
100
Steshed DeviaaDDDvillt Deviation
10
pkm a
Percenagge
90
lol x sag
0.8
0:96
0.73
10a
0.68
0.64
80
0.6
0.58
10
70
pkm phm.
12
de
In Bi
ha
1ol_90
est sq
pkm
bd
10a
hn
10'
10
10a
Mean
(a) Class balance ratios.
(b) Percentages of missing features.
(c) Means and SD of numerical features.
pkm
hs
32
pkm
lol csg
#Categories
40
230
200
og
20
200
pkm, fl pkm 2
hts.a
lt. ni
Is.
lol sc
cig. sq.s
16
64
R
150
Word count
(d) Counts of categorical features.
(e) Distributions of word counts.
(f) Distributions mean RGB pixels.
Figure 2: Multi-aspect of data analysis for MuG (duplicated datasets are merged into one group).
that MUG indeed satisfied the diversity require- ment. Figure shows the properties of datasets in multi-aspect. For the classification task proper- (iiguee 2a), we adopt the Shannon equitability index (Shannon, 1948) (definition in Appendix B.2) to measure the class balance ratio. The index ranges from 0 to 1, and the larger the Shannon equitability index, the more balanced the dataset is. For the feature properties, we include percentages of missing features (Figure 2b), means and stan- dard deviations of numerical features (Figure 2c),
and distributions of image mean RGB pixel values (Figure 2f). In these figures, we merged dupli- cated results from some datasets into one group to make the presentation clean and neat (i.e., fim. 11, pkm 12 are grouped into pkm; hs hs_ac, hs_as, _ss hs_mi js_mr, hs are grouped into hs). As shown in the fig- ures, the eight datasets reflect real-world problems that diverse and challenging. We further study the correlation between category labels and input modalities in MUG. Referring to the t-SNE pro- jection of multimodal embeddings Figure 7, it is evident that MUG exhibits a strong multimodal dependency. In this case, the use of unimodal in-
category counts of categorical features (Figure 2d).
distributions of word counts per sample (Figure 2c).

---

formation alone is inadequate to differentiate be- tween samples belonging to different classes. For a more comprehensive analysis, we encourage in- terested readers to refer to the details provided in Appendix B.3
SWIN (Liu et al., 2021) is another vision trans- former that benefits from hicrarchical architec- ture and the shifted windowing scheme. The pro- posed techniques address several key challenges when adapting transformers in image modality, such as large variations in the scale of visual entities and the high resolution of pixels.
4 Baseline Models
We employ several state-of-the-art unimodal classi- fiers and multimodal classifiers in the experiments. We also proposed our own graph neural network- based multimodal classifier one baseline model to be compared.
In practice, we adopt the following multimodal classifiers in the experiments:
AutoGluon (Erickson et al., 2022) is an ensemble-learning model for multimodal clas- sification and regression tasks. The concept of AutoGluon is stack ensembling, where the final prediction is obtained by combining intermediate predictions from multiple base models. To han- dle multimodal classification, SOTA unimodal classifiers (e.g.. tree models, MLPs, CNNs, trans- formers) are adopted as base models.
4.1 Existing State-Of-The-Are Classifiers
this paper, we adopt the following SOTA uni- modal classifiers the experiments:
Tabular modality classifiers:
GBM (Ke et al., 2017) is light gradient boosting framework based on decision trees. Due to its ability to capture nonlinear relationships. handle complex tabular data, provide feature importance insights, and robustness to outliers and missing values, GBM has achieved state-of of-the-an results in various tabular data tasks,
AutoMM (Shi et al., 2021) where separate neural operations are conducted on each data type and extracted high-level repre- sentations are aggregated near the output layer. Specifically, MLPs are used for tabular modal- ity, and transformers are used for text and image modalities. After that, dense vector embeddings from the last layer of each network are pooled into one vector, and the final prediction ob- tained via an additional two-layer MLP.
1) is a late-fusion model
tabMLP (Erickson et al., 2020) is a multilayer perceptron (MLP) model that is specifically de- signed to work with tabular data. tabMLP con- tains multiple separate embedding layers to han- dle categorical and numerical input features.
4.2 MUGNET
Textual modality classifiers:
RoBERTa (Liu al., 2019) is robustly optimized transformer-basee masked language model (masked LM). RoBERTa builds upon the success of BERT by refining and optimizing its training methodology, and achieves superior per- formance on a wide range of NLP tasks.
MUGNET is our own multimodal classifier which further proposed as a competitor existing mod- cls. We propose three key components to make MUGNET a powerful graph neural network for the multimodal classification task. They are adap- tive multiplex graph construction module, GAT en- coder module, and attention-based fusion module, as shown in Figure 3. Firstly, adaptive multiplex graphs are constructed reflect sample-wis simi- larity within each modality. Then, separate GAT en- coders (Veličković et al., 2018) are employed to ob- tain dense embeddings of samples, by propagating information between neighbors. Finally, tabular, text and image embeddings are combined by inter- modality attention to obtaining the fused embed- ding for multimodal classification. GNNs (Yang et al., 2020: Guo al., 2021) show great capability to leverage the graph structure, propagate informa- tion, integrate features, and capture higher-order relationships. This leads to accurate and robust classification performance across various domains.
Electra (Clark et al., 2020) is another variant of the transformer-based model, which differs from traditional masked LMs like BERT or RoBERTa. While masked LMs randomly mask tokens and predict these masked tokens, Electra is trained as a discriminator to identify whether each token is replaced by generator.
Visual modality classifiers:
VIT (Dosovitskiy et al., 2020) extends the trans- former model image data, by dividing the input image into grid of patches and processing each patch as a token. Empirical results show that VIT outperforms previous SOTA convolutional neural networks in image classification tasks.

---

Adaptive Multiplex
Tabular
Graph Construction
Tab. embs
samples
GAT tab
Inter-miodality Attention
Txt. embs
Fused embs.
Text
samples
Sawes
GAT txt
Img. embs
Image
samples
GAT mmg
MuGNET
Figure 3: Model architecture of MUGNET.
In this work, we propose to regard the whole sam- ples as a correlation network (Wang et al., 2021; Georgantas and Richiardi, 2022) that represents sample-to-sample similarities, while existing mul- timodal classifiers rarely consider this before.
held-out validation set, so that the multiplex graph construction is adaptive to any downstream task.
GAT encoder module. We use the power- ful multi-head graph attention neural network (GAT) (Veličković et al., 2018) the encoder obtain structure-aware representations of samples. Separate GATs are employed for each view of the multiple graph, so that H GAT( F 0), where 7." E R xxx and I represents the learn- able parameters of the GAT encoder. We want to state there is no information leakage in MUGNET, because we follow the inductive learning setting of GNNs (Hamilton et al., 2017) where the GAT encoder is trained the multiplex graph g de- rived from labeled training samples Xt, and new unseen multiplex graph is derived from all sam- ples Xy U Xy at the inference stage. Further- more, we adopt a graph sampling technique (Graph- SAINT (Zeng et al., 2019)) during the GAT train- ing process, to improve the efficiency and general- ization. The graph sampling technique essentially samples subgraph by random walks for each train- ing step. thus the "neighbor explosion" issue is al- leviated with a constrained number of neighbors per node and the variance of GAT reduced with fewer outliers or noise in the sampled graph.
Adaptive multiplex graph construction module. Following the notation defined $5.1, the adaptive multiplex graph construction module first utilizes pre-processing pipelines (e.g., monotonically in- creasing integer mapping for categorical inputs, no alteration for numerical inputs) or pre- pre-trained fea- ture extractors (e.g., CLIP (Radford et al., 2021) for text and image inputs) to obtain dense mul- timodal features F f(X2) Nx +d"+ where F ces for tabular, text, and image modalities. The adaptive multiplex graph construction module then derives multiplex sample-wise similarity graph G- (G G" F")
" denotes feature matri-
(A*,"", (A', F)), where each modality-specific (A',F))
adjacency matrix Am NxN, Vm E i, is calculated based the multimodal features
A SJ! ).
(1)
It is worth noting that the sample-wise similarity function sim is adaptive, and is chosen from co- sine similarity, radial basis function (RBF) ker- nel, or k-nearest neighbor. For these modality- specific graphs, use separate hyperparameters (e.g., threshold for score-based functions, the value of k for k-nearest neighbor) control their sparsity properties. The similarity function and its associated hyperparameters are determined through hyperparameter tuning (Liaw et al., 2018) on the
A Atention-basedd fusion module. After we obtain the structure-aware embeddings of samples from the tabular, text, and image modalities H² He Hi the attention-based fusion module is responsible for fusing them into one single embedding via the attention- fusion module. The attention weight a." E R for j-th sample of modality m is

---

computed as:
5.3 Performance Comparisons
exp(e)))
(2) Table 2a and 2b show the performance of all eval- uated models on MUG. As can be seen, multi- modal classifiers (except AutoMM) consistently outperform unimodal classifiers in both log-loss and accuracy. It demonstrates that the classifi- cation tasks in MUG are multinodal-dependennt where each modality only conveys partial informa- tion about the required outputs. Among the three multimodal classifiers we used, AutoGluon and MUGNET are the top-2 models with well-matched performances. In Table 2a and 2b, AutoGluon achieves the best performance eight times, while MUGNET also achieves the best performance eight times. More specifically, AutoGluon is superior in log-loss whereas MUGNET has better accuracy scores. AutoMM performs the worst among multi- modal classifiers, and it sometimes underperforms unimodal classifiers. Considering that AutoMM trains powerful deep neural networks on small scale of datasets and have observed the gap between the training loss and validation loss, it is highly possible that AutoMM overfitting. While AutoGluon and MUGNET also adopt deep neu- ral networks as base models, they are more robust since AutoGluon proposes a repeated bagging strat- egy and MUGNET utilizes graph sampling tech- niques to avoid overfitting. Among unimodal clas- sifiers, tabular models seem to outperform textual and visual models in most cases (six out of eight datasets). There is slight performance gain com- paring textual models to visual models because textual models are better on five datasets.
aj m' [(tsa) exp(e)"))
oj =wn tanh Wa h2"",,
where ex E R denotes the unnormalized attention weight, Way _ x1 Wa, C IR x denote learnable parameters, and h m E R denotes the j-th row of 71"h (i.e., embedding of j-th sample of modality m). The fused embedding of j-th sample is then calculated by:
h ===aa atht agh, aa
(4)
The fused embedding hj incorporates cross- modalities interactions and provides a complete context for the downstream tasks. An addi- tional two-layer MLP is trained to predict the category of j-th sample Új = softmax Weksz Leaky h2)). We adopt cross-entropy between prediction ğ and target as MUGNET's loss function.
5 Experiments
5.1 Problem Definition
Given finite set of categories y' and labeled train- ing pairs (Ii; Di) E X1 x V, multimodal classifica- tion aims at finding a classifier
Xr -> y such that új f(xz) is a good approxmiation of the unknown label yj for unseen sample zj E Xy. is worth noting that the each multimodal sample x E Xz U Ny consists of tabular fields t, textual fields 8, and image fields (i.e., x = (1,8, (1)))
To better understand the overall performance of models across multiple datasets, we propose using critical difference (CD) diagrams (Demšar, 2006). In CD diagram, the average rank of each model and which ranks are statistically significantly dif- ferent from each other are shown. Figure 4a and 4b show the CD diagrams using the Friedman test with Nemenyi post-hoc test at p < 0.05. In sum- mary, we observe that AutoGluon and MUGNET respectively achieve the best rank among all tested models with respect to log-loss and accuracy, al- though never by statistically significant margin. Moreover, tabular models obtain higher ranks than other unimodal classifiers. The similar observa- tions from Table 2 and Figure support that effec- tively aggregating information across modalities is critical for the multimodal classification task.
5.2 Experimental Setup
We use the official training, validation, and testing splits provided by MuG to conduct experiments. We choose the log-loss and accuracy to evaluate model performance, since these metrics are rea- sonable and commonly used in previous studies. For comparable and reproducible results, all mod- els are trained and tested using the same hardware. Specifically, the machine is equipped with 16 Intel Xcon Gold 6254 CPUs (18 cores per CPU) and one 24GB TITAN RTX GPU. We add an 8-hour time limitation for the training process to reflect real- world resource constraints. The implementation of evaluated models are put in Appendix C.
and hyperparameter details

---

Method
pkm tl
pkm_12 12
hs ac
hs as
hs mr
hs_ss
lol_sc
sq
Unimodal Classifiers
GBM
1.838
2.038
0.911
2.352
0.913
0.603
0.198
1.107
tabMLP
1.442
1.909
1.172
2.155
1.247
0.672
0.533
0.718
RoBERTa
1.834
2.191
1.999
2.393
1.920
1.254
0.847
0.734
Electra
2.907
2.179
2.118
3.155
2.085
1.263
0.611
0.757
VIT
3.680
2.543
1.527
2.786
1.032
2.056
2.049
0.835
Swin
2.657
2.229
2.018
2.795
2.089
1.397
1.470
0.750
Multimodal
Classifiers
AutoGluon
0.973
1.507
0.654
1.793
0.403
0.350
0.159
0.631
AutoMM
1.736
2.029
1.987
2.193
1.836
1.320
0.792
0.674
MUGNET
1.000
1.499
0.922
1.499
0.321
0.442
0.248
0.654
(a) Results in 'log-loss' (the less the better).
Method
pkm tl
pkm_12 12
hs_ae
hs_as
hs ___
hs_ss
lol_sc
csg_sq
Unimodal Classifiers
GBM
0.489
0.489
0.726
0.421
0.737
0.795
0.963
0.610
tabMLP
0.662
0.481
0.627
0.377
0.617
0.776
0.851
0.681
Roberta
0.662
0.466
0.475
0.366
0.535
0.683
0.883
0.688
Electra
0.120
0.466
0.475
0.168
0.535
0.683
0.878
0.702
ViT
0.308
0.406
0.568
0.236
0.787
0.593
0.436
0.674
Swin
0.346
0.451
0.470
0.248
0.536
0.657
0,431
0.702
Multimodal
Classifiers
AutoGluon 0.744
0.617
0.787
0.495
0.879
0.882
0.963
0.766
AutoMM
0.639
0.511
0.475
0.415
0.549
0.671
0,888
0.738
MUGNET
0.774
0.669
0.724
0.572
0.908
0.880
0.968
0.745
(b) Result in 'accuracy' (the more the better).
Table 2: Overall experimental results with explicit modality performance. bold text represents the best performance and the underlined text represents the runner-up performance.
CD=4,247
CD=4.247
6
2
8
4
2
Swin
Swin
AutoGluon
MUGNET
VIT
VIT
MUGNET
AutoGluon
Electra
Electra
tabMLP
Roberta
Roberta
GBM
AutoMM
AutoMM
tabMLP
(a) Log-loss.
(b) Accuracy.
Figure 4: The critical difference diagrams show the ranks of each model for the test data of the eight datasets. The lower rank (further to the right) represents the better performance of model. Groups of models that are not significantly different (p 0.05) are connected by thick lines.
5.4 Efficiency Evaluations
machine learning model, efficiency is also a prac- tical requirement in many applications. Trade-off often exists between how accurate the model is
Although accuracy (or other metrics such as log- loss our case) the central measurement

---

be improved without degrading least one of the other objectives. Following this concept, we ob- serve that tabMLP, GBM, and MUGNET are the models with the best trade-offs between accuracy and efficiency, as these models reside in the Pareto frontier in Figure 6. Mcanwhile, other models are suboptimal with regard to this trade-off, since we can always find solution that has higher accu- racy and better efficiency simultaneously than these models.
Training Duration (scconds)
10a
10a
10a
10
GBM
RoBEikTa
Bleara
ViT
swin
AuonC
AutoMM
MuGNET
Figure 5: Training duration on datasets.
MUGNET
AutoCluon
0.8
Test Accuracy
and how long it takes to train and infer the model. Therefore, we record the training durations and test durations of models examine their efficiency. In Figure 5, we show the aggregated training dura-
0.6
GBM
AutoMM
tabMLP
RoBERTA
0.1
Electra
tion of evaluated models via a box plot. As can be seen, tabular models require an order of magni- tude less training duration than the other models, while AutoGluon stands out as requiring signif- icantly longer training duration. Among tabular models, tabMLP is 4x faster than GBM in terms of the median training duration. Except for tabular models and AutoGluon, other models are approxi- mately lightweight to train. worth noting that AutoGluon hits the 8-hour training duration con- straint on every dataset, thus the variance of its training durations across datasets is very small.
W/T
0.2
Swin
0
1.
1100
10
10-2
10
10
10a
10a
10
Test Duration (seconds)
Figure 6: Mean testing duration and mean normalized accuracy tradeoff's on all datasets.
Conclusion
In Figure 6, show the trade-offs between mean inference time and mean accuracy of models. Since the accuracy is not commensurable across datasets, we first normalize all accuracies through a dataset-wise min max normalization. After the normalization, the best model in each dataset is scaled to 1 while the worst model is scaled to 0. Finally, we take the average on the normalized ac- curacies and the test durations to draw the scatter plot. When both accuracy and efficiency are objec- tives models try improve, there does not exist model that achieves the best in both objectives si- multaneously. As an illustration, MUGNET has the highest test accuracy, but tabMLP has the fastest inference speed. Therefore, we adopt the Pareto- optimal concept to identify which models achieve "optimal" trade-offs. Pareto-optimal is widely used in the decision-making process for multi-objective optimization scenarios. By definition, solution is Pareto-optimal if any of the objectives cannot
This paper presents a benchmark dataset MUGNET along with multiple baselines as a starting point for the machine learning community to improve upon. MUGNET is multimodal classification bench- mark on game data that covers tabular, textual, and visual modalities. All eight datasets and nine evalu- ated baselines open-source and easily-extended to motivate rapid iteration and reproducible ex- periments for researchers. A comprehensive set of analyses included to provide insight into the characteristics of the benchmark. The experimen- tal results reported in the paper are obtained from models trained with constrained resources, which is often required by real-life applications. However, also welcome future works that utilize enor- mous resources. Finally, we hope this work can facilitate research on multimodal learning, and we encourage any extensions to MUGNET to support new tasks or applications such as open- open-domain re- trieval, AT-generated content, multimodal QA, etc.
"Parcto-optimal Definition: https://w. wiki wiki/6sLB

---

Limitations
Xiangjue Dong, Jiaying Lu, Jianling Wang, and James Caverlee. 2023. Closed- book question generation via contrastive learning. In EACL.
While our study emphasizes the importance effi- ciency real-world machine learning applications, we acknowledge certain limitations in our approach. Specifically, we deliberately focused on training and evaluating relatively "small" models within the context of the current era of large vision and lan- guage models (LVLMs) (Li et al.,, 2023; Liu et al., 2023; Lu et al., 2023). As result, the performance of LVLMS on our proposed MUG benchmark re- mains unexplored. Early exploration (Hegselmamn et al., 2023) about applying large language models on tabular classification shows that LLMs can be competitive with strong tree-based models. Based on the explorations and conducted experiments, we speculate LLVMs can not beat efficient ensemble GNN baselines using the same training time constraint. However, provide comprehensive understanding of multimodal classification, further research is expected. would be also intriguing to investigate the performance of LLVMs when pro- vided with unlimited training (ine-tuning) time.
Alexey Dosovitskiy. Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer. Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x 16 words: Transformers for image recognition scale. ICLR.
Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. 2020. Autogluon-tabular: Robust and curate automl for structured data. arXiv preprint arXiv:2003.06505.
Nick Erickson, Xingjian Shi, James Sharpnack, and Alexander Smola. 2022. Multimodal automl for im- age, text and tabular data. In KDD.
Costa Georgantas and Jonas Richiardi. 2022. Multi- view omics translation with multiplex graph neural networks. In WWW.
Xiawei Guo, Yuhan Quan, Huan Zhao, Quanming Yao, Yong Li, and Weiwei Tu. 2021. Tabgnn: Multiplex graph neural network tabular data prediction. dlp-kdd.
Acknowledgement
Yubraj Gupta, Ji-In Kim, Byeong Chae Kim, and Goo- Rak Kwon, 2020. Classification and graphical anal- ysis of alzheimer's discase and its prodromal stage using multimodal features from structural, diffusion, and functional neuroimaging data and the apoe geno- type. Frontiers in aging neuroscience.
This research is partly supported by the National Institute Of Diabetes And Digestive And Kidney Diseases of the National Institutes of Health under Award Number K25DK 135913 and the Division Mathematical Sciences of the National Science Foundation under Award Number 2208412. Any opinions, findings, and conclusions or recommen- dations expressed herein are those of the authors and do not necessarily represent the views, either expressed or implied, of the National Science Foun- dation, National Institutes of Health, the U.S. government.
Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. NewrIPS.
Huan He, Shifan Zhao, Yuanzhe Xi, and Joyce C Ho. 2023. Meddiff: Generating electronic health records using accelerated denoising diffusion model. arXiv preprint arXiv:2302.0435.
Stefan Hegselmann, Alejandro Buendia, Hunter Lang. Monica Agrawal, Xiaoyi Jiang, and David Sontag, 2023. Tablim: Few-shot classification of tabular data with large language models. In AISTATS. PMLR.
References
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. ICCV.
MD Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shi- ratuddin, and Hamid Laga. 2019. A comprehensive survey of deep learning for image captioning. ACM Computing Surveys
Kevin Clark, Minh- Taah-Thaggga Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pre- training text encoders as discriminators rather than generators. In ICLR.
Mahesh Huddar, Sanjeev S Sannakki, and Vijay S Rajpurohit. 2018. An ensemble approach to utterance level multimodal sentiment analysis. In CTEMS.
Janez Demšar. 2006. Statistical comparisons of classi- fiers over multiple data sets. The Journal of Machine learning research.
William C Sleeman Iv, Rishabh Kapoor, and Preetam Ghosh. 2021. Multimodal classification: Current landscape, taxonomy and future directions. CSUR.

---

A Kautzky, T Vanicek, C Philippe, GS Kranz, W Wadsak, M Mitterhauser, A Hartmann, A Hahn, M Hacker, D Rujescu, et al. 2020. Machine learning classification of adhd and he by multimodal seroton- ergic data. Translational psychiatry.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV.
Jiaying Lu, Jinmeng Rao, Kezhen Chen, Xiaoyuan Guo, Yawen Zhang. Baochen Sun, Carl Yang. and Jie Yang. 2023. Evaluation and mitigation of agnosia in multimodal large language models. arXiv preprint arXiv:239.04044
Guolin Ke, Qi Meng, Thomas Finley. Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Yan Liu. 2017. Lightghm: A highly efficient gradient boosting decision tree. NeurlPS.
Mackenzie Leake, Hijung Valentina Shin, Joy O Kim, and Maneesh Agrawala. 2020. Generating audio- visual slideshows from text articles using word con- creteness. In CHI.
Jiaying Lu, Xin Ye, Yi Ren, and Yezhou Yang. 2022. Good, better, best: Textual distractors generation for multiple-choiee visual question answering via reinforcement learning. In CVPR Workshop on O- DRUM.
Jeungchan Lee, Ishtiaq Mawla, Jieun Kim, Marco Loggia, Ana Ortiz, Changjin Jung. Suk-Tak Chan, Jessica Gerber, Vincent J Schmithorst, Robert R Ed- wards, et al. 2019. Machine learning-basse predic- tion of clinical pain using multimodal neuroimaging and autonomic metrics. Pain.
Liqiang Nie, Mengzhao Jia, Xuemeng Song, Ganglu Wu, Harry Cheng, and Jian Gu. 2021. Multimodal activation: Awakening dialog robots without wake words. In SIGIR.
Jiao Li, Xing Xu, Wei Yu, Fumin Shen, Zuo Cao, Kai Zuo, and Heng Tao Shen. 2021. Hybrid fusion with intra-and mross-modality attention for image-recipe retrieval. SIGIR.
John Edison Arevalo Ovalle, Thamar Solorio, Manuel Montes-y-Gómeez and Fabio A. González. 2017. Gated multimodal units for information fusion. In ICLR.
Jumnan Li. Dongxu Li. Silvio Savarese, and Steven Hoi. 2023. BLIP-2: bootstrapping language-imag pre- training with frozen image encoders and large lan- guage models. In ICML.
Ankur Parikh, Xuezhi Wang, Sebastian Gchrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A controlled table-to- text generation dataset. In EMNLP.
Mingzhe Li, Xiuying Chen, Shen Gao. Zhangming Chan, Dongyan Zhao, and Rui Yan. 2020. Vmsmo: Learning to generate multimodal summary for video- based news articles. In EMNLP.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library.
Paul Pu Liang. Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng. Jason Wu, Leslie Yufan Chen, Peter Wu,
Michelle A Lee, Yuke Zhu, al. 2021. Multibench: Multiscale benchmarks for multimodal representa- tion learning. In NeurlPS.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg. Vanderplas, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay. 2011. Seikit-lear: Machine learning in Python. JMLR, 12:2825-282 2830.
Passos,
Richard Liaw, Eric Liang. Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. 2018. Tune: A research platform for distributed model selection and training. arXiv preprint
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma. and Furu Wei. 2023. Kosmos-2: Grounding multimodal large language models the world. arXiv preprint 1.X2:2300..1824.,
Weiming Lin, Qinquan Gao, Jiangnan Yuan, Zhiying Chen, Chenwei Feng, Weisheng Chen, Min Du, and Tong Tong. 2020. Predicting alzheimer's disease con- version from mild cognitive impairment using an ex- treme learning machine-bass based grading method with multimodal data. Frontiers in aging neuroscience.
Ben Poole, Ajay Jain, Jonathan Barron, and Ben Mildenhall. 2022. Dreamfusion: Text-to-3d 3d using 2d diffusion. arXiv preprint a2Xiv:2209. 14988.
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lec. 2023, Visual instruction tuning, arXiv preprint arXiv:2304.00
Fang Qingyun, Han Dapeng, and Wang Zhaokui. 2021. Cross-modality fusion transformer for multispectral object detection. arXiv preprint arXiv:2111.0022)
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint urXiv:1907, 11692.
Yubin Qu, Fang Li, Long Li, Xianzhen Dou, and Hong- mei Wang. 2022. Can we predict student perfor- mance based on tabular and textual data? IEEE Access.

---

Alee Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In ICML.
Jialin Tian, Kai Wang, Xing Xu. Zuo Cao, Fumin Shen, and Heng Tao Shen. 2022. Multimodal disentangle- ment variational autoencoders for zero-shot cross- modal retrieval. In SIGIR.
Laurens Van der Maaten and Geoffrey Hinton. 2008.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Vass. Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot text-to-image gener- ation. In ICML.
Visualizing data using t-sne. JMLRR
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. 2018. Graph attention networks. In ICLR.
Shailaja Keyur Sampat, Maitreya Patel, Subhasish Das. Yezhou Yang, and Chitta Baral. 2022. Reasoning about actions over visual and linguistic modalities: survey, arXiv preprint 2ttv:2207.075666
Hoa Trong Vu, Claudio Greco, Aliia Erofeeva, Somayeh Jafaritazehjan, Guido Linders, Marc Tanti, Alberto Testoni, Raffaella Bemardi,, and Albert Gatt. 2018.
Grounded textual entailment. In COLING.
Claude Elwood Shannon. 1948. mathematical the- ory of communication. The Bell system technical journal.
Minjie Wang. 2019. Deep graph library: Towards efficient and scalable deep learning on graphs. In ICLR workshop representation learning on graphs and manifolds.
Xingjian Shi, Jonas Mueller, Nick Frickson, Mu Li, and Alex Smola. 2021. Multimodal automl on structured tables with text fields. 8th ICML Workshop on AutoML.
Tongxin Wang, Wei Shao, 7hi Huang, Haixu Tang, Jie Zhang, Zhengming Ding, and Kun Huang. 2021. Mogonet integrates multi-omics data using graph con- volutional networks allowing patient classification and biomarker identification. Nature Communicaa- tions.
Luis R Soenksen, Yu Ma. Cynthia Zeng, Leonard Bous- sioux, Kimberly Villalohos Carballo, Liangyuan Na, Holly M Wiberg, Michael L Li, Ignacio Fuentes, and Dimitris Bertsimas. 2022. Integrated multimodal artificial intelligence framework for healthcare appli- cations. NPJ digital medicine.
Te-Lin Wu, Shikhar Singh, Sayan Paul, Gully Burns, and Nanyun Peng. 2021. Melinda: A multimodal dataset for biomedical experiment method classifica- tion. In AAAI.
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. 2021. Wit: Wikipedia-bass based image text dataset for multimodal multilingual machine learning. In SIGIR.
Xueqing Wu, Jiacheng Zhang, and Hang Li. 2022. Text- to-table: A new way of information extraction. In ACL.
Cheng Xu, Xiaofeng Hou, Jiacheng Liu, Chao Li, Tian- hao Huang, Xiaozhi Zhu, Mo Niu, Lingyu Sun, Peng Tang, Tongqiao Xu, et al. 2022. Mmbench: Bench- marking end-to-end multi-modal dans and under- standing their Fardware-softwar implications. arXiv e-prints ar 221
Baohua Sun, Lin Yang. Wenhan Zhang, Michael Lin, Patrick Dong, Charles Young, and Jason Dong, 2019. Supertml: Two-dimensional word embedding for the precognition structured tabular data. In CVPR Workshops.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. 2023. Generative pretraining multimodality.
Keyang Xu, Mike Lam, Jingzhi Pang, Xin Gao, Char- lotte Band, Piyush Mathur, Frank Papay, Ashish K Khanna, Jacek B Cywinski. Kamal Maheshwari, et al. 2019. Multimodal machine learning for automated ied coding. In Machine learning for healthcare con- ference.
Shardul Suryawanshi, Bharathi Raja Chakravarthi, Mi- hael Arean, and Paul Buitelaar. 2020. Multimodal meme dataset (multioff) for identifying offensive con- tent in image and text. In Proceedings of the second workshop trolling, aggression and cyberbullying.
Carl Yang. Jieyu Zhang, Haonan Wang, Sha Li, Myung- Kim, Matt Walker, Yiou Xiao, and Jiawei Han. 2020. Relation learning on social networks with multi-modal graph edge variational autoencoders. In WSDM.
Golsa Tabmasebzadeh, Endri Kacupaj, Eric Müller- Budack, Sherzod Hakimov, Jens Lehmann, and Ralph Ewerth. 2021. Geowine: Geolocation based wiki, image, news and event retrieval. In SIGIR.
Amir 7adeh, Minghai Chen, Soujanya Poria, Erik Cam- bria, and Louis- Phisl-pil Morency. 2017. Tensor fu- sion network for multimodal sentiment analysis. In EMNLP.
Alon Talmor, Oni Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Han- nanch Hajishirzi, and Jonathan Berant, 2020. Mul- timodalqa: complex question answering over text, tables and images. In ICLR.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Ra- jgopal Kannan, and Viktor Prasanna. 2019. Graph- saint: Graph sampling based inductive learning method. In ICLR.

---

Zhixiong Zeng, Shuai Wang, Nan Xu, and Wenji Mao. 2021. Pan: Prototype-basee adaptive network for robust cross-modal retrieval. In SIGIR.
generation (Parikh al., 2020), text-to-table gen- cration (Wu al., 2022), etc. Following this idea, the aligned triples can be utilized to pre-train multi- modal encoders. Furthermore, multimodal datasets that cover more than two modalities can inspire more novel generation applications, such as audio- visual slideshows generation from text (Leake al.. 2020) textual-visual summarization from video- based news articles (Li et al., 2020).
Dong Zhang, Xincheng Ju, Junhui Li, Shoushan Li, Qiaoming Zhu, and Guodong Zhou, 2020. Multi- modal multi-label emotion detection with modality and label dependence. In EMNLP.
Yi Zhang, Mingyuan Chen, Jundong Shen, and Chongjun Wang. 2022. Tailor versatile multi-modal learning for multi-label emotion recognition. In AAAI.
B Details of MUG
Yitan Zhu, Thomas Brettin, Fangfang Xia, Alexander Partin, Maulik Shukla, Hyunseung Yoo, Yvonne Evrard. James H Doroshow, and Rick L Stevens. 2021. Converting tabular data into images for deep learning with convolutional neural networks. Scien- rific reports.
B.1 Prediction Targets
Dataset
Source
Prediction Target
pkm_t1 pkm_tt hs_ac _s_ac
Pokémon
Pokémon's primary Type
Pokémon
Pokémon's 's secondary Type
Appendix
HearthStone
All card's Category
hs as
HearthStone
All card's s Set
A Broad Impact of Multimodal Datasets for Tasks beyond Classification
hs mr
HcarthStone
Minion card's Race
hs_ss lol_se se Lol.
HcarthStone
Spell card's ' School
Skin Category
The proposed multimodal classification benchmark, MUG, a valuable resource to inspire future stud- ies for tasks including but not limited classifica- tion. First, it is natural to utilize MUG to examine the model's abilities to understand the complex world surrounding us (Liang et al., 2021; Xu et al., 2022). The multimodal perception ability is crucial for open-domain retrieval (Tahmasebzadeh al., 2021: Tian et al., 2022; Zeng et al., 2021), multi- modal classification (Zadeh et al., 2017; Huddar al., 2018), multimodal question answering (An- tol al., 2015; Talmor al., 2020; Lu et al., 2022), interactive robots (Nic et al., 2021; Sampat et al., 2022), precision medicine (Soenksen et al., 2022: Lin et al., 2020; Gupta et al., 2020), etc. These applications all involve multi-modal input where each modality conveys partial information and complete understanding can be achieved by tak- ing all modalities into account. Second, MUG that contains aligned tabular, textual, and visual data is also beneficial to multimodal generation applications (Dong et al., 2023; He et al., 2023; Sun et al., 2023; Peng et al., 2023). For instance, many text-to- eoxttt--image generation models (Ramesh et al., 2021; Poole et al., 2022) employ CLIP (Rad- ford et al., 2021) their feature encoder, and CLIP is an alignment-based fusion model trained on semantically equivalent text and image pairs. There also exist many studies exploring unimodal- to to-unimodal generation tasks, such as image-to- text captioning (Hossain et al., 2019), table-to-text
sqg_s
CS:GO
Skin Quality
Table 3: The prediction targets of datasets in MUG.
We identify appropriate categorical columns that can serve as the prediction targets from these four games, with handy reference presented in Table 3. More specifically, provide detailed elaborations about the prediction targets and their corresponding input multimodal features. Some MUG datasets may share same input multimodal features.
pkm t1 and pkm_ 12: Pokémon can be catego- rized into various elemental types, such as Fire, Ice, Normal (non-elemental), and more. Each Pokémon can have up to one primary type (for pkm_tl) and one secondary type (for pkm_12).
17 tabular features: generation. status, height_m, weight abilities_me _umm, lotal_points, hp, attack, defense, sp_attack, sp_ sp_defense, speed, natch_rate base_friene base_friendship,
base base_experience, growth_rate percentage_male male.
text features: name, species, ability ability_2,
image feature: image.
hs and hs_as: Each HearthStone card belongs one certain category (for hs hs_ac) such as min- ion, spell, weapon, etc. Moreover, each card is part of a set (for hs js_as) where new card sets are

---

released periodically introduce new content and strategies to the game.
B.2 Definition of Shamnon Equitability Index.
The Shannon equitability index (Shannon, 1948). also known as the Shannon evenness index or Shan- non's 's diversity index, is measure used in ccology to assess the evenness or equitability of species abundance in a given community or ecosystem. is derived from the Shannon entropy, which quantifies the diversity or richness of species in community. For the classification task properties (Figure 2a), we adopt it to measure the class balance ratio,
- 12 tabular features:
health, attack, cost, type, rarity, collectible, spellSchool, race, durability, overload, spellDamage, set (for hss hssac)/ cardClass (for hs hs_as).
text features: name, id, artist, text, me- chanies.
-maage feature: image.
hs msrr Each HeathStone minion card essen- tially a creature, thus can be divided into differ- ent races (for hs_mr).
H log(k)
# Σ lo
Eн
(5)
log(k)
7 tabular features: health, attack, cost, rarity, collectible, candClass, set.
where denotes the entropy of each class's counts, denotes the dataset containing classes. EØ ranges from 0 to 1, and the large Σn, the more balanced the dataset is.
5 text features: name, id, artist, text, me- chanics.
1 -image feature: image.
B.3 Analysis on Multinodal-dependent.
hs_ss: Each HeathStone spell card, similarly to the minion race, each spell card belongs to a spe- cific school (for hs_ss) such as Shadow, Nature, etc.
To study the correlation between category labels and input modalities in MUG, plot 2D SNE (Van der Maaten and Hinton, 2008) projec- tions of various embeddings for the hs_mrr mr dataset. In the first row of Figure 7, the four subfigures present projections obtained from the raw features of tabular, textual, visual, and fused modalities, separately. Essentially, we conduct unsupervised dimension reduction (e.g.. SVD) on the raw fea- tures and then use t-SNE to obtain 2D projections. For tabular features, numerical columns are kept they are, and categorical columns are transformed into numbers between and n p_class 1. For tex- tual features, we first transformed them into token count vectors, and then use TruncatedSVD (Pe- dregosa et al., 2011) 1) to reduce the number of di- mensions to reasonable amount (e.g., 50) before feeding into t-SNE. For visual features, we conduct PCA (Pedregosa et al., 2011 1) each color channel to reduce the number of dimensions (e.g.. 30) well.
tabular features: cost, rarity, collectible, cardClass, set, attack.
text features: name, id, artist, text, me- chanics.
-1 image feature: image.
lol_sc: champion skin in LoL a cosmetic alteration to the appearance of the champion. De- pending on the rarity and price, a champion's skin belongs to specific category (for lol_sc). It is worth noting that the champion skins stylish decorations that have nothing to do with race, gender, or other unethical variables.
3 tabular features: id, price, soldInGame. skinName, concept, model, particles, animations, sounds, releaseDate.
- 7 text features:
For a neat presentation, we select subgroup of categories in the hs_mr dataset and assign different colors to samples belonging to different categories. For the fused raw features, we simply concatenate the three single modality features without any fur- ther modifications. The 2D projection of fused fea- tures is obtained following the same procedure, in unimodal features. As can be seen in these four subfigures, samples from different categories are clustered together no matter what modality is used as the input. The second row of Figure shows the
- I image feature: image.
esg_sq: Similar to champion skin in LoL, CS:GO skins alter the appearance of weapons. The pre- diction target is the skin quality according to its rarity (for sq).
- tabular features: id, availability skinCate- gory, minPrice, maxPrice.
text features: skinName.
- image feature: image.

---

Tab raw feats
Txt raw feats
Img raw feats
Fused raw feats
90
50
20
20
10
10
10
10
20
-20-
-40
-15
20
-25
Tab trained embs
Txt trained embs
Img trained embs
Fused trained embs
20
15-
10
10-
-5-
-80
-20-
-20
-50
BEAST
DEMON
MURLOC
EMENTALL
MECHANICAL
Figure 7: 2D t-SNE projections of raw features (the 1st row) and trained embeddings (the 2nd row), based on unimodal (Ist to 3rd columms) or multimodal (the 4th column) inputs.
2D t-SNE projections obtained from embeddings of models trained on tabular, textual, visual, and fused modalities, where these embeddings are ob- tained from the output of the penultimate layers. The models used are tabMLP, RoBERTa, Vit, and MUGNET (evaluated baselines as in Sec 4). Compared to the t-SNE projections using raw fea- tures, all projections using trained embeddings pro- vide better insights into categorical structures the data. Among all subfigures, the one using fused em- beddings is significantly better than the others, in which the separation between different categories is almost perfect with only small number of points mis-clustered. In summary, MUG is multimodal- dependent that requires multimodal information to distinguish samples from different classes.
embedding dimension of each categorical feature as min(100, 1.6 * num came_ctt all hidden layer sizes as 128, and the number of layers 4. For RoBERTa, we use the "RoBERTa-base" variant. For Electra, use the "Electra-base- discriminator" variant. For ViT, use the "vit base catee_aachh___20000 224" variant. For SWIN, we use the "swin Swwin_base_patiee wwwpeeeeeach__idddoooooooo variant. For AutoGluon, we use its "multimodal- best_quality" preset. For AutoMM, we use its default preset. For our own MUGNET, we optimize it using AdamW with a learning rate set 0.001 and cosine annealing learning rate schedule. Regarding the graph sampling strategy, set the number of root nodes generate random walks as 80% of the original number of nodes, and the length of each random walk as 2. MUGNET chooses other hyperparameters via HPO (Liaw et al., 2018). The search space includes the sample-wise similarity function sim e [cosine sim, RBF kernel, k-nearest neighbor) used in Equation (1). More specifically, (i) when sim cosine sim or sim RBF kernel, HPO of MUGNET also search along their associated graph sparsity hyperparameter spy E {0.5, 0.75, 0.95). (ii) when sim := k-nearest neighbor, MUGNET search along k E (5, 10, 32].
C Implementation and Hyperparameters of Baselines
We implement the evaluated models using open-source codebases (Ke et al., 2017; Paszke et al., 2019; Wang, 2019; Erickson et al., 2020; Shi et al., 2021: Erickson et al., 2022), and models' hyperparameters without specification the maximum
set default. For GBM, number of leaves in one tree as 128, the minimal number of data inside one bin 3, and the feature fraction ratio in one tree as 0.9. For tabMLP, we follow (Erickson et al., 2020) to adaptively set the


Reviews:
[{"review_id": "FmOcRBZzmA", "reviewer": "Reviewer_TBbD", "summary": "", "questions": "A. It's a bit uncommon to use a graph-based neural network for classification tasks on tabular/visual/text tasks. What is the reason/intuition for using a graph-based architecture for MUGNet?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer TBbD,    \n\nWe'd like to thank you for your careful review and valuable comments. We believe the constructive feedback will improve the manuscript and increase its potential impact on the community.\n\n---\n\n### Responses to \"Reasons to Accept\"\n\n> It is also uncommon to have a multimodal classification dataset from the video games domain, which makes this dataset more valuable for the research community.\n> The authors included comprehensive analysis on the data and features of the dataset to show that all modalities are needed to solve the tasks. \n\n**Response**: We thank the reviewer for the encouraging comments.\n\n---\n\n### Responses to \"Reasons to Reject\"\n\n> In the experiments, some of the tasks (such as lol_sc) can already be almost perfectly solved by benchmarked models, which makes them less challenging or valuable for the research community (although most of the tasks still remain challenging).\n\n**Response**: Thanks for pointing out that the *loc_sc* task appears relatively easy for existing multimodal classifiers, with the best benchmarked model achieving an accuracy of 96.8%. Although there is still room to improve, we agree that this particular task is relatively easy. Despite this, the remaining five tasks in the dataset still pose significant challenges. Your insight will guide us in designing future benchmarks that strike the right balance between being challenging and valuable for advancing the field. Thank you for your constructive input.\n\n---\n\n### Responses to \"Questions For The Authors\"\n\n> It's a bit uncommon to use a graph-based neural network for classification tasks on tabular/visual/text tasks. What is the reason/intuition for using a graph-based architecture for MUGNet?\n\n**Response**: The decision to opt for a graph-based architecture stems from the exciting potential demonstrated by Graph Neural Networks (GNNs) in various classification tasks. While GNNs have been predominantly applied in graph-related settings, inspired by the rationale that samples with similar tabular/textual/visual features tend to be assigned with the same category label, we believe that MuGNet is powerful and efficient in capturing such signals and it provides a novel perspective to solving multimodal classification tasks. Our empirical results also support this intuition.\n\n**Action**: In the revised manuscript, we will add discussions as such to motivate the design of MuGNet in Sec.4.2.\n\n---\n\n### Responses to \"Missing References\"\n\n> In general, I believe there should be more discussion about what MUG provides that the listed existing datasets can't provide in section 2.1. Also, while the authors included several domain-specific multimodal benchmarks, there are also existing benchmarks for multimodal classifications that are more general and not restricted to certain modality combinations that needs to be compared to MUG, such as MultiBench [1] and MMBench[2].\n\n**Response**: We greatly appreciate your input and the references you provided. Both MultiBench and MMBench are relevant works. MultiBench is indeed an insightful work, offering a diverse range of 15 multimodal datasets. However, it's important to note that MultiBench lacks tasks that encompass the combination of tabular, textual, and image modalities simultaneously. Regarding MMBench, we didn't include it due to its release date in August 2023, which fell after the EMNLP submission deadline.\nAs Sec.2.1 states, there exist very few tabular-text-image classification benchmarks. Among them, PetFinder focuses on pets, and HAIM-MIMIC-MM focuses on the healthcare domain. MugNet provides another rare video game domain. This serves as an essential contribution to diversifying the available benchmarks and addressing underexplored areas.\n\n**Action**: We will cite and discuss the missing references in the revised manuscript.\n\n---\n\n### Response to \"Typos Grammar Style And Presentation Improvements\"\n\n> Figure 4 is overlapped with the last row of text above it (line 324). Also, Figure 4 is too small for readers to see the content details.\n\n**Action**: We will fix the overlapping issue of Figure 4 in the revised manuscript. Regarding the size issue, we will be able to provide a larger version of Figure 4 in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a valuable new dataset for multimodal classification problems. The dataset contains data from 3 different modalities, and some features are naturally missing which makes the tasks even more challenging. It is also uncommon to have a multimodal classification dataset from the video games domain, which makes this dataset more valuable for the research community.\n\nThe authors included comprehensive analysis on the data and features of the dataset to show that all modalities are needed to solve the tasks. Moreover, they benchmarked the dataset on several well-known unimodal and multimodal models, and demonstrated how the dataset/benchmark can be used to comprehensively evaluate multimodal classification models, including performance and performance-speed tradeoffs. Also, on several tasks, even the best-performing models are still quite far from perfect, meaning that the tasks remains challenging for future researchers to explore.", "reasons_to_reject": "In the experiments, some of the tasks (such as lol_sc) can already be almost perfectly solved by benchmarked models, which makes them less challenging or valuable for the research community (although most of the tasks still remain challenging).\n\n", "questions_to_authors": "A. It's a bit uncommon to use a graph-based neural network for classification tasks on tabular/visual/text tasks. What is the reason/intuition for using a graph-based architecture for MUGNet?", "paper_topic_and_main_contributions": "This paper presents a new benchmark for multimodal classification benchmark based on game data. The data is collected from 4 vastly different video games with 8 total classification tasks, and the tasks involves information from 3 modalities (tabular, text and visual). There are features that are naturally missing for some data points, which made the tasks more challenging. The authors then benchmarked several different unimodal and multimodal models on the tasks, including several well-known models as well as a new model called MugNet. The experiments showed that (1) the tasks do require all modalities to solve well and (2) while existing models can do reach a decent performance, some tasks remains challenging.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 4 is overlapped with the last row of text above it (line 324). Also, Figure 4 is too small for readers to see the content details.\n\n"}, "extra_text": {"Missing_References": "In general, I believe there should be more discussion about what MUG provides that the listed existing datasets can't provide in section 2.1. Also, while the authors included several domain-specific multimodal benchmarks, there are also existing benchmarks for multimodal classifications that are more general and not restricted to certain modality combinations that needs to be compared to MUG, such as MultiBench [1] and MMBench[2].\n\n[1] https://arxiv.org/pdf/2107.07502.pdf\n[2] https://arxiv.org/pdf/2212.01241.pdf"}}, {"review_id": "NZ0GjHRQgH", "reviewer": "Reviewer_SX92", "summary": "", "questions": "I would like to thank the authors for their work. I have the following questions that also elaborate on my concerns above.\n\nA. The dataset and tasks were not clear to me. For instance, the target classes for each game made little to no sense to me, and I believe this will also be true for readers who are not familiar with the games. This is important because for the dataset to be adopted widely by the multimodal learning community, its structure should be interpretable to a wider audience. I would encourage the authors to include additional details about the task (for instance, what are the target classes (not simply stating their names but what it means -- for instance, predicting the \"skin category/quality\" could mean so many things without proper elaboration), and including qualitative examples for each of the eight tasks).\n\nB. Related to my previous comment, it was not clear to me how important is evaluation on datasets from games. I could imagine a similar tabular+text+image dataset being curated from Wikipedia (WIT does this to some extent: https://github.com/google-research-datasets/wit). It would help if the authors discussed the motivation for a game-based dataset and the challenges/opportunities it presents that datasets from other domains cannot present. \n\nC. Besides a new dataset, the authors propose a new approach to MugNET. An ideal evaluation of the effectiveness of the proposed approach would involve comparing its performance against baselines on existing benchmark datasets. However, the authors limit their evaluation to the new dataset. This limits the inferences about the applicability of the proposed approach as it is not empirically shown to generalize to other tasks. I would encourage the authors to specifically motivate why the tasks within the proposed benchmark warrant a new approach and how MugNET delivers on those requirements.\n\nD. During data cleaning, the authors mention that they group all the classes with small representation in the overall dataset into one single target class. I wonder how does this compare to dropping these classes? In the former approach, the grouped target class could have high heterogeneity, which could raise questions around the purpose of that class.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer SX92,\n\nThanks for your careful consideration. We greatly appreciate the positive comments and address the concerns below.\n\n---\n\n### Responses to \"Reasons to Accept\"\n\n**Response**: Thank you for acknowledging our contributions on the benchmark creation, data analysis, proposed baselines, and comprehensive experiments.\n\n---\n\n### Responses to \"Reasons to Reject\"\n\nWe will address each point along with its related questions in \"Questions For The Authors\".\n\n> My major concern with the paper is that the dataset is not well-motivated.\n\n> (Q.B) Related to my previous comment, it was not clear to me how important is evaluation on datasets from games. I could imagine a similar tabular+text+image dataset being curated from Wikipedia (WIT does this to some extent: https://github.com/google-research-datasets/wit). It would help if the authors discussed the motivation for a game-based dataset and the challenges/opportunities it presents that datasets from other domains cannot present.\n\n**Response**: We appreciate your thoughtful feedback on the motivation behind the MuG benchmark.\nMuG's significance extends beyond merely constructing a benchmark for enhancing game domain classifiers. It is also beneficial to the investigation of innovative strategies and techniques in underrepresented domains where the large-scale pre-training samples are limited, and where novel data distributions are prevalent.\nImportantly, as highlighted in Sections 1 and 2.1, MuG addresses a critical gap in the realm of tabular-text-image multimodal classification. Existing benchmarks rarely encompass all three modalities simultaneously, and this gap is crucial in reflecting the diversity of real-world applications. For instance, tasks like 48-hour mortality prediction in HAIM-MIMIC-MM or adoption prediction in PetFinder, showcase the practical relevance of MuG.\nFurthermore, our decision to collect data from the video game domain is grounded in the objective of introducing diversity to the multimodal learning community. This domain offers unique challenges and opportunities that datasets from other general domains, such as Wikipedia, may not provide. \n   \n**Action**: We will certainly enhance the discussion in the Introduction and Related Work sections to elaborate on these motivations and distinctions.\n\n> several crucial details that are needed to understand the datasets and tasks are missing from the paper.\n\n> (Q.A.) The dataset and tasks were not clear to me. For instance, the target classes for each game made little to no sense to me, and I believe this will also be true for readers who are not familiar with the games. This is important because for the dataset to be adopted widely by the multimodal learning community, its structure should be interpretable to a wider audience. I would encourage the authors to include additional details about the task (for instance, what are the target classes (not simply stating their names but what it means -- for instance, predicting the \"skin category/quality\" could mean so many things without proper elaboration), and including qualitative examples for each of the eight tasks).\n\n**Response**: Thank you for the valuable suggestion. Please note again that one major motivation for us to collect data from video games for prediction tasks there is to bring diversity to the multimodal learning community. That being said, we should have better elaborated the prediction targets and provided more real instances, to make MuG more accessible for potential readers and researchers who are not familiar with the incorporated video games.\n\n**Action**: Regarding the prediction targets, we plan to add the following detailed elaboration about the source video games and the prediction targets in Appendix B.1.\n\n- Pok\\'emon: Pok\\'emon is a popular video game centered around fictional creatures called \"Pocket Monsters\" that trainers capture and train to battle each other. Regarding the prediction targets, Pok\\'emons are categorized into various elemental **types**, such as Fire, Ice, Normal (non-elemental), and more. Each Pok\\'emon can have one primary type (for pkm_t1) and one secondary type (for pkm_t2). \n- HearthStone: HearthStone is an online collectible card game developed by Blizzard Entertainment, featuring strategic gameplay where players build decks and compete against each other using a variety of spells, minions, and abilities. Regarding the prediction targets, each card belongs to one certain **category** (for hs_ac) such as minon, spell, weapon, etc. Each card is part of a **set** (for hs_as) where new card sets are released periodically to introduce new content and strategies to the game. Each minion card is essentially a creature, thus it can be divided into different **races** (for hs_mr). Each spell card. Similarly to the minion race, each spell card belongs to a specific **school** (for hs_ss) such as Shadow, Nature, *etc*.\n- League of Legends (LoL): League of Legends is a widely popular multiplayer online battle arena (MOBA) video game developed by Riot Games where teams of players compete in fast-paced matches, utilizing unique champions with distinct abilities to achieve victory. Regarding the prediction targets, a champion skin in LoL is a cosmetic alteration to the appearance of the champion. Depending on the rarity and price, a champion's skin belongs to a specific **category** (for lol_sc).\n- \"Counter-Strike: Global Offensive\": CS:GO is a popular multiplayer first-person shooter video game developed by Valve and Hidden Path Entertainment, where players join teams to compete in objective-based matches involving tactical gameplay and precise shooting. Similarly, skins alter the appearance of weapons. The prediction target is the **skin quality** according to its rarity (for csg_sq).\n\nFurthermore, we will provide more real instances of the eight tasks in MuG.\n\n> The proposed approach is only evaluated on the new dataset, and it is not clear how effective it will be on existing tasks over the baselines considered in the paper\n\n> (Q.C) Besides a new dataset, the authors propose a new approach to MugNET. An ideal evaluation of the effectiveness of the proposed approach would involve comparing its performance against baselines on existing benchmark datasets. However, the authors limit their evaluation to the new dataset. This limits the inferences about the applicability of the proposed approach as it is not empirically shown to generalize to other tasks. I would encourage the authors to specifically motivate why the tasks within the proposed benchmark warrant a new approach and how MugNET delivers on those requirements.\n\n**Response**: It's important to clarify that our submission falls under the \"Resources and Evaluation\" track, where the primary focus is on the creation and benchmarking of the MuG dataset. MuGNet is introduced as one of the baseline methods, not really an important novel design. That being said, we understand your concern regarding the limited evaluation scope on MuGNet. Our intention is to avoid diverting attention from the core contribution of this paper. We agree that a thorough evaluation of MuGNet's performance on existing benchmark datasets is an important aspect that merits further investigation, such as in a follow-up research paper.\n\n> Some of the design choices made while curating the dataset are not well-motivated/discussed\n\n> D. During data cleaning, the authors mention that they group all the classes with small representation in the overall dataset into one single target class. I wonder how does this compare to dropping these classes? In the former approach, the grouped target class could have high heterogeneity, which could raise questions around the purpose of that class.\n\n**Response**: The suggested dropping strategy is an interesting alternative for data cleaning. We did not try that during our data cleaning. Indeed, we only conduct grouping strategy on *hs_ac* and *lol_sc*. The number of samples in sparse classes is relatively small in MuG.\n\n---\n\n### Response to \"Typos Grammar Style And Presentation Improvements\"\n\n> The anonymous link to the benchmark (in the abstract) was not accessible to me\n\n**Action**: We have tested the anonymous link and it is accessible now (sometimes the anonymous link may be out of work due to the downtime of the anonymous Github service). \n\n> Line 189 seems to have a typo: diversity instead of diverse?\n\n**Action**: Thanks for pointing out the typo. We will fix it and carefully check for other typos in the revised version.\n\n> Figure 4 overlaps with some of the text above it\n\n**Action**: We will fix the overlapping issue in the revised manuscript.\n\n> A broader comment about the structure of the paper: several critical details about the dataset and proposed approach are in the appendix instead of in the main paper. I would suggest the authors move some of the analyses to the appendix to be able to keep the technical details of the dataset and the approach in the main paper, as those are the key contributions of this work\n\n**Action**: Thanks for the valuable suggestion. We will be able to add more technical details to the main paper when submitting the camera-ready version. \n\n---\n\n### Response to \"Justification For Ethical Concerns\"\n\n> The authors mention that one of the sub-tasks is predicting the champion's skin category in the LoL game (line 175 and Table 3 and appendix). They do not provide additional context about what this target variable means in the context of the game (with which I am personally unfamiliar). It would be good to assess if this poses any ethical concerns. More broadly, there is a lack of discussion around other target variables within this benchmark dataset which makes it a bit opaque to evaluate (in terms of ethical concerns as well as usability for the community)\n\n**Response**: Thank you for bringing this up. We have provided a detailed elaboration of the prediction targets in \"Resposne to (Q.B)\". The champion skins in LOL are stylish decorations that have nothing to do with races/genders or other unethical variables. \n\n> The authors discuss the licensing of the data they have curated in B.1 (Appendix), but they do not mention the specific licenses for LoL and CS:GO.\n\n**Response**: For the data source of LoL and CS:GO, all data are collected from public web content, and no user-specific private information is collected.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The new dataset is publicly available and will add a new dimension to multimodal research (including tables beyond the usual vision and language modalities). The authors also claim that the dataset will be publicly available. \n- The authors also propose a new multimodal learning approach that models sample-similarity using within-modality graph neural networks and then aggregates the modality-specific representations.\n- The authors show experiments with several baselines and demonstrate the competitiveness of their proposed approach on the new dataset.", "reasons_to_reject": "- My major concern with the paper is that the dataset is not well-motivated, and several crucial details that are needed to understand the datasets and tasks are missing from the paper\n- The proposed approach is only evaluated on the new dataset, and it is not clear how effective it will be on existing tasks over the baselines considered in the paper\n- Some of the design choices made while curating the dataset are not well-motivated/discussed", "questions_to_authors": "I would like to thank the authors for their work. I have the following questions that also elaborate on my concerns above.\n\nA. The dataset and tasks were not clear to me. For instance, the target classes for each game made little to no sense to me, and I believe this will also be true for readers who are not familiar with the games. This is important because for the dataset to be adopted widely by the multimodal learning community, its structure should be interpretable to a wider audience. I would encourage the authors to include additional details about the task (for instance, what are the target classes (not simply stating their names but what it means -- for instance, predicting the \"skin category/quality\" could mean so many things without proper elaboration), and including qualitative examples for each of the eight tasks).\n\nB. Related to my previous comment, it was not clear to me how important is evaluation on datasets from games. I could imagine a similar tabular+text+image dataset being curated from Wikipedia (WIT does this to some extent: https://github.com/google-research-datasets/wit). It would help if the authors discussed the motivation for a game-based dataset and the challenges/opportunities it presents that datasets from other domains cannot present. \n\nC. Besides a new dataset, the authors propose a new approach to MugNET. An ideal evaluation of the effectiveness of the proposed approach would involve comparing its performance against baselines on existing benchmark datasets. However, the authors limit their evaluation to the new dataset. This limits the inferences about the applicability of the proposed approach as it is not empirically shown to generalize to other tasks. I would encourage the authors to specifically motivate why the tasks within the proposed benchmark warrant a new approach and how MugNET delivers on those requirements.\n\nD. During data cleaning, the authors mention that they group all the classes with small representation in the overall dataset into one single target class. I wonder how does this compare to dropping these classes? In the former approach, the grouped target class could have high heterogeneity, which could raise questions around the purpose of that class.", "paper_topic_and_main_contributions": "The authors propose a new dataset to aid the training and evaluation of multimodal models. The dataset comprises tables, text, and images from games. Additionally, the authors propose a new multimodal classification approach that models sample similarity in multimodal data using graph neural networks (GNNs). The authors conduct several analyses to provide insights into the datasets and do an exhaustive evaluation to compare the proposed approach with competitive baselines. The results demonstrate the effectiveness of their approach.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The anonymous link to the benchmark (in the abstract) was not accessible to me\n- Line 189 seems to have a typo: diversity instead of diverse?\n- Figure 4 overlaps with some of the text above it\n- A broader comment about the structure of the paper: several critical details about the dataset and proposed approach are in the appendix instead of in the main paper. I would suggest the authors move some of the analyses to the appendix to be able to keep the technical details of the dataset and the approach in the main paper, as those are the key contributions of this work", "Justification_for_Ethical_Concerns": "- The authors mention that one of the sub-tasks is predicting the champion's skin category in the LoL game (line 175 and Table 3 and appendix). They do not provide additional context about what this target variable means in the context of the game (with which I am personally unfamiliar). It would be good to assess if this poses any ethical concerns. More broadly, there is a lack of discussion around other target variables within this benchmark dataset which makes it a bit opaque to evaluate (in terms of ethical concerns as well as usability for the community)\n- The authors discuss the licensing of the data they have curated in B.1 (Appendix), but they do not mention the specific licenses for LoL and CS:GO."}}, {"review_id": "D5H9lWSzj4", "reviewer": "Reviewer_L86v", "summary": "", "questions": "I don't quite understand why \"Pidgey\" is labeled as \"Normal.\" When training a model on this sample, I'm uncertain whether the input-output mapping learned here can generalize to unseen data. For instance, can the model identify \"Snorlax\" as \"Normal\" based on the mapping learned from the Pidgey example?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer L86v,\n\nThanks for reviewing our manuscript. We appreciate your time and comments. Our replies to the comments are as follows.\n\n---\n\n### Responses to \"Reasons to Accept\"\n> The paper introduces a new benchmark dataset for multimodal learning, along with insights and experimental results that can guide future research in this area.\n\n**Response**: We appreciate the positive comments. \n\n---\n\n### Responses to \"Reasons to Reject\"\n> It is not clear to me how people can build a better multimodal model using the proposed new dataset.\n\n**Response**: We believe that the creation of a new benchmark in an underrepresented domain, accompanied by easily interpretable quantitative evaluation metrics, serves as a foundational step in the development of robust and generalized multimodal models. Our MuG datasets stand out due to their unique composition of tabular-text-image modalities, and originating from the relatively unexplored video game domain.    \nTo address the concern, let us delve into the rationale behind our approach. Firstly, the scarcity of benchmarks encompassing all three of these modalities simultaneously is highlighted in both Section 1 and Section 2.1.  Our benchmark thus fills a significant hole for the multimodality learning community. Secondly, our focus extends beyond merely constructing a benchmark for refining game domain classifiers. It is about investigating strategies and techniques to effectively train models in scenarios where an abundance of pre-training samples is lacking. Overall, MuG facilitates rapid adaptation to novel data distributions, which is a critical aspect of real-world applicability.    \n**Action**: We would add more discussion on the motivation and utility of MuG in the revised manuscript.\n\n> Perhaps it would be more convincing to demonstrate how benchmarks for real-world tasks improve when using a model trained on this dataset.\n\n**Response**: Thanks for the suggestion. It would be interesting to see if models pre-trained on MuG would deliver better performance on other datasets. However, we believe it is a bit unfair to say datasets and tasks for video games are not \"real-world\", so themselves are not worth studying. The whole video game industry is a huge real-world industry, and this work is filling a big hole for this industry, as thus for the multimodality learning community as well. \nSpecifically, we would like to emphasize that this paper focuses on data collection and benchmarking. We manually curate and clean the datasets, and provide comprehensive data analysis to show how all modalities are necessary to solve the classification task. Regarding the benchmarking part, we evaluate six unimodal classifiers, two multimodal classifiers, and our proposed GNN-based multimodal classifier. Some subtasks remain challenging for powerful transformer-based models, which can be used to measure technical designs of multimodal classifiers. We believe even without considering how much algorithms developed for the video game industry can apply to other domains, our current contributions are significant regarding creating the first domain-specific benchmark in the particular industry.\n\n**Action**: We will add the idea of \"pre-training on MuG, test on other benchmarks\" as an important future direction in Sec.6 Conclusion.\n\n---\n\n### Responses to \"Questions For The Authors\"\n> I don't quite understand why \"Pidgey\" is labeled as \"Normal.\"\n\n**Response**: Thank you for pointing out the confusion introduced by our toy example. Table 3 of Appendix B shows that each sample in the Pok\\'emon dataset can have one or two types. For the specific sample *Pidgey* in Figure 1, its type_1 is *Normal*, and its type_2 is *Flying*. It is easy to induce Pedgey's type_2 from its bird-like appearance and abilities such as *Keen Eye*. The type_1 *Normal* is a bit tricky. In the world view of Pok\\'emon, the *Normal* type is considered a *non-elemental* type. Since *Pidegey* has no abilities related to *Fire* or *Ice*, and it has no elemental-related appearance such as a burning fire on the tail, *Pidegey* is considered as *Normal* for its type_2.    \n**Action**: We will modify the toy example in the revised manuscript. For *Pidgey*, we will use \"type_2=*Flying*\" as the illustration to make the toy example more accessible.\n\n\n> When training a model on this sample, I'm uncertain whether the input-output mapping learned here can generalize to unseen data. For instance, can the model identify \"Snorlax\" as \"Normal\" based on the mapping learned from the Pidgey example?''\n\n**Response**: When models infer whether *Snorlax* has a *Normal* type, models learn the shared patterns of training examples that are in the same type (*e.g.* no elemental related abilities or appearances).     \nMoreover, experimental results support that multimodal classifiers can learn from the multimodal patterns of training samples for identifying Pok\\'emon types. Table 2 (b) shows best classifier achieves *77.4* and *66.9* accuracy in type_1 and type_2 tasks, while the worst classifier only achieves *12.0* and *40.6*. The performance gap verifies that a better understanding of the multimodal input features delivers better performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper introduces a new benchmark dataset for multimodal learning, along with insights and experimental results that can guide future research in this area.", "reasons_to_reject": "It is not clear to me how people can build a better multimodal model using the proposed new dataset. Perhaps it would be more convincing to demonstrate how benchmarks for real-world tasks improve when using a model trained on this dataset.", "questions_to_authors": "I don't quite understand why \"Pidgey\" is labeled as \"Normal.\" When training a model on this sample, I'm uncertain whether the input-output mapping learned here can generalize to unseen data. For instance, can the model identify \"Snorlax\" as \"Normal\" based on the mapping learned from the Pidgey example?", "paper_topic_and_main_contributions": "The paper introduces a new dataset for researchers to improve upon in the field of multimodal learning. The benchmark includes eight datasets from various game genres, covering tabular, textual, and visual modalities, along with multiple baselines and experimental results from state-of-the-art classifiers. The paper also provides multi-aspect data analysis to demonstrate the diverse and multimodal-dependent properties of the dataset. This work might facilitate research on multimodal learning.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Summary:**
This paper presents a new benchmark in the field of multimodal learning for multimodal classification, leveraging on game data. The data is collected from four vastly different video games with a total of eight classification tasks, each tasks involves information from three modalities: tabular, text and visual data. 
The authors conduct benchmarks involving different unimodal and multimodal models, including several well-known ones. Additionally, the authors propose a new multimodal classification approach, called MugNet, which models sample similarity in multimodal data using graph neural networks (GNNs).
The authors execute an exhaustive evaluation to compare the proposed approach against competitive baselines. The results demonstrate the effectiveness of their approach. This work might facilitate research on multimodal learning.

**Strengths:**
The reviewers acknowledge the following strengths in this contribution:
1. The paper introduces a valuable new dataset for addressing multimodal classification challenges, with the authors committed to making the dataset publicly available.
2. The rarity of a multimodal classification dataset derived from the domain of video games enhances its significance within the research community.
3. The authors propose an innovative multimodal learning approach that leverages within-modality graph neural networks to model sample similarity and subsequently aggregates modality-specific representations.
4. Comprehensive data and feature analyses are included in the paper, illustrating the indispensability of all modalities for solving the tasks.
5. The authors conduct benchmarking on the dataset using various well-established unimodal and multimodal models, highlighting how the dataset and benchmark can facilitate comprehensive evaluations of multimodal classification models, encompassing performance and performance-speed tradeoffs.
6. The experimental results offer valuable insights for guiding future research in this domain.

**Weaknesses:**
Reviewers share certain concerns with this paper. The dataset lacks clear motivation, and critical details necessary for comprehending the datasets and tasks are absent. The proposed approach is solely evaluated on the new dataset, and its effectiveness against existing tasks and baseline models considered in the paper remains unclear. It might be more persuasive to showcase the improvement in benchmarks for real-world tasks achieved by employing a model trained on this dataset. Additionally, some of the tasks in the experiments can already be solved nearly perfectly by the benchmarked models

**Author-Reviewer discussion and acknowledgment:**
The authors provided clarifications addressing the concerns raised by the reviewers and have delineated the improvements to be made, during the rebuttal response and discussion phase. All reviewers have responded and acknowledged the authors' arguments.

**Conclusion:**
The paper is well-structured, but reviewers suggest that the authors rectify the identified typos. Furthermore, reviewers recommend that the authors include additional references.
Decision: Accept-Findings