Table 4: Performance on maximally covered datasets.

Arch. || AlexNet || Agg. || Mean || Type/Eval || SL

Arch. || AlexNet || Agg. || Mean || Type/Eval || MEN

Arch. || AlexNet || Agg. || Max || Type/Eval || SL

Arch. || AlexNet || Agg. || Max || Type/Eval || MEN

Arch. || GoogLeNet || Agg. || Mean || Type/Eval || SL

Arch. || GoogLeNet || Agg. || Mean || Type/Eval || MEN

Arch. || GoogLeNet || Agg. || Max || Type/Eval || SL

Arch. || GoogLeNet || Agg. || Max || Type/Eval || MEN

Arch. || VGGNet || Agg. || Mean || Type/Eval || SL

Arch. || VGGNet || Agg. || Mean || Type/Eval || MEN

Arch. || VGGNet || Agg. || Max || Type/Eval || SL

Arch. || VGGNet || Agg. || Max || Type/Eval || MEN

Source || Wikipedia || Text

0.310

0.682

0.310

0.682

0.310

0.682

0.310

0.682

0.310

0.682

0.310

0.682

Source || Google || Visual

0.340

0.503

0.334

0.513

0.358

0.495

0.367

0.501

0.342

0.512

0.332

0.494

Source || Google || MM

0.380

0.711

0.370

0.719

0.379

0.711

0.365

0.716

0.380

0.714

0.365

0.716

Source || Bing || Visual

0.325

0.567

0.316

0.554

0.310

0.526

0.303

0.520

0.304

0.551

0.289

0.507

Source || Bing || MM

0.373

0.727

0.360

0.725

0.364

0.723

0.350

0.724

0.361

0.727

0.349

0.719

Source || Flickr || Visual

0.234

0.483

0.224

0.441

0.238

0.407

0.236

0.385

0.243

0.460

0.226

0.385

Source || Flickr || MM

0.350

0.715

0.343

0.711

0.347

0.689

0.344

0.703

0.354

0.702

0.339

0.696

Source || ImageNet || Visual

0.313

0.561

0.313

0.561

0.341

0.540

0.411

0.603

0.404

0.584

0.401

0.578

Source || ImageNet || MM

0.362

0.713

0.362

0.713

0.373

0.719

0.401

0.731

0.427

0.727

0.412

0.723

Source || ESPGame || Visual

0.018

0.448

0.026

0.376

0.063

0.487

0.050

0.434

0.125

0.506

0.106

0.451

Source || ESPGame || MM

0.208

0.686

0.187

0.672

0.243

0.700

0.246

0.696

0.269

0.708

0.260

0.698

5.1 Maximum coverage comparison. Table 4 shows the results on the maximally covered datasets. This means we cannot directly compare between data sources, because they have different coverage, but we can look at absolute performance and compare network architectures. The first row reports results for the text-based linguistic representations that were obtained from Wikipedia (repeated across columns for convenience). For each of the three architectures, we evaluate on SimLex (SL) and MEN, using either the mean (Mean) or elementwise maximum (Max) method for aggregating image representations into visual ones (see Section 2). For each data source, we report results for the visual representations, as well as for the multi-modal representations that fuse the visual and textual ones together. Performance across architectures is remarkably stable: we have had to report results up to three decimal points to show the difference in performance in some cases. For each of the network architectures, we see a marked improvement of multi-modal representations over uni-modal linguistic representations. In many cases, we also see visual representations outperforming linguistic ones, especially on SimLex. This is interesting, because e.g. Google and Bing have full coverage over the datasets, so their visual representations include highly abstract words, which does not appear to have an adverse impact on the method’s performance. For the ESP Game dataset (on which performance is quite low) and ImageNet, we observe an increase in performance as we move to the right in the table. Interestingly, VGGNet on ImageNet scores very highly, which seems to indicate that VGGNet is somehow more “specialized” on ImageNet than the others. The difference between mean and max aggregation is relatively small, although the former seems to work better for SimLex while the latter does slightly better for MEN.