Self-Guided Masked Autoencoders for Domain-Agnostic Self-Supervised Learning
Authors: Johnathan Wenjia Xie
Yoonho Lee
Annie S Chen
Chelsea Finn
Venue: ICLR 2024 poster (2024)
Keywords: self-supervised learning
domain-agnostic learning
masked modeling
protein biology
chemistry
particle physics
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Masking inputs based on a model's attention map can create a strong, domain-agnostic masking strategy for masked modeling.

Abstract: Self-supervised learning excels in learning representations from large amounts of unlabeled data, demonstrating success across multiple data modalities. Yet, extending self-supervised learning to new modalities is non-trivial because the specifics of existing methods are tailored to each domain, such as domain-specific augmentations which reflect the invariances in the target task. While masked modeling is promising as a domain-agnostic framework for self-supervised learning because it does not rely on input augmentations, its mask sampling procedure remains domain-specific. We present Self-guided Masked Autoencoders (SMA), a fully domain-agnostic masked modeling method. SMA trains an attention based model using a masked modeling objective, by learning masks to sample without any domain-specific assumptions. We evaluate SMA on three self-supervised learning benchmarks in protein biology, chemical property prediction, and particle physics. We find SMA is capable of learning representations without domain-specific knowledge and achieves state-of-the-art performance on these three benchmarks.

Published as a conference paper at ICLR 2024
SELF-GUIDED MASKED AUTOENCODERS FOR DOMAIN- AGNOSTIC SELF-SUPERVIH LEARNING
Johnathan Xie Stanford University
Yoonho Lee Stanford University
Annie S. Chen Stanford University
Chelsea Finn Stanford University
ABSTRACT
Self-supervised learning excels in learning representations from large amounts of unlabeled data, demonstrating success across multiple data modalities. Yet, extend- ing self- sufe-superiieed learning to new modalities is non-trivial because the specifics of existing methods are tailored each domain, such as domain-specific augmen- tations which reflect the invariances in the target task. While masked modeling is promising as conainnagnostic framework for self-supervised learning because it does not rely on input augmentations, its mask sampling procedure remains domain-specific. We present Self-guided Masked Autoencoders (SMA), a fully domain-agnosted masked modeling method. SMA trains attention based model using a masked modeling objective while learning masks to sample without any domain- specific assumptions. We evaluate SMA three self- supervipe learning benchmarks in protein biology, chemical property prediction, and particle physics. We find SMA is capable of learning representations without domain-specifie knowl- edge and achieves state-of-the- performance on these three benchmarks.
INTRODUCTION
Self-supervised learning is a powerful framework for learning representations and is appealing especially due to its ability to leverage large amounts of unlabeled data. The pursuit of scaling self-supervised learning to larger datasets has led to impressive results the natural language processing (Devlin et al. 2018 Brown et al. 2020 Raffel et al.) (020)) and computer vision (Chen itt al., (2200) He et al. Oquab et al. 2023) domains. While many works in machine learning have aimed to improve results by introducing domain-specific inductive biases, various studies
[ee
have observed that the advantages of inductive biases diminish at large data scales (Dosovitskiy
et al. 2020 Geirhos et al. 2018]] [Sttonn 2019). Therefore, it is striking that many self-supervised learning methods rely closely on Somain-specifte knowledge (Chen et al. 2020 Grill et al., 2020 Caron et 202T.. Simultaneously, the domain-specific choices make it difficult to apply these self-
supervised learning methods to new domains. Motivated by the potential for improved performance and greater applicability to new domains, we aim to develop a Nomain-agnosen self-supervised learning algorithm.
One common class of self- self-supervised learning approaches is contrastive learning (Chen et al. 2020 Tian et al.) 2020 Oord et al.] 2018 Wu et al. (0200) which operates by encouraging models to represent similar examples in nearby parts of the representation space. Despite the simple core idea, these methods necessarily rely on domain-specifie augmentations generate examples with similar semantic content (Chen et al. 2020 Oord et al. 2018 Grill et al. 2020). An alternative approach to self-supervised learning masked modeling (Devlin al.) 2018 He al., 2022) where the model is trained to reconstruct parts of the input that are masked out. Towards the goal of domainn-gnosse self-supervised learning method, masked modeling is promising because the idea of masking out parts of an input is widely applicable and various works show strong performance across a diverse array of data modalities (Devlin al. 2018 He al., 2022 Xie et al. [022 Baevski et al. 2022 2020) Yu et al. [2022] Wang et al. 2022).. However, generalizing these methods across different domains remains challenging because data modalities differ in how raw input values are related. Previous methods have used domain-specifie mask sampling methods or different tokenizers for each domain.
"Correspondence to jwxie @ stanford.cdu Wee make the code available this link

---

Published as a conference paper at ICLR 2024
While these strategies have been effective, they make masked modeling non-trivial extend to new domains because they require the development of new tokenizers which are also error prone, lack robustness, and are difficult to maintain (Xue et al. 2022 Jaegle et al. 2021b). Rather than rely on tokenizers or domain -omain-specific assumptions, we demonstrate that relations between raw inputs can be learned during masked pre-training which in turn simultaneously drives the sampling of useful masks for masked modeling.
The main contribution of this paper is Self-guided Masked Autoencoders (SMA), a self-supervised learning method based on masked modeling that is entirely domain-agnostic as it does not use any form of tokenizer or priors regarding the structure of raw inputs. For both cross-attention (Lee et al.) 2019) Daegle et al. (221a)) and self attentonnn /Vaswani et al. (017)) architectures, we compute masks based on the attention map of the first encoding layer during masked prediction training. This way of producing input masks is motivated by recent observations that such attention maps correspond to semantic regions of the input (Jaegle al. a Caron et al. 202T]. We perform experiments on three data modalities: protein biology, chemical property prediction, and particle physics, and on all three domains, SMA achieves state-of-the-art performance while remaining domain-agnostic. Our results demonstrate that much of the domain-spectie knowledge used by prior methods to guide model training can be found within the unlabeled data.
RELATED WORK
Masked modeling. A common method for self-supervised learning is for a model to receive to receive partially masked input sequence and learn to predict the masked values. Prior works have
used this approach in the language domain (Devlin et al. 2018 Liu et al. 2019 Raffel et al.
2020
Lewis et al. (019)) and shown that it scales well to large datasets and models (Chowdhery et al. 2022
Brown et al. (020). Recent works extend masked modeling to the image (He et al. [022 Xie et al.
2022), video (Feichtenhofer al.) 2022). audio (Schneider al. 2019 Baevski et al. [020 [uu et al.
2022). and point cloud (Yu et al. 2022] Pang et al.] 2022) domains. A key advantage of masked
modeling approaches that they are substantially less reliant on domain-specific augmentations (He
ct al.
approaches (Chen et al. (2020) Oord et al., 2018: Grill ct al. 2020) and are more stable as they are not susceptible to representation collapse. However, masked modeling methods still require tailored domain-specift masks or tokenizers to learn useful representations (Xie et al., 2022] Raffel et al. 2020). In comparison to prior works, we propose a system to generate a challenging masking objective based on a network's 's attentions, alleviating the need for domain-specific masks and tokenizers.
2022) compared to joint-embedding
Damain-agnostic learning. With the introduction of the Transformer architecture Vaswani et al. 2017), which can effectively process sequences of data vectors with few assumptions, representation learning methods have become more unified across domains. Though the Transformer has become ubiquitous for processing high-level input tokens, the preprocessing required for each domain still differs drastically and effective tokenization methods
non-trivial to develop for new domains. Recently, new cooss-attention-basee architectures (Jaegle et al. 2021b a Lee et al.) 2019 Yu et al.) 2023) have enabled processing of raw inputs such as pixels or characters. These architectures demonstrate a path to truly aggnott learning with a model that can be applied out of the box to any domain. Our method leverages these new architecture designs to create end-to-end domain-agnostic self-supervised learning without the need for tokenization.
Learning corruptions. Related to our work are methods for automatically learning augmentations and corruptions for representation learning. Early methods simply involved determining which hand-designed augmentations apply to a training example (Cubuk et al., 2018 Lim al., 2019
Cubuk et al.) 2020 [uzzkii 2022), however recent approaches (Suzuki] [022]] Tamkin et al. 2020
have leveraged fewer inductive biases with regards to the domain by automating a majority of the augmentation pipeline. Most related to our work are methods which learn to sample masks in contrastive learning (Shi et al. (002) and video pretraining (Bandara et al. (023). Compared to prior works, ours does not rely the use of any domain-specific tokenization additional augmentations to learn representations.
Deep learning for science. Recently there have been many advances in applying deep learning techniques in various scientific domains. These methods often leverage scientific knowledge of each domain to design architectures (Brandes al. 2022 Zhou et al. 2023) or representation learning
2

---

Published as a conference paper at ICLR 2024
2023 Ahhmad al. 2022 p. However, these methods reliant few known invariances (Zhou et al., 2023 Qu and Gouskos 2020) within each scientific domain or additional engineered features (Brandes et al. 2022) improve training which is challenging to extend to new domains. In comparison to these works, we propose a framework to learn representations from seratch in data-driven manner without reliance domain knowledge leading more generally applicable representation learning. We demonstrate that strong masking policy that corresponds to contiguous parts can be learned without any inductive priors or tokenization. Our modeling method demonstrates a path toward representation learning that is based solely on provided data and is not reliant on any prior knowledge (LeCun al. 2015).
methods (Zhou al.)
BACKGROUND
In this section, we provide an overview of the masked prediction objective which we build on top of in our method, attention-based modeling, and we formally define the concept of donain-agnostic learning.
3.1 MASKED PREDICTION
Our method is based on masked prediction, a commonly used objective for self-supervised learning. Given only subset of the input tokens, and the remaining tokens masked, the model is trained to predict these masked tokens. We consider an input sequence consisting of n tokens X (Ci...., C.). We denote the masked and unmasked indices as M C (1,..., n) and U .1. , M, with a total of m masked tokens and u = n mn unmasked tokens. The task of predicting the masked tokens {C given the unmasked tokens vee is most informative when the model must rely on the semantic context given by the unmasked tokens to make accurate predictions. Because many modalities such as image or video data have near-duplication or strong dependencies in their low-level inputs a model can simply learn to predict masked tokens by interpolating the value of the nearest unmasked tokens, which is not very helpful for representation learning.
Thus, to ensure that the model learns to infer semantic context from the unmasked tokens, the masking procedure must carefully designed to ensure that the masked prediction problem is not trivial. For example, in the language domain, the masking procedure typically masks out contiguous span of tokens such as a word (Devlin et al. 2018 Liu et al., 2019: Raffel et al. 2020)) In the image domain, the mask typically corresponds to contiguous region of pixels (He et al.) 2022 Xie et al., (022). As we are interested in developing a fully aagnssi method, we refrain from assuming any prior knowledge about the structure of low-level inputs. Instead, we propose data-driven approach to producing masks that correspond to contiguous parts.
3.2 DOMAIN-AGNOSTI LEARNING
The Perceiver (Jaegle et al. 2021b) architecture introduces a notion of domain -aguosii process by arguing that full shuffling of input tokens along with their positional embeddings should produce an equivalent output. We extend this definition to a full learning process order to formalize the definition of domain-agnostic that we adhere throughout our paper. Formally, for unlabeled training set of & raw data samples, (X(1), ...... ((k)), where each ( E Raxdraw let P represent the set of all permutations of indices {1...., n]. A learning process L for a network N can be defined as domain-agnoster agnostic if
Vp E C(N, (X(1) x()))) C(N, (xx()) .p.(()))()))
The process of reshaping the raw modality inputs into the shape n x draw must only involve knowledge of what dimensionality the data is and is generally just simple flattening operation with the possibility of padding if necessary.
This definition of domain-agnostic differs from the definition of permutation invariance in that requires strictly consistent shuffling of the individual tokens within each data sample (*) across the dataset. This means that positional information can still be learned by the network, but there no prior knowledge of how related two input tokens are or their causality.
3

---

Published as a conference paper at ICLR 2024
Attontion Matrix
Sum
Training Objective: Mesieed Inpuis
Ieputs, +
(a) Compute Attention Map
(b) Randomly Select Queries
(c) Masked Prediction
Figure 1: Self-guided Masked Autoencoders (SMA). (a) We first extract the cross-attention map from latent queries onto the full input array. (b) Next, add the rows of the attention matrix corre- sponding randomly selected subset of the queries, and produce input mask by selecting the largest values of the attention sum. (c) Our pre-trate pre-training objective is to reconstruct the original inputs from the masked input sequence.
3.3 ATTENTION
To compute masks in a completely domain-agnostic fashion, we leverage information pertaining to semantic groups encoded in attention maps (Jaegle et al. 2021b Caron al. (02T). Two common methods for computing attention across an imput are self-attention (Vaswani et al. 2017) where
query and key values are projected from the same set of vectors and cross-attention (Jaegle al.) 2021b; Lee et al. 2019) where query values are projected from a separate set of vectors. To compute attention maps, first we must embed the raw inputs X E IR Reaxdraw by using linear projection E E @draw Xdembed and positional embeddings learned from scratch P E Innxdembed to produce XE+P-
Paxdembed, From these inputs, produce attention keys Ke E R=xds using linear projection. The queries for attention Qe E Rlxdi can either be projected from the embedded inputs X in the case of self-attention, or learnable latents in the case of cross-attention From these queries and keys, can compute the initial unnormalized attention map:
A QeKe Vdh
T
ER
As our method is applicable to both self-attention and cross-attention maps, we do not specifically denote the difference between these cases in the rest of the paper.
4 SELF-GUIDED MASKED AUTOENCODERS
In this section, we describe self- sulf-guided masked autoencoders (SMA), a comain-agnostic ap- proach to self-supervised learning that samples masks from the masked prediction model's 's own attention representations. Our core mask extrac- tion procedure is visualized in Figure and we provide overview of the full training loop in Algorithm
Algorithm 1 SMA training
Input: Data D {x("....) reconstruction loss L, initial parameters l. masking ratio r.
while Not converged do
X D
A + Attention XC)
Sec. 3.3
Á +- ApplyM ask(
(5)
Intuitively, masking continguous groups of highly correlated tokens is desirable for masked modeling approaches because requires more complex modeling of the input, which in turn facilitates the learning of representations that capture important features. Thus, our goal is to identify and mask tokens that are highly correlated in a single group. One potential way to achieve this is by leveraging the masked prediction model's own internal attention representations, which have been shown to correspond to semantic regions of the input (Jaegle et al.) 2021b) Caron et al., 2021). By using these attention representations to mask highly correlated and redundant tokens. This is especially
H Encode 00; Á, X)
Sec. 4.2 b (6)
O < U psample (f; H)
0 (V)((O,))
Sec. 4.2
Return
guide the masking process, important in the scientific domains we explore, where identifying all correlated parts the input would otherwise require specialized knowledge of the scientifie domain.
In the of self-attention,
4

---

Published as a conference paper at ICLR 2024
4.1 SELECTING TOKENS TO MASK
We now describe our method for selecting tokens for masking from an attention map (2) 2) of either self-attention (Vaswani al. 2017) cross-attention (Lee et al. 2019 Jaegle b02 We find in practice that each query assigns large, attention to highly corre latd groups of inputs. Therefore, to mask all the inputs of single "group," we can mask the top attentions of given query. However, the naive application of this masking algorithm is problematic for a few reasons. First, because the top attention tokens chosen for each query have significant overlap, we often do not actually mask the target masking ratio, and this issue worsens as n increases. Second, the iterative nature of the algorithm leads to poor complexity with the repeated top-k operations, and poor parallelization.
Instead, we can compute an approximation of this procedure that we find to be equally effective in practice while achieving the desired masking ratio and parallelizing well.
To select masked indices, we first randomly choose subset of query indices R C [I,.... Then, with masking ratio r the function to generate a mask of nr elements for given attention map A is defined as
0 KeepTopK(rr (v, = {
u; is in the top elements of v otherwise.
(3)
(1I Aois
Attention.ask(AA/ KeepTopk
Σ A
Σ SE
Ans
(4)
VE
EE
We follow Shazeer et al. 2017 in defining the K KeepTopK operation. In practice, attention archi- tectures often use multi-headed (Vaswani et al.] 2017) approach capture different relations in single attention layer. Therefore, rather than an attention matrix A E R'x", the attention matrix is often of shape A E hxxxxn To select queries be masked across heads, we find we can aggregate the attentions after softmax normalization, but prior to the masking operation by summing across the head dimension. This aggregation operation performed only for the purpose of computing a mask, and does not affect the final attention mask. Then, we can mask attentions across heads such that there is no information flow from masked tokens. With this, the resulting masked attention matrix after masking can be written as
-=
SoftMar(A)), [nr1) + A). A).B.
34
(5)
This full operation can be viewed as approximation of selecting the top attentions of subset of the queries then masking them by setting their values to large negative value prior to the softmax operation. Our masking design is especially efficient and in practice does not add any significant compute as it only requires an extra summation, top-k, and softmax operation. Additionally, the entire masking operation is contained only the first layer the masked inputs have attention value and are dropped the layer output preventing any information flow to the model (He et al. 2022).
The full masking operation is visually summarized in parts (a) and (b) of Figure
4.2 MASKED RECONSTRUCTIOS TRAINING
Following the initial attention encoding, these hidden vectors are further processed using repeated Transformer (Vaswani et al. 2017) layers. To create reconstruction predictions, we follow a process similar to prior works (He et al.) 2022. Jaegle et al., 2021a). Specifically, we upsample the processed hidden vectors H E I to same dimensionality as the input by projecting decoding query values Q, E x from P. the model's positional embeddings. Additionally, project keys Ko IRlxda and values V, E [gldd, from H. Using these values, we compute the initial output:
KoKo Vdz
0 SoftMar
VV. Raxd,
6
"The one-dimensional masking vector expanded in the two initial dimensions, times then times order to fully mask across these dimensions. [r]] means integer rounding.
5

---

Published as a conference paper at ICLR 2024
Method
Domain Agnostic Remete Homology
Fluorescence
Stability
ProteinBERT Brandes
2022
0.22
0.66
0.76
No Pretrain
0.10
0.60
0.66
Random Masking
0.22
0.67
0.76
(ouns)
Table 1: Transfer learning comparison on TAPE Benchmark (Rao et al., 2019) downstream tasks. We report accuracy for Remote Homology and Spearman correlation for Fluorescence and Stability.
Then, we compute raw output predictions by using an additional MLP to project O to the original raw value dimension draw- Finally, compute reconstruction loss using either mean-squared error for continuous raw values or cross-entropy for discrete raw values. This reconstruction loss is computed only on masked indices m as shown in part (c) of Figure The masked prediction model parameters are updated via mini-batch gradient descent to minimize the reconstruction loss. Importantly, only single masked prediction model is used during the entire training process, as our method learns to sample masks while simultancously optimizing the masked prediction objective.
EXPERIMENTS
In this section, we aim answer the following experimental questions:
1. On wide range of domains, can SMA learn masks that are effective for learning represen- tations masked modeling without any domain knowledge?
2. Can SMA outperform prior nomain--seciffe methods on representation learning benchmarks across wide range of domains using both self-attention and cross-attention architectures?
To test the efficacy of our method we study pre-training over an unlabeled training set followed by downstream transfer some related task within the same domain. We compare against the prior state-of-the of-the-art in the protein biology, chemistry, and particle physics domains. Values in the protein biology and particle physics domains are the mean of three random seed values, while in the chemistry domain we take the mean of values across three scaffold splits commonly used by prior works (Ramsundar et al. (2019) For all experiments, we provide further details on hyperparameters and model configurations in Section B the appendix.
Model Architecture. It is important that our method is applicable to both self-attention and cross- attention architectures since often the form of many modalities have very long sequence lengths causing standard self-attention to be too costly to compute over such long sequences. Therefore, in the protein domain we adopt cross attention architecture based on the Perceiver IO (Jaegle et al. a02 model. However, the max sequence length in the chemistry and particle physics domains are relatively short, 256 and 28 respectively. Therefore, we use standard transformer self-attention architecture (Vaswani et al.] (017) in these domains. In order to vectorize inputs for discrete modalities such as text, protein, and molecule sequences we map each character to learned vector which is then summed with learned positional embedding. In continuous domains such as images and particle physics, we use a linear projection for each continuous input to project them to the same dimension as positional embeddings which are then summed.
5.1 PROTEIN PROPERTY PREDICTION
Unlike well studied discrete domains such as natural language, the protein biology domain does not have well-established effective inductive biases or tokenizers. We apply our method in this domain and find that it is capable of learning non-trivial masking strategy and outperforms hand designed biases. We follow the benchmark setting of TAPE (Rao et al. 2019: Tamkin et al.) 2021 which involves pre-training over the Pfam training set for 10 epochs. To assess self-supervised representation quality, we fine-tune over two regression tasks (Fluorescence and Stability) where we report Spearman correlation and classification task (Remote Homology) where we report accuracy.
We find that SMA is able to outperform both the previous state-of-the-ee aart methods and the random masking baseline. Compared the prior state-of-the-art work of BRRt (Brandes al. 2022

---

Published as a conference paper at ICLR 2024
Domain Agnostic
ROC (higher better f)
RMSE (lower better J)
BRBP
BACE
HIV
ESOL
PresSioly
Lipo
ChemBERTa-2
(hhmd
MLM-5M
0.701
0.793
0.986
MLM-IOM
0.696
0.729
1.009
MLM-77M
MTR-5M
X
0,742
0.698
0.734
0.735
0.758
0.987
MTR-10M
0.733
0.783
0.744
MTR-77M
0.728
0.799
0.798
Uni- Mol--MM--M Zhou nl. 2023
X
0,729
0.857
0,808
0.788
1,48
0.603
0.763
0.730
0.877
1.58
Random Masking-1.6M
0.723
0.798
0.765
0.642
1.22
0.631
SMA-L66 (Ouns)
0,750
0.843
0.789
0.623
1,09
0.609
Table 2: MoleculeNet property prediction comparison using deepchem Ramsundar et al. (0019)
scaffold splits. We compare against the state-of-the-arr SMILES- based model, ChemBERTa-2 Ahmad et al. 2022). and the satte-ofthheaase graph-based model Uni-Mol Zhou et al.
2023
Methed
Accunacy (6)
Tabker
Phster 2021
Accuracy (%)
Architecture
Pretraining
1k
10k
100k
TahNet
Arik and Pfister
2021
Nome
57.47
66.66
72.92
TabNet
Ank and Phister
A72T
Learned Masking
61.37
68.06
73.19
Transformer
Vaswani et
2017
None
65.27
70.86
74.05
Callloo
XGBoast 166 12i0nir 2011
Dobiliz 200 Can 2013
72.7
72.6
Tranffommrr
Vaswan el al.
2017
Random Masking
68.07
72.57
76.54
FTT Gee-lay 2: MM41
72.9
Transfomerr
Vaswam el 2017
Guided Masking (Ours)
69.47
74.04
77.88
Tonehrmiica perali stall 7p
750
Tranforme: Sertch
72.7
(a) Transfer learning comparison on HIGGS benchmark test set. We
Tandormer Rasdon Madking
76.11
finetune pretrained models over training subsets of different sizes to assess self-supervised learning quality.
Tondlomer Gnikel (aknn (Oun)
74.8
(b) HIGGS small benchmark classi- fication accuracy. Random masking and Guided masking methods are pre- trained over only the HIGGS small training set.
which uses domain knowledge to design a custom architecture and additional sequence labels in their supervisory task, our method improves sule-suueo via more challenging masks.
5.2 CHEMICAL PROPERTY PREDICTION
We compare SMA to the previous state-of-the-art works on MoleculeNet regression and classi- fication tasks (Wu et al. 2018) which span a wide range chemical property prediction tasks. Prior works mainly ly either graph-based approaches which represent molecules 3D graphs, or SMILES [Weininger 1988) strings. We choose to represent input molecules as SMILES strings which allows us apply our method with minimal modification. Whereas other SMILES- use chemistry specific tokenizers that aggregate molecule information, for the sake of generality we choose map strings to discrete values using simple UTF-8 character mapping. Following prior work, we use the cross validation splits from deepchem (Ramsundar et al. 2019) which uses scaffold splitting to test generalization across di different molecule types. We compare against the state-of-the-art model,
methods
and the mate-offtheeart graph-based model, Uni-Mol. We pre-train on the training set of the guacamol (Brown et al. 2019) dataset containing 1.6M molecules, an order of magnitude smaller than the training sets of both ChemBERTa-2 (Ahmad et al. 2022) and Uni-Mol (Zhou et al. 2023). Our method outperforms all 6 ChemBERTa- configurations on the three shared benchmark tasks and outperforms Uni-Mol three out of six benchmark tasks while being comparable in a fourth. Compared to Uni-Mol which uses atom location information and augments their data by generating atom conformations, our method is far simpler and is direct generalization of our algorithm to this discrete domain.
EMILES--assee
5.3 PARTICLE PHYSICS PROCESS CLASSIFICATION
We use the HIGGS Whiteson, 2014) particle physics dataset where the task is to distinguish between
Higgs Boson process and background process. The input consists of 21 kinematic values and 7 hegh-level features calculated from the kinematic features. We benchmark under two different settings. First, we compare using the same self-supervised learning setting as TabNet (Ark and Phister 2021
7

---

Published as a conference paper at ICLR 2024
Pretraining Method
Domain Agnostic
MNLI-(m/mm)
QQP
QNLI
SST-2
CoLA
STS-B
MRPC
RTE
Avg
392k
363k
108k
3.5k
Word
x
79.0
88.3
87.4
89.9
40.8
82.2
82.3
54.5
75.6
None
43.5
75.4
61.6
60.3
0.0
28.0
70.3
49.1
48.5
Random
75.9
88.7
84.7
86.8
36.2
83.4
76.2
55.2
73.4
SMA (ours)
77.5
89.1
87.4
88.3
80.9
62.1
76.7
Table 4: Transfer learning comparison on GLUE benchmark evaluation set. Tasks are ordered in decreasing order of training set size. We report F1 score for QQP and MRPC, Spearman's p for STS-B, and accuracy for other tasks. We find that SMA outperforms the three baselines (No Pretrain, Random, and Word) with the same architecture, and at the current scale, we find that our method can outperform the popular word-masking method by approximately 1.5% when averaged across GLUE tasks.
where we pre-train over the full 10.5m sample training set, then fine-tune over a selected subset of the training set of size 100,000, 10,000, and 1000 samples and report binary classification accuracy in Table 3a Second, we consider a supervised tabular dataset split consisting of 62,752 training samples and 19,610 test samples |Gorishniy et al. 2021). Under this setting pre-train only over the same labeled set available to all supervised methods then fine-tune of the labeled set and report results in Table 3b Because HIGGS is common tabular data benchmark, all prior methods technically domain-agnostic as there does not exist general -omain-specific knowledge pertaining to tabular datasets.
We find that under the self- supervuueee learning setting in Table 3a SMA is able to provide a consistent improvement in classification accuracy over both no pre- pre-training and random masked pre-training across all fine-tuning dataset sizes. Additionally, we find our pre-training method to yield substantially greater gains over supervised models when compared to the learned masking method of TabNet (Arik and Pfister 2021) which aims to mask out irrelevant features. Under the supervised tabular split in Table 3b] where the pre-training set and fine-tuning labeled sets are identical we still find SMA yields a substantial improvement. Whereas all prior supervised methods differ by only 0.6% accuracy, SMA able to outperform the previous state-of-the art by 1.9% demonstrating our pre- pre-training method can yield significant gains unmatched by using only supervised training.
5.4 COMPARISON WITH
MASKS
While our method is best used for domains without strong priors such as the aforementioned scientific domains, we aim to compare the viability of generated masks for representation learning with hand- designed masks from well studied domains such as the image and language domains. Additionally, we qualitatively assess generated masks which are included in Section Of the appendix. Due to these two domains having long sequence lengths in their raw form, we use cross attention architecture for both domains.
5.4.1 NATURAL LANGUAGE TASKS
For natural language pre-training we use a concatenation of English Wikipedia and BooksCorpus and assess the representation quality by fine-tuning the GLUE (Wang et al. 2018) benchmark excluding the WNLI dataset as is common practice since BERT (Devlin et al. 2018). We average these scores over three fine-tuning seeds. To map text sequences discrete values, use a basic UTF-8 character level mapping (Jaegle et al. 2021a).
To assess representation quality, we evaluate the development set of the GLUE dataset and report the unweighted average of F1 scores for QQP and MRPC, spearman correlations for STS-B, and accuracy for other tasks in Table 4 We use a single set of hyperparameters across all tasks which is further described in the appendix along with additional details on our model architecture and pre-training hyperparameters
We compare our method to three alternatives using the same architecture: (1) No pre-training, (2) Random masking, (3) Word-level masking, a domain- speciffc masking method. For word masking, we reproduce the results of Perceiver 10 (Jaegle et al. 2021 where a word is considered sequence
'Due to computational constraints our model and pre-training set size considerably smaller than the original paper.
8

---

Published as a conference paper at ICLR 2024
Pretraining Method
Domsin Agnostic
w/o augmentation
w/ augmentation
Patch
79.6
84.3
None
52.5
62.7
Random
77.6
85.6
SMA (ours)
78.5
86.1
Table 5: Top-1 accuracy transfer learning comparison on ImageNet-100 evaluation set. We
consider setting of both using augmentation and not using augmentation during downstream finetuning. All models are first pretrained for 200 epochs over the training set using different masking methods. We find that our method outperforms random masking both with and without augmentation during downstream classification.
of space delimited characters. SMA outperforms the three baselines with the same architecture, and at the current scale, we find that our method can outperform the 1.5% when averaged across GI.E tasks.
method by approximately
5.4.2 IMAGE CLASSIFICATION
To assess image domain self supervupe learning, we pre-train all models on the ImageNet- 100 Tian
iet al., (2200) subset. In all pre-training runs we also resize images to resolution of 192 x 192.
Though this does require some small amount domain knowledge, is computationally infeasible to process ImageNet (Deng et al. 2009) images at their raw resolution which varies widely and would require a context length in the millions. To assess representation quality, we fine-tune on ImageNet-100 100 under two settings, one with augmentation where we apply mixup (Zhang et al. 2017). cutmix YYun et al. 2019), solarization, gaussian blur, grayscale, and color jitter, and one without any augmentations applied during fine-tuning.
We present ImageNet- 100 results in T We find that our self-guided masking method performs well both with and without augmentation compared to the three baselines of (1) No pre-training, (2) Random masking, and (3) Masking random 16 x 16 patches. We found that with well-tuned strong augmentations, the random masking model was capable of outperforming the patch masking model. This may be due to a combination of a few factors. First, the the random masking method is generally the best learning positional embeddings, which quite important in the image domain given the large set of positional embeddings. Second, we found the random masking model required a high drop path rate (Huang et al. 2016) 0.3 to perform well, which is not applied for either the patch masking method nor our learned masking method. We suspect the high drop path rate can somewhat mitigate the issue of trivial solution involving interpolating nearby pixels as conjunction with the high masking rate, it is unlikely for the adjacent pixels of a masked pixel to be recovered. Finally, it is quite possible that many of the representations that are learned through pre-training processes could be eventually recovered with well tuned augmentation since the pre-training and fine-tuning sets are the same for this setting (Touvron et al. 2022).
For this reason, test fine-tuning without any augmentation and find the random masking model to perform worse than the other methods. In comparison, we find that our self guided masking method capable of learning both strong positional embeddings given that the generated mask shapes irregular compared to image patches that are consistent static squares, and when the learned masks have consolidated into full contiguous regions, our model can learn complex image representations similar to the patch masking model.
CONCLUSION
In this paper, we have presented method, Self-Guided Masked Autoencoders (SMA), for learning representations from unlabeled data without any prior knowledge about our input. By leveraging large-scale pre-training, our approach alleviates the need for hand-designed masks and inductive priors, making it especially effective in domains without well-studied priors. Our results across three scientific domains demonstrate that prior knowledge is not necessary to train competitive represen- tation learning methods. In summary, our approach offers a promising direction for unsupervised representation learning without the need for dopain-specific inductive priors.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We thank Eric Anthony Mitchell for advice on pre-training language models. We also thank Shirley Wu, other members of IRIS Lab, and anonymous reviewers for helpful discussions and feedback. The compute for this work was supported by a HAI Google cloud credit grant sponsored by Fei Fei Li. Chelsea Finn is a CIFAR fellow. This work was supported by NSF, KFAS, and ONR grant
N00014-20-1-2675.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
W. Ahmad, F. Simon, S. Chithrananda, G. Grand, and B. Ramsundar. Chemberta-2: Towards chemical foundation models. arXiv preprint arXiv:220.007/1, 2022. [page
S. Ô. Arik and T. Pfister. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages (679-6687, 2021. [page 7 16
Badirli, X. Liu, Z. Xing, A. Bhowmik, K. Doan, and S. S. Keerthi. Gradient boosting neural networks: Grownet. arXiv preprint 2rXiv:2002.07971, 2020. Ipage 7
A. Baevski, Y. Zhou, A. Mohamed, and M. Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems. 33: 12449-12460, 2020. [page]] 2
A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli. Data2vec: A general framework for self-supervised learning speech, vision and language. In International Conference on Machine Learning, pages 1298-1312. PMLR, 2022. [page]]
W. G. C. Bandara, N. Patel, A. Gholami, M. Nikkhah, M. Agrawal, and V. M. Patel. Adamae: Adaptive masking for efficient spatiotemporal learning with masked autoencoders. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14507-14517, 2023. Ipage 2
N. Brandes, D. Ofer, Peleg, N. Rappoport, and M. Linial. Proteinbert: universal deep-learning model of protein sequence and function. Bioinformatics, 38(8):2102-2110, 2022. [page] 2
N. Brown, M. Fiscato, M. H. Segler, and A. C. Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling. 5993)1006-1100; 2019. [page
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, al. Language models are few-shot learners. Advances neural information processing systems, 33:1 1111187799,, 2020. [page]]
M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self supervuped vision transformers. Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-96600 9660, 2021. [page 2
T. Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016. [page [6
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020. [page]] 2
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung,
C. Sutton. S. Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. [page[]
E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q V. Le. Autoaugment: Learning augmentation policies from data. arXiv preprint 1rXiv:1805.. 095001 2018. [page]
E. D. Cubuk, B. Zoph, Shlens, and Q. Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020. [page
J. Deng. W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 148-255. Teee, 2009. [page
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04005, 2018. [page [
11

---

Published as a conference paper at ICLR 2024
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXhv:2010.11929, 2020. [page]] D
C. Feichtenhofer, H. Fan, Y. Li. and K. He. Masked autoencoders as spatiotemporal learners. arXiv preprint arXiv:2205.09) 13, 2022. [page
R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel. Imagenet-trained biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811. 2018. [page]] !
Y. Gorishniy, I. Rubachev, V. Khrulkov, and A. Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34:18932- 2021. [page 16
Y. Gorishniy, I. Rubachev, and A. Babenko. On embeddings for numerical features in tabular deep learning. Advances in Neural Information Processing Systems, 55:24991-25000 2022. [page
J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, Doersch, Avila Pires.
Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances neural information processing systems, 18:211211-22844 2020. [page 2
K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 16000-16009, 2022. [page]
G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger. Deep networks with stochastic depth. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 646-661. Springer, 2016. Ipage 9
A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding, S. Koppula, D. Zoran, A. Brock, E. Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.144: 147.95,5,. 2021a. [page 5 6
A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pages 4651- 1- 4664. PMLR, 2021b. [page 3
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:
2014. Ipage 16
G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems, 30, 2017. [page 16)
Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature. 52 521((553):436 444, 2015. Ipage
J. Lee. Y. Lee, J. Kim, A. Kosiorek, S. Choi, and Y. W. Teh. Set transformer; A framework for -nnvaaaiiinnnnniianntt neural networks. In International conference on machine learning, pages 3744-3753. PMLR, 2019, [page 2
attention-basee
M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettle- moyer. Bart: Denoising sequence-to-sequenee pre-training for natural language generation, translation, and comprehension. arXiv preprint 19Xiv:1910.13400 2019. [page
S. Lim, I. Kim, T. Kim, C. Kim, and S. Kim. Fast autoaugment. In Advances in Neural Information Processing Systems, pages 6665 6675,66652220 2019. [page[]
Y. Liu, M. Ott, N. Goyal, J. Du M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoy- anov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692. 2019. [page 2
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 2017. [page 16
12

---

Published as a conference paper at ICLR 2024
B. McFee, T. In Proceedings of the 21st International Conference on World Wide Web, pages 909-916, 2012.
D. P. Ellis, and G. R. Lanckriet. The million song dataset challenge.
Ipage [6
A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprini arXiv: 1807.03748, 2018. [page 1
M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304. 07193, 2023. [page[] [
Y. Pang, W. Wang. F. E. Tay, W. Liu, Y. Tian, and L. Yuan. Masked autoencoders for point cloud self-supervised learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Parl II, pages 604-621. Springer, 2022. [page 2
S. Popov, S. Morozov, and A. Babenko. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint
wwtei//999003311 2019. Ipage 7 16
L. Prokhorenkova, G. Gusev, A. Vorobev, A. V. Dorogush, and A. Gulin, Catboost: unbiased boosting with categorical features. Advances in neural information processing systems, 31, 2018. [page]] 7
16
H. Qu and L. Gouskos. Jet tagging via particle clouds. Physical Review D, 101(5))056019, 2020. [page 3
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555 2020. [page I 3
B. Ramsundar, P. Eastman, E. Feinberg, Gomes, K. Leswing. A. Pappu, M. Wu, and V. Pande. Deepchem: democratizing deep-learing for drug discovery, quantum chemistry. Materials Science and Biology. https://gitthbb. comfdtepchem/ddeepheen (accessed Aug 8, 2017), 2:2785-2791, 2019. Ipage
R. Rao, N. Bhattacharya, N. Thomas, Y. Duan, Chen, Canny, P. Abbeel, and Y. Song. Evaluating protein transfer learning with tape. Advances in neural information processing systems, 32, 2019. [page]
S. Schneider, A. Baevski, R. Collobert, and M. Auli. wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint arXiv: 05862, 2019. [page
N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and Dean. Outrageously large neural networks: The sparsely-ges gatted mixture-of-exee expertsss layer. arXiv preprint 1 2017. Ipage 5)
Y. Shi, N. Siddharth, P. Torr, and A. R. Kosiorek. Adversarial masking for self sule-supervised learning. In International Conference on Machine Learning, pages 20026-20040. PMLR, 2022. [page] 2
W. Song, C. Shi, Z. Xiao, Z. Duan, Y. Xu, M. Zhang, and J. Tang, Autoint: Automatic feature inter- action learning via self-attentive neural networks. In Proceedings of the 28th ACM international conference on information and knowledge management, pages 1161-1170, 2019. [page 7 16
R. Sutton. The bitter lesson. Incomplete Ideas (blog), 13(1), 2019. [page]] D
Suzuki. Teachaugment: Data augmentation optimization using teacher knowledge. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10904 10914, June 2022. [page 2
A. Tamkin, M. Wu, and N. Goodman. Viewmaker networks: Learning views for unsupervised representation learning. arXiv preprint 2rXiv:2010.07432, 2020. [page[]
A. Tamkin, V. Liu, R. Lu, D. Fein, C. Schultz, and N. Goodman. Dabs: A domain-agnostic benchmark for self-supervised learning- arXiv preprint arXiv:2111. 2021. [page
13

---

Published as a conference paper at ICLR 2024
Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pages 776-794. Springer, 2020. [page 9
H. Touvron, M. Cord, and H. Jégou. Deit iii: Revenge of the vit. Computer Asion-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV, pages 516-533. Springer, 2022. Ipage 9
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017. [page 2
6 7
A. Wang. A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07450 2018.
ppage
R. Wang, R. Shivanna, Cheng, Jain, D. Lin, L. Hong, and E. Chi. Den v2: Improved deep cross network and practical lessons for web-scale learning to rank systems. In Proceedings of the web conference 2021, pages 1785-1797, 2021. [page]] 7
W. Wang. H. Bao, L. Dong. J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, et al. Image foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 10442, 2022. [page
D. Weininger. Smiles, chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36, 1988. [page
D.
Whiteson. HIGGS. https://bb.orr/002244//////33
UCI [page 7
Machine
Learning Repository, 2014.
DOI:
Z. Wu, B. Ramsundar, E. N. Feinberg, J. Gomes, C. Geniesse, A. S. Pappu, K. Leswing, and V. Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9(2):113-530, 2018. [page
Z. Wu, S. Wang, Gu, M. Khabsa, F. Sun, and H. Ma. Clear: Contrastive learning for sentence representation. arXiv preprint arXiv:2012 2020. [page]]
Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9653-9663, 2022. [page]
H. Xu, Li, A. Baevski, M. Auli, W. Galuba, F. Metze, C. Feichtenhofer, et al. Masked autoencoders that listen. arXiv preprint vrXiv:2207.064000 2022. [page]] 2
L. Xue, A. Barua, N. Constant, Al-Rfou. S. Narang, M. Kale, A. Roberts, and C. Raffel. Byt5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291-306. 2022. [page]]
L. Yu, D. Simig, C. Flaherty, A. Aghajanyan, I.. Zettlemoyer, and M. Lewis. Megabyte: Predicting million-byte sequences with multiscale transformers. arXiv preprint arXiv:2305.07185, 2023.
[page 2
X. Yu, I.. Tang, Y. Rao, T. Huang, J. Zhou, and J. Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19313-19322, 2022. [page
S. Yun, D. Han, J. Oh. S. Chun, Choe, and Y. Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032, 2019. [page
H. Zhang, M. Cisse. Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1710.09412 2017. [page 9
14

---

Published as a conference paper at ICLR 2024
G. Zhou, Z. Gao, Q. Ding, H. Zheng, H. Xu, Z. Wei, L. Zhang, and G. Ke. Uni-mol: A universal 3d molecular representation learning framework. In The Eleventh Internationaa Conference on Learn- ing Representations, 2023. Bt/aa/a/ss/aaaarrrrsssss//////rr
Ipage 2
15

---

Published as a conference paper at ICLR 2024
Method
RMSE (lower is better ()
TabNet (Arik and Pfister
202
8.909
Autoint
Song al 2019
8.882
Grow
Badiri of al 2020
8.827
Gonshiny
DON2
8.890
NODE
Pupoy el al 2019
8,784
ResNet
Gonsliniy et
2021
8.846
CatBoost
Prokhorenkova
2018
8.877
XGBoost Chen and Guesirin
2016
8.947
FT-T (Gorshmy et al. 2021
8.855
Transformer Seratch
8.809
Transformer Random Masking
8.762
Transfommrr Guided Masking (Ours)
8.695
Table 6: Year prediction for audio features.
(a) Unmasked refer- ence
(b) Patch mask visu- alization
(c) Random mask vi- sualization
(d) Learned mask vi- sualization view
(e) Learned mask vi- sualization view 2
Figure 2: Visualization of learned masks for ImageNett-100. We show two sampled mask views generated from the same model weights to demonstrate our method sample diverse masks even with static weights. We observe that masks are well clustered with respect to location and have minor emphasis color values.
Urmakal Refermes
Uezally, would lzaring uncand lising nooms plasing taya. But
minion hiss prusicclly cuattmi::
Randors Markin
Usaalls, le would cica acore the living coom.
Ni
ii in
Bus
one look minton uns ical eanamiic
Wood Marking
Usualls would
aound the roons plaine wit toss. But just look
sent fum peacicilly caucocic.
Leansed Mask
Usually sculd tearing cround
look mision
casmmnk.
Unaalls,
Table Visualization of learned masks on English Wikipedia + BooksCorpus. We find that our method learns mask views that are clustered around specifie positions with minor focus space delimiters. Actual sequences trained on are far generally far longer often nearly the max sequence size of 1024 therefore there may be some discrepancies in the actual masking ratio.
A ADDITIONAL TABULAR RESULTS
In addition the HIGGS tabular benchmark, also benchmark on the year prediction from audio features tabular dataset (McFee et al. 2012). We use the same settings as Gorishniy et al, (202T and report RMSE in Table 6
B ADDITIONAL IMPLEMENTATION DETAILS
For experiments, we optimize using Adam Kingma and Ba 2014) with decoupled weight de-
cay Loshchilov and Hutter (2017) and set 31 0.9, B2 0.999. Our other main hyperparameters are listed in Tables 8 to
C MASK VISUALIZATIONS
We visualize samples of masks generated for images in Figure and text in Table For each mask, we show two mask views generated using the same set of weights to demonstrate how our method create diverse masks with the same of weights which is crucial ensure the mask
16

---

Published as a conference paper at ICLR 2024
config
value sross
confi
rindom
guided
comfig
valuur
atcention type
Weight decay
weight decay
embed dimension
512
learning rate
le-4
3e-4
learning role
[le-4, (ee-4,
soguence length
1024
spocts
epocls
50 [64, 128,
number of layers
batch size
256
basch size
qk dimension
256
mosking
0.2
0.15
şk dimension
fauction
crossentropy
1024
(a) Protein biology architecture
(b) Protein biology pre-training
(c) Protein biology fine-tuning
Table Protein Biology hyperparameters
prediction model does memorize training samples. While our masks appear be more biased toward positional as opposed to value information we suspect this is because the set of attention queries from which we generate attentions static across all inputs. This means that the queries and thus attentions are calibrated to efficiently encode all inputs and therefore in the in this case positional information such as locality is generally more representative of input correlations across a whole dataset. For similar reason previous Perceiver iterations Jaegle et al. (2021b) used repeat modules that cross attended the input again after processing the initial set of latent vectors, however applying method to subsequent set of cross attentions is expensive it would require recomputing the entire block of layers again.
We find the generated image masks to be well clustered with respect position which is expected given the high degree of correlation between pixel token and it's adjacent values. Furthermore, due to masking method we find that the generated masks form continuously masked regions which creates a more challenging prediction task requiring complex reasoning over unmasked tokens.
D RUNNING TIME INFORMATION
In developing our method, we found the repeated top-k operation challenging to parallelize because must ensure each query masks unique tokens meaning the top-k operations need be done sequentially to exclude the indices that were masked by other queries. This led to the approximation that we describe in Equation 5 in our main paper. In addition to alleviating the bottleneck of needing to compute the top-k operations iteratively, the approximation also improves the time complexity of the total top-k operations which accounts for majority of the time in the masking process.
Suppose we select m queries from which we compute k inputs to mask for each from total of n inputs. Without our approximation, the average-case runtime complexity is (m k:)) as we must perform m top-k operations. In comparison, with our method, the average case complexity is only O(n + mn) because we just need to perform a single top-k operation with mk selected values. Since m, the number of queries selected is generally a fraction of n, and it is always the case that mk < (we cannot mask more than or equal to 100% of inputs), our approximation reduces the complexity of top-k costs from squared to linear with respect to (the number of inputs).
Empirically, the running time without our proposed approximation on the image domain 1.13 steps/second. In comparison, the approximation training runs at 3.58 steps/second. Note that with the approximation, the time to compute masks does not significantly impact train step time (less than 2% of the time per step), but without the approximation, the time compute masks becomes the bottleneck for large input counts. Additionally, empirically we found no difference in performance between the two methods across domains, and approximation actually improves stability in testing.
'For Higgs small experiments where the pre-train only 63k samples, we pretrain for 800 epochs.
17

---

Published as a conference paper at ICLR 2024
config
value
dicetion type
kelf-amtetioon
weightt deaay
0.01
weight docay
0.01
length
256
learning
le-4
learning
[le-4, (e-55
lagers
oprcchs
epechs
gk cimension
7788
basch size
12s
batch size
32
qk dimeusion
masking
loss Function
crosecutiogy
(a) Chemistry architecture
(b) Chemistry pre-training
(c) Chemistry fine-tuning
Table 9: Chemistry hyperparameters
confi
rundom
guided
allention type
velf-attenion
weight dreay
0.01
weight doeay
1e-5
length
28
learning
1e-4
learning
3e-5
number logers
spochs bauch size
epochs
100
qk dimension
128
256
batch size
126
gk
masking
0.2
dimension
128
loss function
meun squared error
(a) Particle physics architecture
(b) Particle physics pre-training
(c) Particle physics fine-tuning
Table 10: Particle physics hyperparameters
potch
Scranch
Milention type embed dimension
croes ertee-illettion
weight docay
0.01
weight docay
0.11
learning
Ic-4
learning rase
Ie-4
soquence length
36864
spochs batch size
epochs
200
300
mimber of layers
128
batch size
128
gk dimension gk hends
moskcing
loss flinction
ment-squrred erce
dimension
lacents
512
Image architecture
(b) Image pre-training
Image fine-tuning
Table 11: Image hyperparameters
guided
word
config
value
dicuntinn type
cross- dit--iitttionn
weight daeay
0:01
weight docay
0.01
embed dimension
512
learning
le-4
learning
2e-5
soguence longth
epochs
epochs
number of layers
16
batch size
256
butch size
32
gk
marking
gk hends
loss fuaction
crossestropy
dimension
1024
lacents
(a) Natural language architecture
(b) Natural language pre-training
(c) Natural language fine-tuning
Table 12: Natural language hyperparameters
18

---

Published as a conference paper at ICLR 2024
Queries
Reconstruciion Prediction
Decoding
Dccodiig Keys
Reconstruction
Transformrr Layere
Latentt
Quedee
Hidden
Hidden States
Masked Attontion Matris
Sum
Figure 3: End-to-end diagram of SMA
Overlap in Top Attentions For Image Domain
0.50
0.48
0.46
Overlap Ratio
0.44
0.42
0.40
0.38
0.36
10000
20000
30000
40000
50000
Number of tokens
Figure 4: Relationship between number of inputs and the top attention overlap ratio for images. We find as we increase the number of inputs, the overlap ratio increases as well.
19


Reviews:
[{"review_id": "tLyBwv828l", "reviewer": "Reviewer_36zB", "summary": "This paper presented a Self-Guided Masked Autoencoder for domain-agnostic SSL, with experiments largely focusing on data from scientific domains such as biology, chemistry, and physics. It selects tokens to mask via attention maps from the first encoding layer (either self-attention or cross-attention) and masks the tokens with high aggregated attention weights. The authors argue that such an approach masks highly correlated semantic regions regardless of domain priors. The authors show strong results in protein biology, chemical property prediction, and particle physics.\n\nThis paper works on an interesting topic with great potential impact and it is clearly written; however, the paper slightly lacks quality, therefore the reviewer recommends a borderline rejection.", "questions": "1. Page 5, “First, because the top attention tokens chosen for each query have significant overlap, we often do not actually mask the target masking ratio, and this issue worsens as $n$ increases.“\n* How much overlap did the top attention tokens have? Did the authors quantify them? How does this issue worsen as $n$ increases – a log or linear relationship?\n\n2. Page 5, “...while achieving the desired masking ratio and parallelizing well.”\n* Unfortunately, there are no follow-up experiments or discussions on better parallelization. Why is this the case, and how much parallelization improvement does the proposed method bring?\n\n3. Page 5, “let $\\mathcal{P}$ represent the set of all permutations of indices” and Eq.(1).\n* If $p$ is a permutation, what does $pX^{(i)}$ mean in Eq.(1)? And more importantly, why Eq.(1) defines domain-agnostic? It is not clear as there are no direct citations or proof supporting this claim (the Perceiver paper did not seem to include any specific math statement like this). The reviewer would appreciate more explanation on this part.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality: the method is inspired by Perceiver (Jaegle 2021b) and adapted the latent query technique to the attention module in masked autoencoder. Although the method is not particularly novel, the reviewer reckons that this is the first work to improve MAE’s domain-agnostic property via attention-map-based mask selection. There are other mask selection works, but they are primarily domain-specific [1, 2].\n\nClarity: the paper has a good flow and is, in general, easy to read.\n\nSignificance: domain-agnostic SSL is an important research topic as the community is seeing the merging of multi-domain, multi-modality pretraining. This paper serves as a nice step forward in this direction by using a generic attention-based mask selection technique for MAE pre-training.\n\n[1] Li, Gang, et al. \"Semmae: Semantic-guided masking for learning masked autoencoders.\" Advances in Neural Information Processing Systems 35 (2022): 14290-14302.\n\n[2] Wilf, Alex, et al. \"Difference-Masking: Choosing What to Mask in Continued Pretraining.\" arXiv preprint arXiv:2305.14577 (2023).", "weaknesses": "Post-rebuttal update: the responses include new experimental comparisons and successfully address all of the reviewer's concerns. Therefore, the reviewer updated the rating.\n\n------\n\nOriginality: the paper did not cite or compare other domain-agnostic SSL methods, either contrastive [3] or masking [4]. Also, the key components, latent query tokens (Jaegle 2021b) and the KeepTopK (Shazeer et al. 2017) are not novel, further weakening the originality of this work.\n\nQuality: the quality of this work is lacking. Empirical performance improvement can be limited, such as results for the MoleculeNet (Table 2), where the proposed method is sometimes worse than the baseline method (BACE and HIV of Uni-Mol-20M), or improvement can seem limited (Lipo). While a lower performance is common, since there are only limited baselines (TabNet for the HIGGS benchmark and baselines for the HIGGS benchmark are all from 2021 or prior) and this work is quite empirical, the performance difference can seem noticeable. Nevertheless, the reviewer is not familiar with the benchmarks and did not extensively search for new work with higher results, and will hugely benefit from a response from the authors explaining why the baselines are few. \n\nClarity: some parts of the paper seem confusing; the details are in the Questions section.\n\n[3] Verma, Vikas, et al. \"Towards domain-agnostic contrastive learning.\" International Conference on Machine Learning. PMLR, 2021.\n\n[4] Yang, Haiyang, et al. \"Domain invariant masked autoencoders for self-supervised learning from multi-domains.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LgjkppOY81", "reviewer": "Reviewer_zapB", "summary": "This paper proposes a new mechanism called Self-Guided Masked Autoencoders (SMA), that acts as a generic masking procedure in the embedding space, and is therefore agnostic to the nature of the input data. SMA is evaluated on a wide variety of tasks ranging from molecular and chemical property prediction tasks, to image classification and NLP tasks. A reasonable level of performance is demonstrated without domain-specific data augmentations throughout the tasks.", "questions": "1) How do you tune the masking parameters, such as the number of queries and input to mask ? (masking ratio) How difficult is it ?\n\n2) Do you need to mask queries ? Could you simply mask the input randomly ? Could you clarify if this corresponds to “Random Masking” in your tables ?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1) The idea of generic masking in the self-attention layers directly is novel and promising. Generic masking was explored previously with the data2vec models but here the mechanism seems to be more principled and applicable to any domain using transformers.\n\n2) The initial results on various tasks without domain-specific data augmentation is encouraging and might lead with further exploration to general and principled architectures for self-supervised learning on any kind of data.\n\n3) The results on biology and chemistry tasks are convincing and competitive with prior work.", "weaknesses": "1) The results on image classification and NLP tasks are only on toy datasets and seem to be too preliminary to convince people from these communities to try the approach. The gains are marginal and only a small set of methods are compared. The final model is far from the state-of-the-art in NLP and vision. I would recommend if possible to be more ambitious and demonstrate results on more large scale tasks such as linear evaluation on ImageNet.\n\n2) Some design choices are not well motivated and should be ablated properly. For example masking ratios, number k of elements in Eq.3", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hVnOjbbbGk", "reviewer": "Reviewer_ijD8", "summary": "This work tackles the problem of domain-agnostic self-supervised learning which does not assume any prior knowledge about the domain itself. The authors propose Self-Guided Masked Autoencoders (SMA) that computes masks based on the attention map of the the model at the first layer. SMA is shown to outperform random masking and other domain-specific baselines on a wide-range of tasks including protein/chemical property prediction, particle physics classification and natural language tasks.", "questions": "1. What are the hyperparameters in SMA? Is masking ratio the only hyperparameter? Can the authors explain why certain domains require a high masking rate compared to others (as mentioned in Table 7 to 11)\n2. How would the performance differ if a domain-specific tokenizer is used in Chemical property prediction?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem of domain-agnostic self-supervised learning is an important problem given it’s wide applicability particularly in deep learning for science. The proposed method is simple and elegant.\n- The authors show results on a wide-variety of domains with impressive performance without any domain knowledge. It is interesting to see that the results with SMA (without any domain-specific tokenizers) are comparable/better than other methods with domain knowledge.", "weaknesses": "- Missing important related work:\n    - DABS [1, 2] is a benchmark for domain agnostic self-supervised learning algorithms. This benchmark consists of semiconductor wafers, multispectral satellite imagery, protein biology, bacterial genomics, particle physics, Speech Recordings, Chest Xrays. DABS also has baselines in the form of Generalized masked autoencoding, Capri (Hybrid Masked-Contrastive Algorithm), e-Mix and Shuffled Embedding Detection (ShED). Demonstrating the effectiveness of SMA on this benchmark would strengthen the paper. I understand that the authors have already shown results on particle physics and protein datasets but comparing with these baselines would lead to a more complete results section. The authors can also discuss and compare SMA with these baselines.\n    - The authors should compare and contrast with related literature [3, 4, 5].\n- The authors can run some ablations to better understand the proposed method, SMA. For instance, how does the masking ratio impact performance in various domains? It may be interesting to analyze the performance if the masks are computed in second/ third layer (or any kth layer) instead of first layer in all the experiments.\n\n\n[1] Tamkin, Alex, et al. \"DABS: a Domain-Agnostic Benchmark for Self-Supervised Learning.\" *Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)*. 2021.\n\n[2] Tamkin, Alex, et al. \"DABS 2.0: Improved datasets and algorithms for universal self-supervision.\" *Advances in Neural Information Processing Systems* 35 (2022): 38358-38372.\n\n[3] Wu, Huimin, et al. \"Randomized Quantization: A Generic Augmentation for Data Agnostic Self-supervised Learning.\" *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 2023.\n\n[4] Lee, Kibok, et al. \"i-mix: A domain-agnostic strategy for contrastive representation learning.\" arXiv preprint arXiv:2010.08887 (2020).\n\n[5] Verma, Vikas, et al. \"Towards domain-agnostic contrastive learning.\" International Conference on Machine Learning. PMLR, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MHlHnPTNtd", "reviewer": "Reviewer_Md4o", "summary": "The paper introduces the Self-guided Masked Autoencoders (SMA), a domain-agnostic self-supervised learning technique based on masked modeling. Distinguishing itself from traditional self-supervised methods, which often incorporate domain-specific knowledge, the SMA refrains from using any form of tokenizer or making assumptions about the structure of raw inputs. Instead, it dynamically computes masks based on the attention map of the initial encoding layer during masked prediction training. The authors demonstrate SMA's effectiveness across three diverse domains: protein biology, chemical property prediction, and particle physics, where it achieves state-of-the-art performance without relying on domain-specific expertise.", "questions": "The main questions are listed in Weaknesses. I'd raise my score if they were appropriately addressed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1 - Domain-Agnostic: \n\nSMA is designed to be entirely domain-agnostic, ensuring it can be applied widely without needing domain-specific adjustments.\n\nS2 - Dynamic Mask Learning: \n\nRather than depending on fixed tokenizers or pre-defined masking strategies, SMA innovatively learns the relationships between raw inputs to determine useful mask sampling.\n\nS3 - Decent Performance on several datasets: On all evaluated benchmarks, SMA not only competes but surpasses the state-of-the-art, indicating its potential as a leading approach in self-supervised learning.", "weaknesses": "W1 Experiments:\n\nWhile the authors report the results of ImageNet100, the results on the full dataset are also expected to ensure a comprehensive evaluation. Additionally, I'm also curious about the pre-trained encoder's performance on segmentation and object detection tasks. \n\nFor tabular datasets like HIGGS, the results are promising. However, I'd suggest authors extend the work to broader tabular datasets as the performance of deep learning-based models may vary a lot. Additional experiments are not super expensive in this domain but will give a more comprehensive evaluation. \n\n W2 Training and Inference Efficiency:\n\nAuthors in this paper claim that the proposed feature space masking is efficient. However, unless I missed it, I failed to see related statistical results/analysis to prove such a claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EYLzj3Moym", "reviewer": "Reviewer_eSva", "summary": "The paper proposes a domain-agnostic masked modeling approach for MAE's in the context of self-supervised learning, which can operate across both self and cross-attention architectures as well as different domains. An interesting idea, with a coherent story, and partially promising results.", "questions": "1) One can argue masking highlight correlated areas on the input actually poses an easier job to the prediction model no? \n\n2) Why would repeated top-k(.) operations be too complex or hard to parallelize for different tokens? And why would (3) help to elevate this bottleneck?\n\n3) In Table 1,  why shouldn’t one pick random masking? It seems to function as good as SMA, and the prior art, no? It’s way more efficient as well.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper reads well, and has a coherent story. \n- The idea is somewhat novel, to my understanding, even though the impact might be marginal compared to a random equivalent. \n- Experiments are elaborate and diverse in different domains, with further supporting results in the appendix.", "weaknesses": "1) I am missing an end-2-end sketch of the proposed architecture. I understand the goal is to keep it somewhat agnostic to task, architecture (self/cross-attention etc), but essential components can still be sketched. Algorithm 1 (which appears by the way too early in the text) to some extent covers this, but not totally.   \n2) Am I right to assume (3) - (5) are a mathematical notation of Fig. 1? If so, this is one of those examples where math becomes more of a problem than a solution! \n3) The improvement SMA offers in settings without strong priors (such as protein property prediction, and chemical property prediction) is marginal, which is fine. However, here a simple random masking pattern seems to work pretty well, and that can be an efficient natural choice which is difficult to argue against. (See Tables 1, 2, 3 and 5). \n4) I think it would be useful to have some reflections on the computational complexity (time and/or model space) of SMA. Is it the reason why ImageNet-100 is chosen instead of ImageNet itself?\n\n5) The paper can benefit from another proof read; e.g. here are some minor suggestions:\n\n    a) On page 2: These architectures demonstrates … => Demonstrate\n\n    b) On page 3: Formally for an unlabeled … => Formally, \n\n    On Page 4: the inputs of a single \"group,\" => “group”, \n\n    On Page 9: also use resize image … ? Doesn’t read well.\n\n    And several others. \n\n6) Other suggestions:\n\n    a) I wouldn’t use capital letters for sets (but \\mathcal{}) to avoid confusion with a constant value. \n\n    b) I would avoid using two different l’s for the loss function and for query dimensions.\n\n    c) Clarify the notation of mixed floor/ceiling in (3), in the subsequent paragraph.  \n\n   d) Please refer to table numbers in the text, this way the reader has to look for table content to associate it with descriptions. Table 5 is all of a sudden cited in the text! \n\n    e) Even though self-explanatory to some extent, the algorithm is left unreferenced and unexplained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)