Efficient Heterogeneous Meta-Learning via Channel Shuffling Modulation
Authors: Minh Hoang
Carl Kingsford
Venue: ICLR 2024 poster (2024)
Keywords: Meta Learning; Deep Learning Architecture; General Machine Learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: Heterogeneous Meta Learning via Channel Shuffling Modulation

Abstract: We tackle the problem of meta-learning across heterogenous tasks. This problem seeks to extract and generalize transferable meta-knowledge through streaming task sets from a multi-modal task distribution. The extracted meta-knowledge can be used to create predictors for new tasks using a small number of labeled samples. Most meta-learning methods assume a homogeneous task distribution, thus limiting their generalization capacity when handling multi-modal task distributions. Recent work has shown that the generalization of meta-learning depends on the similarity of tasks in the training distribution, and this has led to many clustering approaches that aim to detect homogeneous clusters of tasks. However, these methods suffer from a significant increase in parameter complexity. To overcome this weakness, we propose a new heterogeneous meta-learning strategy that efficiently captures the multi-modality of the task distribution via modulating the routing between convolution channels in the network, instead of directly modulating the network weights. This new mechanism can be cast as a permutation learning problem. We further introduce a novel neural permutation layer based on the classical Benes routing network, which has sub-quadratic parameter complexity in the total number of channels, as compared to the quadratic complexity of the state-of-the-art Gumbel-Sinkhorn layer. We demonstrate our approach on various multi-modal meta-learning benchmarks, showing that our framework outperforms previous methods in both generalization accuracy and convergence speed.

Published as conference paper at ICLR 2024
EFFICIENT HETEROGENEOUS META-LEARNING VIA CHANNEL SHUFFLING MODULATION
Minh Hoang
Carl Kingsford
Computer Science Department Carnegie Mellon University Piusburgh, PA 15213, USA andang@andrew cmu, edu
Ray and Stephanie Lane
Computational Biology Department Carnegie Mellon University Pittsburgh, PA 15213, USA carlk cmu, edu
ABSTRACT
We tackle the problem of meta-learning across heterogenous tasks. This prob- lem seeks to extract and generalize transferable meta-knowledge through learning from tasks sampled from a multi-modal task distribution. The extracted meta- knowledge can be used to create predictors for new tasks using small number of labeled samples. Most meta-learning methods assume a homogeneous task distri- bution, thus limiting their generalization capacity when handling multi-modal task distributions. Recent work has shown that the generalization of meta-leerrnng de- pends on the similarity of tasks in the training distribution, and this has led to many clustering approaches that aim to detect homogeneous clusters of tasks. However, these methods suffer from a significant increase parameter complexity. To over- come this weakness, we propose new heterogeneous meta-learning strategy that efficiently captures the multi-modality of the task distribution via modulating the routing between convolution channels in the network, instead of directly modu- lating the network weights. This new mechanism can be cast as permutation learning problem. We further introduce novel neural permutation layer based on the classical Benes routing network, which has sub-quadratic parameter complex- ity in the total number of channels, compared to the quadratic complexity of the sfate-ofthhe-rs Gumbel-Sinkhon layer. We demonstrate our approach on various multi-modal meta-learning benchmarks, showing that framework outperforms previous methods in both generalization accuracy and convergence speed.
INTRODUCTION
Few-shot learning (FSL) is a challenging problem where the goal is to learn new concepts with only small number of labeled samples, similar to how humans learn things by incorporating prior knowledge and context. One promising approach to tackle FSL problems is meta-learning, which learns to extract and generalize transferable meta-knowledge from distribution of tasks and quickly adapt it to unseen tasks. Many meta-learning methods (Yoon et al., 2018; Antoniou et al., 2019; Ra- jeswaran et al., 2019) are built upon the model-agnostic meta learning (MAML) framework (Finn et al., 2017). The MAML method Iearns a single set of model parameters for any arbitrary network architecture, and then fine-tunes it to deliver high performance on unseen tasks. However, most MAML variants assume a homogeneous task distribution in which all tasks originate from the same concept domain, and assume that transferable knowledge globally shared among all tasks (Vuorio et al., 2019). These assumptions constrain the generalization capacity of these meta-learners when handling multi-modal task distributions, for which the task-specific optimal parameters could di- verge significantly from one another. For example, if the task distribution consists of modes that are far apart (c.g., animal and vehicle recognition tasks), it would be impossible to find an initialization that is simultaneously close to all modes.
Recent work has demonstrated that the generalization of MAML and, by extension, its many vari- ants, indeed related to the similarity of tasks in the training distribution (Zhou al., 2021). This perspective aligns with many previous clustering approaches that aim to detect homogeneous clus- ters of tasks which MAML-based learners can be effectively applied (Zhou et al., 2021; Yao al.,

---

Published as conference paper at ICLR 2024
2019: Vuorio et al., 2019). Zhou et al. (2021) seeks to learn ensemble of initializations, each of which is set to represent a cluster of tasks (i.e., a mode in the task distribution). This is achieved via augmenting the MAMI loss function with an assignment step. The cluster assignment heuris- tic, however, is conditioned on the single-mode, vanilla MAML initialization and thus is likely not optimal in a multi-modal setting.
Alternatively, Yao et al. (2019) and Vuorio et al. (2019) propose to implicitly cluster tasks using the embedding vectors of their few-shot data, In particular, Vuorio al. (2019) applies modu- lation network on the learned task embedding to modulate the mcta-initializatiin of the predictor model. yielding the task fask-specific parameters. Yao et al. (2019) adopts similar idea, but further im- poses explicit hierarchical structure on the task space through jointly optimizing several task cluster centroids. The estimated parameter modulation is then applied to the nearest centroid based on their embedding distance. While both methods are capable of addressing the task heterogeneity challenge, they suffer from significant increase in parameter complexity since their respective mod- ulation networks must scale with the size of the predictor model (c.g., for an average convolutional architecture with millions of parameters. the modulation network is essentially a million-output map). Even when the modulation is applied layer-wise, learning to generate that many variables is still a challenging task. This thus prevents applying these tactics on larger architectures.
To overcome this weakness, we aim to develop new heterogeneous meta-learning strategy that efficiently captures the multi-modality of the task distribution via modulating the routing between neurons in the network, instead of directly modulating the network weights. Our approach is par- tially inspired by the ShuffleNet architecture (Zhang et al., 2018), which employs convolutional channel shuffling to encode a highly expressive solution space. The phenomenal success of Shuf- fleNet, which achieves comparable performance to state-of-the-ar models that have many-fold more parameters, suggests that adapting the routing configure oonfiuurtionn (i.e., implied by the channel shuffling order) can potentially emulate the modulation of many neurons without incurring the extra compu- lational costs.
This insight motivates to reformulate the weight modulation network in previous heterogeneous meta-learning approaches (Yao et al., 2019; Vuorio et al., 2019) as a routing modulation network that controls task-specific shuffling of convolution channels. In particular, given task embedding vector, our modulation network learns to generate permutation matrix which simulates the channel shuffling operator when multiplied with the output of convolution layer. To model this permuta- tion network, one can adopt the Gumbel- layer (Mena et al., 2018), which differentiably transforms general square matrices discrete permutation matrices in the limit of temperature pa- rameter. The permutation network can be optimized via learning a mapping f R² >> RC² where and C are respectively the task embedding dimension and the number of convolutional channels.
However, accurately learning dense C matrix given from limited training data can be challeng- ing, especially for large convolutional networks. To overcome this, we propose an even more com- pact formulation of the permutation module based on the classical Beneš routing network (Beneš, 1964), which can emulate any C-permutation using at most C' log, C' binary switches that pairwise permute adjacent indices. Finally, to enable end-to-end learning of this compact permutation net- work, approximate the discrete switches by applying the same Gumbel-softmax transformation in Mena et al. (2018) to x 2 general matrices. The main contributions of this paper are:
1. We develop a more efficient heterogencous meta-lcarning framework that estimates the different modalities in the task distribution via modulating the network routing configu- rations. This modulation operator takes the form of a permutation network that performs channel shuffling based on the few shot training data of task. Our meta routing modula- tion (MRM) framework is presented in Section 3.2.
2. We propose compact formulation of the above permutation network based on a continuous relaxation of the classical Benés network (Beneš, 1964), which we call the Gumbel- -enés layer (Section 3.3). The Gumbel-Benés layer trains efficiently with limited data and scales better the number of convolution channels than previous state-of-the-arr baselines. Our approach is the first to draw a connection between the classical Benés network and archi- tecture modulation for meta learning.
3.
Our framework outperforms existing methods in terms of generalization accuracy and run- time on various multi-modal meta-learning benchmarks (Section 4).
2

---

Published as conference paper at ICLR 2024
2 RELATED WORK
Meta-Icarning. Existing meta-learning approaches can be broadly classified into three families: metric-based, model-based, and optimization-aase methods. Model-based approaches (Sukhbaatar et al., 2015; Graves et al., 2014) aim to recognize the task identity from its few-shot data and use the task identity to adjust the model state accordingly. While these methods perform well on certain task domains, they require fixing the model architecture and thus are difficult apply arbitrary use cases. Metric-based methods (Snell et al., 2017; Vinyals et al., 2016) learn a task similarity metric (based on observed data) which can be used to perform inference on new tasks Sun et al. (2021) and Patacchiola et al. (2020) respectively offer a Bayesian view on metric-based and model- based meta- learning. Optimization-based methods (Finn al., 2017: Yoon al., 2018; Antoniou et al., 2019; Rajeswaran et al., 2019) learn a single model initialization that is amenable to fast adaption and can be applied to any arbitrary architecture. However, most existing metric-based and optimization- based methods assume that single metric model or parameter initialization is sufficient to capture the entire task distribution.
Heterogeneous meta-learning. Heterogeneous meta-learning (HML) is an emerging area that de- velops meta-learning techniques that can generalize well to tasks drawn from a multi-modal distri- bution. The majority of existing HML approaches account for task heterogeneity via one of two approaches. Yao et al. (2019) and Zhou et al. (2021) explicitly maintain several local meta initial- izations (i.e., task clusters), to which observed tasks are assigned during training. The effectiveness of these methods depends on the quality of the many heuristics employed, such as the number of clusters and the distance metric used for cluster assignment. On the other hand, Yao et al. (2019); Vuorio et al. (2019); Liu et al. (2021); Triantafillou et al. (2021) adopt a modulation strategy that modify some components of a global meta initialization depending on some learned task embed- ding vector. While these approaches do not require expert understanding of the task distribution, the methods proposed by Yao al. (2019): Vuorio al. (2019); Liu et al. (2021) are expensive to learn, especially with large architectures, as the modulation vector scales with the number of parameters. Li et al. (2022) and Triantafillou et al. (2021) work around this scalability issue by localizing the modulation to several adapter components within the model architecture. This heuristic, however, does not modulate the remaining weights of the network, and thus assumes that the global model can adapt solely with these components. We instead routing modulation model that modulate the entire network at significantly more inexpensive cost.
Routing neural networks. Routing neural networks or neural routing refers to technique in neural network architecture where information is selectively passed between groups of neurons based on some learned decision rules. This can be accomplished through the of routing algorithms or specialized routing layers in a neural network. The most common form of routing is by pruning computational paths (e.g., setting certain weights to zero), which is typically used to induce spar- sity (Shazeer et ul., 2017) in the network for computational efficiency, or to prevent catastrophic forgetting in continual learning scenarios (Collier et al., 2020). Random channel shuffling was introduced by Zhang et al. (2018) in the context of designing compact architectures to improve model expressiveness. The ShuffleNet architecture was subsequently extended to explicitly learn the shuffling order (Lyu et al., 2020) (i.e., via optimizing for the permutation matrices that control the shuffling). Freivalds et al. (2019) proposed another differentiable neural routing formulation via continuous approximation of the classical Beñes routing network Beneš (1964). However this ap- proach only mimics the discrete shuffling of Beñcs network in spirit, offering no guarantee that the post-shuffling information will be preserve, and thus is not suitable for channel shuffling modula- tion. To the best of our knowledge, neural routing for meta learning has only been considered by the work of Cai al. (2022) in the form of heuristic pruning. In this paper, we introduce differentiable reparameterization of the Beñes network that almost precisely models permutations, and explicitly learns to modulate channel shuffling given observed task data.
3 METHOD
3.1 PRELIMINARIES
In the meta-learning setting, we are given a task distribution T, where each task T3 ~ T consists of a dataset D. and learning objective L,. Similar many other meta-learning studies, we adopt
3

---

Published as conference paper at ICLR 2024
TASK
MErABATcchizz:
CONV
CONV
PFW
ROUTER
ROUTER
PROTO
The overview of our meta-routing approach (MRM). Task data is first embedded & (D,)). (D,) is then used to compute prototypical network loss (Snell et al., 2017) and generate channel routing matrices interleaving convolutional layers of the predictor network. Our loss function com- bines the MAML loss (over a meta-task batch) and the prototypical network loss (per task).
Figure 1:
simplified setting where all tasks T share the same learning objective L and each dataset D; (xij, Yij }" contains n-shot supervised learning samples. The goal of meta-learning is to train a meta-model M. that maps any task T; a parameter vector 0, in the weight space of some predictor model G, such that M. minimizes the expected learning loss (over random lasks):
1
n Σ
a M(Ti)). (
M. argmin ETST n j=1
argmin ENTT Cct (M(T))
(1)
where (8, D) denotes the averaged objective value evaluated model architecture G with pa- rameters over all data points in D. Towards this goal, the MAML framework (Finn et al., 2017) fine-tuning gradient descent step with respect to D; given some base initializa-
models M. (Tt) tion 0.. That is, (T,) A_ v f (0., D:), where 7 denotes the step size. To obtain the base initialization 0., Finn et al. (2017) proposes to optimize the following loss function:
0. = argmin E F (o viVol (O,D)) D
(2)
where {D( > denotes the train- validaaaio split of D, Intuitively, the goal of this loss function to find single initialization 0, such that, given the fine-tuning step at the time of evaluating M. (T:), the adapted parameters will yield the best performance in expectation.
3.2 HETEROGENEOUS META LEARNING VIA ROUTING MODULATION
Motivated by our discussion above and previous works that established that learning a single initial- ization 0, is sub-optimal when the task distribution T is multi-modal, we now introduce our hetero- geneous meta-learning approach (Fig. 1). To account for task-heterogeneity,, Yao et al. (2019) and Vuorio et al. (2019) apply task-specific modulation of the base parameter 0. follows:
MT,
mo(0., D.) v/ C D.), D:),
where mo(0., D,) abstracts the modulation operator that takes the form mo(6., D,) = (D,) in both Yao al. (2019) and Vuorio et al. (2019), C denotes the point-wise multiplication operator, and vi denotes some arbitrary embedding protocol that maps a lask datasct to the weight space of the predictor G. For example, Vuorio al. (2019) models & as an attention mechanism, whereas Yao et al. (2019) pre-trains ProtoNet task embedding (Snell et al., 2017) for task clustering and applies fully connected network to generate (D())) from the cluster centroids. Both methods, however, suffer from high additional complexity since the output dimension of ve is the prohibitively large number of parameters G. To work around this shortcoming, we instead apply the task-specific modulation tactic on the architecture routing level:
M. 0. Vo (.. (0., D.)
4

---

Published as conference paper at ICLR 2024
Figure 2: Benés network for C Input channels are grouped pairwise. Each square block denotes learnable binary switch that either maintains or permutes the ordering of its input pair. Routing between layers of switches are performed in deterministic manner, where one input is routed the top half and the other to the bottom half of the inner subnetwork (Beneš, 1964).
To concretely describe modulation operator, will first re-imagine the architecture G sequence of neural layers (G1, G2, Gx ), such that for any arbitrary input x, we can rewrite G(x) GM GM-1 where C, is the number of feature channels and d; is the (flattened) feature dimension of each channel. Then, our modulation operator can be succinctly applied through interleaving a sequence of routing layers {Ri, Re, RM-11 } in between the predictor layers of G; that is:
G1 (x). We assume that the output of layer Gj has dimension C; x di,
mo( = GM RM-1 GM-10 R1 G1,
(5)
where each routing layer takes the form of (Q; & Py(D:)Q for some intermediate feature Q E xd, such that P3 maps D; to Cj C2 permutation matrix. Intuitively, the goal of P3 is to re-route the information flow between layer Gj and Gy+1 of the predictor net in response to the current task. is thus appropriate for P, to generate permutation matrix, such that the information channels are shuffled without degrading their signals.
To construct such map, we first compute the ProtoNet embedding (Snell et al., 2017) of D, and apply convolutional layer which subsequently transforms this embedding into general square matrix in RC, To approximate the discrete permutation constraint, we could directly apply Gumbel-Sinkhorn layer (Mena et al., 2018), whose output is guaranteed converge to permutation matrix in the limit its temperature parameter. However, the Gumbel-Sinkhorn layer does not scale well with the total number of channels in G, and we will show in the next section that the permutation constraint can be upproximated more compactly using a novel layer that we call Gumbel-Benös. For convenience,
further let denote the combined weights of {PI, P2.... ... PM-1), which fully specifies our modulation network. The base predictor parameters 0, and the modulation parameters y, can be jointly optimized by extending the MAML loss function (Finn et al., 2017):
(0.,T.) argmin ET,
ct (
(0-n VeCt Vo (!) D:) + AL ACProso(D.; (6)
where Crroto denotes the PROTONET loss (Snell et al., 2017) and À is trade-off hyper-parameter.
3.3 GUMBEL-BENES ROUTING LAYER
The Gumbel--Snkhorr layer is a differentiable transformation that approximately produces permu- tation matrices. However, order to generate sparse permutation matrix of size C' C, it is necessary that the Gumbel-Sinkhorn layer also receives as input dense C x C matrix. Due to this requirement, each routing layer would require at least a quadratic number of parameters in terms of Cj (c.g., for the convolutional map) to generate an input to the Humbel-SSnkhhor layer. Over- MM all, the addition of the entire modulation component would yield an extra O' C3) learnable parameters. Although this additional complexity would be smaller than the total number of param- eters in G in most cases, it would become very expensive for larger architectures. To overcome this challenge, adopt classical result from Beneš (1964), which shows that any permutation of C
5

---

Published as conference paper at ICLR 2024
channels can be emulated by exactly 2 log2 C' layers of C/2 binary switches (i.e., 2 x 2 permuta- tion matrix). Within each layer, we divide the input channels into groups of two and permute their features pairwise via multiplying with the content of the binary switches. The output of one layer is then forwarded to the next via a deterministic exchange step (BeneK, 1964) (see Appendix D for detailed description). An example routing configuration for channels is shown in Fig. 2. was shown that the Benës network has congestion of 1 (Beneš, 1964), meaning there are no two differ- ent permutations that share the same switch configurations. As a result, it is sufficient to compactly model channel routing with just C3 log2 C2 parameters at any layer Rg-
Finally, we can redefine our routing layer R, using the Benés network. This is achieved by first re- formulating the convolutional map to produce a stacked tensor of continuous switch configurations. We then apply the Gumbel-Sinkhorr transformation on these continuous configurations to approx- imate the discrete binary switches. Finally, we perform the shuffle-exchange steps recursively to permute the input channels. Formally, we describe below the computational pathway of the routing layer Ry given the initial state and input So & Q and task data D;:
U3 (z())
1. 1] '1 [
S S ((
]]
GS(U, KJ)
vk E (1,C,/2),
S/[2k]
S exchange (5i) (5:),
where & denotes the PROTONET embedding, f, maps task embedding to its corresponding contimu- ous switch configurations, GS(U, k)) denotes the Gumbel-Sinkhorn transform of the component of Uj corresponding to the 1thh switch of (th layer. The exchange step refers to the deterministic routing of the Benès network. The entire routing system of any Benés network for 2 inputs can be efficiently constructed in a recursive manner (see Appendix D). Each S, denotes the output of the 7th Benés layer, and S2 logg e is the output of the routing layer Rj.
4 EXPERIMENTS
We compare the performance of our method using the Gumbel-Sinkhorn permutation layer (Mena et al., 2018) (MRM-GS) and our proposed Gumbel-Bene& routing layer (MRM-GB) against sev- eral meta-learning baselines, including MAML (Finn et al., 2017), its first-order approximation FO-MAML (Antoniou et al., 2019), prototypical network (PROTONET) (Snell et al., 2017), multi- modal aggnsssc meta learning (MMAML) ( (Vuorio et al., 2019), universal representation learning (URL) (Li et al., 2022), and hierarchically structured meta learning (HSML) (Yao et al., 2019). We adapt the MAML and PROTONET implementations from the LEARN2LEARN pack- age (Amold al., 2020). Experiments are conducted Our implementation is available at htt hst / //gttttt/ hub.......... com www/ooioineeecoeccc Kings ford ford- /mrggb.
GTX-3080 GPU with 13GB memory.
For MRM-GS, MRM-GB, MMAML, HSML, parameterize the task embedding network and the predictor network using a CNN architecture with 4 hidden convolutional blocks and a feed- forward classification layer. Each block consists of 3 x 3 convolution layer, followed by BATCH- NORM, MAXPOOL and RELU activations. All convolution layers have C' = 32 or 64 hidden chan- nels, depending on the specific task distribution. The mapping from task embedding to modulation parameters is parameterized by a 1-layer, TANH-activated feed-forward neural network, whose out- put dimension depends on the method (e-g., approximately C2 for MRM-GS, C' Coo,, C' for MRM- GB and 9C2 for MMAML). We apply the modulation to the first hidden convolutional layer.
MAML, FO-MAML and URL have no embedding network. For fair comparison against the above methods, we parameterize the predictor network by a two-headed CNN architecture with 4 hidden convolutional blocks per head. Their oulputs are then concatenated and forwarded to the classifica- tion layer for prediction. For URL, used the channelwise adapters suggested by Li et al. (2022) after each convolutional block as it is the most similar strategy to our channel routing layer. Last, PROTONET has no predictor network and performs prediction via clustering the input embeddings. For the same reason as above, we parameterize its embedding network by a similar two-headed CNN architectures (no classification layer).
Meta-learning vision baselines. The OMNIGLOT dataset (Lake et al., 2015) consists of 1623 hand- writlen characters from 50 different alphabets and writing systems. We randomly split the dataset
6

---

Published as conference paper at ICLR 2024
by class into train (1100 classes), validation (100 classes), and test sets (423 classes), suggested by Ravi & Larochelle (2017). The MINI-MMAGENET dataset (Vinyals et al., 2016) is a subset of the larger ImageNet dataset (Russakovsky et al., 2015) that contains 60000 images from 100 object categories. We randomly split the dataset by category into train (64 categories), validation (16 cat- and (IGSAW-MINI-LMAGENN
egories), and test sets (20 categories). The JIGSAW-OMNIGL datasets are obtained by segmenting the training imgees in the respective original datascts into 2 tiles and randomly permuting these tiles to simulate 24 different task modalities. Finally, dataset combines: (a) The VGGFLOWER 102 dataset (Triantafil- lou et al., 2020) consisting of 102 classes of flowers (between 40 to 258 images per class); (b) the FGVCAIRCRATI AFT dataset (Maji et al., 2013; Triantafillou et al., 2020) consisting of 102 classes of aircraft (100 images per class); and (c) the FGVCFUNGI dataset (Triantafillou et al., 2020) consist- ing of 1394 classes of fungi, with a total of 89760 images.
The
Methods
OMNIGLOT
MINI-
JIGSAW- OMNIGLOT
JIGSAW-MINI- IMAGENET
IMAGENET
MAML
0.977 + 0.028
0.616 + 0.059
0.944 + 0.044
0.571 + 0.095
FO-MAML
0.960 + 0.020
0.552
0.921 + 0.068
0.548 + 0.054
PROTONET
0.933 0.062
0.005
0.860 0.054
MMAML
0.976 0.015
0.146
0.940 0.077
0.587
HSML
0.969 + 0.101
= 0.942 0.128
0.572 0.096
URL
0.971 0.048
0.599
0.119
0.929 = 0.020
0.569 + 0.040
MRM-GS
0.971 1 0.006
0.613 0.151
0.949 1 0.036
0.601 1 0.121
MRM-GB
0.981 + 0.011
0.614 + 0.005
0.950 + 0.033
0.603 + 0.069
Table 1: Average test accuracy of various bascline methods OMNIGLOT, MINI-IMAGENET datasets and their JIGSAW variants. Standard deviations are obtained over 5 random test task batches.
4.1 META-LEARNING FOR UNIMODAL TASK DISTRIBUTION
We show that our method performs robustly on the traditional homogeneous meta-learning setting despite the multi-modal treatment. We train all baseline methods on random batches of tasks drawn from (a) the OMNIGLOT dataset; and (b) the MINI-IMAGENET dataset. All tasks consist of ran- domly drawn images from 5 distinct labels. For each label, the task dataset contains n, support and ng query images. For training. both the support and query images are used to train the meta-learners. For testing, we perform fast adaptation using the support image and measure the test accuracy on the query images. We sample batch of 32 training tasks per epoch to train each baseline method, and then evaluate their averaged performances over 5 random test tasks. We respectively let n, 1, ny = 15 and n. = 5, ny = 5 for the OMNIGLOT and MINI-IMAGENET experiments,
Table records the average test accuracy (over 5 test tasks) for cach bascline method over 2000 training epochs. We defer the plots of training loss and test accuracy versus training epochs to the Appendix. Vanilla MAML achieves the best accuracy of 0.626 + 0.059 the MINI-IMAGENET dataset, and second best accuracy of 0.977 + 0.028 on the OMNIGLOT dataset. These results are expected since their respective task distributions are unimodal. While other multimodal approaches (e.g., MMAML (Vuorio et al., 2019), HSML (Yao et al., 2019), and URL (Li et al., 2022)) tend to underperform in this standard setting, our methods are significantly more competitive. Specifically, on the OMNIGLOT datasct, MRM-GB achieves the best performance of 0.981 + 0.011. On the MINI-IMAGENET dataset, MRM-GS GS and MRM-GB achieve the third and second best classifica- tion accuracies of 0.613 1 0.151 and 0.615 1 0.005 respectively.
4.2 META-LEARNING FOR MULTI-MODAL TASK DISTRIBUTION
We further conduct experiments to demonstrate the performance of our method in two different settings of task heterogeneity. In the first experiment, simulate the multi-modality of the task distribution by applying a jigsaw transformation to the training images in the OMNIGLOT and MINI- IMAGENET datasets. Specifically, each training/test image first segmented into 2 x 2 smaller tiles. For each sampled task, we then randomly draw a permutation of these 4 tiles and shuffle them accordingly to systematically derive new tasks that belong 4! 24 different modalities.
7

---

Published as conference paper at ICLR 2024
Table 1 records the average test accuracy (over 5 test tasks) for each baseline method on the JIGSAW- OMNIGLOT and JIGSAW-MINT- MAA--MIII-IMMGGEEE datasets, respectively over 3000 and 5000 training epochs. In this setting, the unimodal approaches (e.g., MAML, FO-MAMI, and PROTONET) generally perform worse than the multimodal approaches. This observation confirms the need for multi-modal meta learning. We further observe that PROTONET consistently demonstrates the weak- est performance on these task distributions. This is most likely because PROTONET tries to assign similar embeddings to images of the same label, which include different jigsaw permutations of the same image. While our approaches also make use of the PROTONET loss to embed tasks, the speci- ficity of the shuffling will be captured by meta-training both the predictor network and the modulator network using the MAML loss. As result, our methods consistently achieve the best (MRM-GB) and second best MMMMGGS) classification accuracies in both datasets.
Methods
INJECT FLOWERS
INJECT AIRCRAFT
INJECT FUNGI
INJECT ALL
MAML
0.611 + 0.015
0.522 + 0.069
0.469 = 0.077
0.491 + 0.115
MMAML
0.609 + 0.045
0.568 1 0.059
0.501 = 0.072
0.527 + 0.049
HSML
0.615 + 0.007
0.540 + 0.101
0.512 = 0.128
0.526 + 0.091
URL
0.598 + 0.008
0.555 + 0.053
0.509 = 0.117
0.512 + 0.041
MRM-GS
0.608 0.006
0.573 + 0.051
0.522 = 0.066
0.529 + 0.045
MRM-GB
0.612 0.011
0.564 + 0.065
0.529 = 0.094
0.537 + 0.063
Table Average test accuracy of various baseline methods on the multi-modal FLOWER- AIRCRAFT-FUNGE dataset. We present the converged performance after injecting each modality into the task stream (at epoch 1230, 2460 and 3690 respectively), and the converged performance when all modalities are injected into the task stream at the beginning. Standard deviations are ob- tained over random test task batches.
As the JIGSAW setting is naturally suited for channel shuffling approach, we further simulate a more realistic multi-modal task distribution via grafting three diffcrent image datasets: (a) VG- GFLOWERS 102, (b) FGVCAIRCRAFT and (c) FGVCFUNGI (Triantafillou et al., 2020). The com- bined FLOWERS-AIRCRAFT
dataset thus has three distinct task modalities. We initialize the task distribution with only data from (a) (epoch 0), and subsequently inject data from (b) af- ter 16000 sampled train/test tasks (epoch 1230); and from (c) after 32000 sampled train/test tasks (epoch 2460s). We use a batch size of 8 tasks per epoch instead of 32 as in the above experiments.
Table 2 records the average test accuracy (over 5 test tasks) for MAML and all multi-modal meta- learning baselines. The INJECT FLOWERS, INJECT AIRCRAFT and INJECT FUNGI columns respec- tively report the classification performances after 1230 training epochs starting from each injection point. The INJECT ALL column reports the average test accuracy (at epoch 3690) of the meta-learner when all three modalities introduced from the start. Additionally, Fig. 3(a) and Fig. 3(b) plot the training loss vs. epoch for each baseline on these two scenarios.
As expected, each task injection causes degradation in terms of average test accuracy due to the introduction of new modality in the distribution. This is also shown most clearly in Fig. 3(a), in which there is spike in training loss following the injection of a new dataset into the task stream (i.c., marked by the vertical dotted lines). This confirms that the meta initialization learned on one modality cannot be easily adapted address tasks from another modality, and thereby further confirms the need to address task-heterogeneity in meta-learning. Out of all baselines, MAML experiences the worst degradation, in which the classification performance at epoch 3690 24% worse than that at epoch 1 1230. Our methods MRM-GS and MRM-GB perform competitively in the unimodal setting (up to epoch 1230), and outperform other basclines after new task modalities have been injected into the task stream. We also expect that the simultancous injection scenario is slightly easier than sequential injection. This is because the meta-learner will not be skewed towards solving the earlier introduced modalities. Indeed, we observe better performances and training losses in all baselines compared to that of the sequential injection scenario. In this setting, our method, MRM-GB still achieves the best accuracy of 0.537 + 0.063.
Last, in both scenarios (i.e., jigsaw and datasel grafting), we show that our meta-routing method MRM GB interestingly achieves an implicit clustering of tasks into the right modality. This is
8

---

Published as conference paper at ICLR 2024
demonstrated via the respective t-SNE plots of the task embeddings prior their discrete transfor- mation into permutation matrices (see Appendix E).
1.6
1.6
URL
HSML
MRM-GS
URL HSML
MRM-GB MAML
Training
Training
1.0
1.0
1000
2000
3000
4000
1000
2000
3000
4000
Epochs
Epochs
Figure 3: Training loss epochs for various baseline methods the multi-modal FLOWERS- AIRCRAFT-FUNGI dataset. We demonstrate on two settings: (a) each component dataset is intro- duced after 1230 training epochs in listed order (the injection points are marked by the dotted vertical lines); and (b) all component datasets are simultaneously introduced from the beginning.
4.3 ABLATION STUDIES
To better understand our methods, conducted several ablation studies and defer the result the Appendix of this paper. In particular, Appendix B shows the 2-dimensional t-SNE plot of 10000 MRN-GB learned task embeddings for two meta-learning scenarios. The t-SNE embeddings form distinct clusters separated by task modalities, and thus demonstrate that our routing method is ca- pable of identifying related task groups. Appendix further shows that our methods are more parameter-effiieen than existing methods, especially with larger number of channels in 4-layer convolutional architecture. Appendix shows that our method performs robustly with the ResNet- 18 architecture. Finally, we conducted ablation experiments to investigate the importance of varying the parameter and the importance of routing layer placement. Our results are respectively shown in Appendix H and Appendix I.
5 CONCLUSION
Previous meta-learning methods assume a homogeneous task distribution, which limits their gener- alization ability when dealing with multi-modal task distributions. Recent works have attempted to rectify this problem, but suffer from increased complexity in terms of parameters. To overcome this limitation, we propose novel strategy for heterogeneous meta-learning. Our approach efficiently captures the multi modai-mo of the task distribution by modulating the routing between convolution channels in the network. This mechanism can be viewed as a permutation learning problem, which we model and solve using compact neural permutation layer based on the classical Benes routing network. Our Gumbel-Benn layer exhibits sub-quadratic parameter complexity in the total number of channels, in contrast to the quadratic complexity of -fate-ff-the-art Gumbel-Sinkhorn layers. We validate our approach on various multi-modal meta-learning benchmarks, demonstrating superior performance compared to previous methods in terms of both generalization accuracy and runtime. Theoretical understanding of our approach will a future research consideration.
Societal Impact. While applications of our work to real data could result in ethical considerations, this is an indirect, unpredictable side-effect of our work. Our experiment uses publicly available datasets to evaluate the performance of our algorithms; ethical considerations raised.
Ackmowledgements This work was supported in part by the US National Science Foundation [DBI- 1937540. III-2232121]. the US National Institutes of Health (R01HG012470 and by the generosity of Eric and Wendy Schmidt by recommendation of the Schmidt Futures program. Conflict of Inter- est: C.K. is co-founder of Ocean Genomies, Inc.

---

Published as conference paper at ICLR 2024
REFERENCES
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your MAML. International Conference on Learning Representations, 2019.
Sébastien M R Amold, Praateck Mahajan, Debajyoti Datta, Ian Bunner, and Konstantinos Saitas Zarkias. Icarn2lcarn: A library for Meta-Lcarning research, August 2020. URL http: / //aaxxi..rgg/aaaaaeeee//aa hbtp:/bs/22e 2008. 12284.
Václav E Beneš. Permutation groups, complexes, and rearrangeable connecting networks. Bell System Technical Journal, 43(4):1619-164)) 1964.
Jicang Cai, Saced Vahidian, Weijia Wang, Mohsen Joncidi, and Bill Lin. Neural Routing in Meta
Learning. Workshop on Meta-Learning Neural Information Processing Systems, 2022.
Mark Collier, Ef Kokiopoulou, Andrea Gesmundo, and Jesse Berent. Routing networks with co- training for continual learning. Workshop on Continual Learning at the International Conference on Machine Learning, 2020.
C. Finn, P. Abbeel, and Levine. Model-agnostic meta-learning for fast adaptation of deep net- works. In Proceedings of the International Conference Machine Learning, 2017.
Karlis Freivalds, Emils Ozoligš, and Agris Šostaks. Neural shuffle-exchange networks-sequenn processing On log n) time. Advances in Neural Information Processing Systems, 32, 2019.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural Turing machines. 0rXiv:1410.5400 2014.
arXiv preprint
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametrization with Gumble-Softmax In International Conference on Learning Representations, 2017.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266): 1332-1338, 1502-366)-1332-1333, 2015.
Wei-Hong Li, Xialei Liu, and Hakan Bilen. Cross-domain few-shot learning with task- speciic adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 7161-7170, 2022.
Yanbin Liu, Juho Lee, Linchao Zhu, Ling Chen, Humphrey Shi, and Yi Yang. A multi-mode mod- ulator for multi-domain few-shot classification. In Proceedings of the IEEE/CVE International Conference Computer Vision, pp. 8453-8462, 2021.
Jiancheng Lyu, Shuai Zhang, Yingyong Qi, and Jack Xin. AutoShufflcNet: Learning permutation matrices via exact Lipschitz continuous penalty in deep convolutional neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 608-616, 2020.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.3 5151, 2013.
G. Mena, D. Belanger, S. Linderman, and J. Snoek. Learning latent permutations with Gumbel- Sinkhorn networks. Proc. ICLR, 2018.
Massimiliano Patacchiola, Jack Turner, Elliot J Crowley, Michael O' O'Boyle, and Amos J Storkey. Bayesian meta-lcarning for the few-shot setting via deep kernels. Advances Neural Information 2020.
Processing Systems,
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-leamning with im- plicit gradients. Advances in Neural Information Processing Systems, 32, 2019.
Sachin Ravi and Hugo I Larochelle. Optimization as a model for few-shot learning. In International Conference on Learning Representations, 2017.
10

---

Published as conference paper at ICLR 2024
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3):211-252 2015.
Noam Shazeer, Azalia Mirhoseini, Krzysziof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-expent layer. In International Conference on Learning Representations, 2017.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Ad- vances in Neural Information Processing Systems, 30, 2017.
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. In Advances in Neural Information Processing Systems, pp. 2440-2448, 2015.
Zhuo Sun, Jijie Wu, Xiaoxu Li, Wenming Yang, and Jing-Hao Xue. Amortized bayesian prototype meta-learning: A new probabilistic meta-learning approach to few-shot image classification. In International Conference on Artificial Intelligence and Statistics, pp. 1414-1422. PMLR, 2021.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta- dataset: dataset of datasets for learning to learn from few examples. In International Conference Learning Representations, 2020.
Eleni Triantafillou, Hugo Larochelle, Richard Zemel, and Vincent Dumoulin. Learning universal template for few-shot dataset generalization. In International Conference on Machine Learning, pp. 10424 10433. PMLR, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using I-SNE, Journal of Machine Learning Research, 9111), 2008.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in Neural Information Processing Systems, 29, 2016.
Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J Lim. Multimodal model-agnostic meta- learning via task-aware modulation. Advances in Neural Information Processing Systems, 32, 2019.
Huaxiu Yao, Ying Wei, Junzhou Huang, and Zhenhui Li. Hierarchically structured meta-learning. In International Conference on Machine Learning, pp. 7045-7054. PMLR, 2019.
Yoon, T. Kim, O. Dia, S. Kim, Y. Bengio, and S. Ahn. Bayesian model-agnostic meta-learning. In Proc. NeurIPS, 2018.
Xiangyu Zhang, Xinyu Zhou. Mengxiao Lin, and Jian Sun. ShuffleNet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6848 6856, 2018.
Zhou, Y. Zou, X. Yuan J. Feng, C. Xiong, and S. Hoi. Task similarity aware meta learning: Theory-inspired improvement on MAML. In Uncertainty in Artificial Intelligence, pp. 23-33. PMLR, 2021.
11


Reviews:
[{"review_id": "7u0JYrSaVL", "reviewer": "Reviewer_ppvH", "summary": "This paper deals with the heterogeneous meta-learning problem from the perspective of channel shuffling modulation.   \nInstead of the conventional fixed backbone + modulation layer scheme for multiple task distributions, this paper seeks to find a task-specific channel permutation routing mechanism.   \n\nThe idea is motivated by ShuffleNet and implemented with a task-specific prototypical vector to learn the shuffling operation. For the permutation matrix, this paper first adopts the Gumbel-Sinkhorn layer to generate the permutation. Then, the classical Benesˇ routing network is utilized to improve the efficiency. \n\nExperiments are conducted on several heterogeneous datasets and compared with various MAML-based methods. As a parameter-efficient method, the proposed method also performed generally better than baselines.", "questions": "Please see above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- (1) This paper provides a different perspective for heterogeneous meta-learning, i.e., channel shuffling by permutation learning. It can be a good alternative to the modulation-based methods. \n- (2) The utilization of the traditional Beneˇs network can improve the efficiency of the proposed method. \n- (3) Experiments are conducted on the meta-learning benchmark, and the results are fair and generally better than other MAML-variant methods.", "weaknesses": "- (1) The authors describe the heterogeneous meta-learning problem from the MAML perspective. However, similar to heterogeneous meta-learning, researchers also used the terminology multi-domain/cross-domain to describe the diverse task distribution when dealing with few-shot meta-learning problems, e.g., [R1]. Besides Li et al. (2022), other related works applying feature modulation can also be discussed, such as [R2, R3]. \n- (2) I am wondering why permutation and channel shuffling work compared with modulation. The intuition of modulation assumes a strong shared backbone, and each dataset/distribution performs task-specific adjustments for its data. For the permutation case, it is better to visualize what permutation can be learned for different distributions. \n\n\n[R1] Triantafillou, E., Zhu, T., Dumoulin, V., Lamblin, P., Evci, U., Xu, K., Goroshin, R., Gelada, C., Swersky, K., Manzagol, P.A. and Larochelle, H., 2019. Meta-dataset: A dataset of datasets for learning to learn from a few examples. arXiv preprint arXiv:1903.03096.\n\n[R2] Triantafillou, E., Larochelle, H., Zemel, R., & Dumoulin, V. (2021, July). Learning a universal template for few-shot dataset generalization. In International Conference on Machine Learning (pp. 10424-10433). PMLR.\n\n[R3] Liu, Y., Lee, J., Zhu, L., Chen, L., Shi, H., & Yang, Y. (2021). A multi-mode modulator for multi-domain few-shot classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 8453-8462).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lxf7sMFduW", "reviewer": "Reviewer_aRVA", "summary": "**Edit: I have raised my score to 6 to reflect the author's updates to the submission.**\n\n\nThe paper considers the problem of meta-learning across heterogeneous tasks when the model is expected to extract and generalize the meta-knowledge and transfer it to quickly learn the novel tasks. Whereas, many meta-learning models focus on the tasks coming from the same distribution (homogeneous), the authors consider the directly the tasks coming from a multi-modal task distribution (heterogeneous setting). Recently, many appeared many methods that tackle the heterogeneous meta-learning, but they suffer from a significant increase in parameter complexity. As an alternative approach, the authors propose a novel strategy which incorporates modulating the routing between convolution channels in the network. The paper introduce a novel neural neural permutation layer based on the classical Benes routing network. Finally, the proposed method is compared against various meta-learning benchmarks.", "questions": "I would like to see especially the following experiments and improvements:\n1. Provide results for the 1-shot 5-way classification setting on Mini-Imagenet.\n2. Please, compare with state-of-the-art methods as suggested in Weaknesses section. \n3. The work will be better if you could test the Gumbel-Benes layers in ResNets also.\n4. Add the Related Work section regarding other deep learning works utilizing the Benes networks/routing algorithm.\n\n**Questions:**\n\n1. Another place where the Gumbel-Benes routing layers might be beneficial are standard methods working on the Meta-Dataset [1], which is a created really heterogeneous tasks dataset. The best methods working on Meta-Dataset are often ResNets with some adapting layers within, so they are really within the scope of the Gumbel-Benes layers. For the example methods, please see the rank tables in [2].\n\n\n**References:**\n\n[1] Triantafillou, E., Zhu, T., Dumoulin, V., Lamblin, P., Evci, U., Xu, K., ... & Larochelle, H. (2019). Meta-dataset: A dataset of datasets for learning to learn from few examples. arXiv preprint arXiv:1903.03096.\n\n[2] https://github.com/google-research/meta-dataset", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper has a few significant strengths overall, which I will outline below:\n1. Considering directly the heterogeneous tasks setting which I found especially significant within the Meta-Learning.\n2. The main idea of introducing the Gumbel-Benes routing layer is very interesting and seems to be novel in Meta-Learning.\n3. Leveraging the complexity of permutation layer inspired by the Benes network is fair.\n4. I like experiment presented in Figure 3.\n5. The presentation is clear. Overall, the flow of the manuscript is well-organized.", "weaknesses": "However, despite the strengths, the paper has a few major and minor weaknesses. I will focus on the experiments section especially, because I recognize it as insufficient: \n\n1. The results of all methods, presented in the Table 1 and Table 2, are usually within their standard deviations. Because of that, I will not support the claim that the presented method is significantly better than others.\n2. The methods that are chosen for the comparison are not a current state-of-the-art methods, having a few years. I strongly encourage authors to include comparison with other methods like [1] or [2].\n3. The comparison are considered only across a single neural network architecture. I admit that the scenarios of using the Gumbel-Benes routing layers are limited into CNN architectures. However, there is not the reason to skip the comparison on highly popular in Meta-Learning, ResNet architectures. I will suggest for sure comparison on at least ResNet-10 (as in, e.g., [3]).\n4. The lack of more challenging 1-shot 5-way classification on Mini-Imagenet dataset. \n5. As far as I know, there is the first example of the Benes networks-inspired Meta-Learning approach. However, there were others works based on the Benes routing, e.g, [4]. I would like to see the Related Work section given credits to that works and comparing the proposed method with them.\n\n**References:**\n\n[1] Sun, Z., Wu, J., Li, X., Yang, W., and Xue, J.-H. Amortized bayesian prototype meta-learning: A new probabilistic meta-learning approach to few-shot image classification. In International Conference on Artificial Intelligence and Statistics, pp. 1414–1422. PMLR, 2021.\n\n[2] Ye, H.-J., Hu, H., Zhan, D.-C., Sha, F.: Few-shot learning via embedding adaptation with set-to-set functions. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8808–8817 (2020)\n\n[3] Patacchiola, M., Turner, J., Crowley, E. J., O’Boyle, M., and Storkey, A. J. Bayesian meta-learning for the few-shot setting via deep kernels. Advances in Neural Information Processing Systems, 33, 2020.\n\n[4] Freivalds, K., Ozoliņš, E., & Šostaks, A. (2019). Neural shuffle-exchange networks-sequence processing in o (n log n) time. Advances in Neural Information Processing Systems, 32.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OTR9AwGMXi", "reviewer": "Reviewer_d54Q", "summary": "This paper introduces a new meta-learning approach for heterogeneous sets of tasks. The method is presented as a permutation learning problem in which convolution channels are shuffled via a continuous switch network. This trainable routing network is parameterized by input embeddings learned via the prototypical loss (Snell et al., 2017), in which the centroids encode the prototype shuffling parameters corresponding to each task. The method is favorable compared to using Gumbell-Sinkhorn layers (Mena et al., 2018) which use routing embedding dimensions quadratic in the number of channels, hence requiring larger networks with more parameters and longer inference times. The proposed method is also empirically compared to a few other baseline algorithms that do not account for heterogeneity, matching SOTA performance in single task regime and outperforming alternatives in the heterogeneous task setup.", "questions": "Even though the method reduces the nominal parameter complexity of the required routing network, it might still be over parametrizing the space by a log factor. Specifically, in order to define an ordering of $C$ channels one should require no more than $C$ values (each with at least $\\log_2C$ bits). I suspect that one can derive a (continuous) shuffling using embeddings of size $C$ instead of $C \\log C$ by using a sorting network (e.g. Bitonic sort) in a similar fashion as the Benes network is used in this work, but with a different switch type: $(U_j, channel_j)$ pairs are sorted by the key values and deltas between key values (passed a sigmoid) can form continuous switches. Can such a mechanism work in the use cases presented by the paper, and how would it compare to using the MRM-GB in terms of parametrization and evaluation speed?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality and significance: The presented method shows clear improvements over the studied baselines by showing how a task-based ordering can be derived from a smaller dimensional embeddings: $C \\log C$ v.s. $C^2$. The empirical analysis uses both problems with synthetically generated tasks (JIGSAW), as well as problems formulated by grafting multiple tasks. The results show the effectiveness in heterogeneous meta-learing domains.\n\nQuality and clarity: The paper is easy to follow and the experiments are explained in sufficient detail.", "weaknesses": "The paper is not self-contained and requires prior knowledge of prototypical networks and Gumbel-Sinkhorn layers. A reader not familiar with both topics will have to consult the references for basic definitions of key components of the proposed method (namely $L_{proto}$ and  $GS(.)$). A minimal overview of these should have been included in the main body of the paper, or at least in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rrF7YdzbYe", "reviewer": "Reviewer_KAtC", "summary": "The paper addresses the issue of meta-learning methods assuming a uniform task distribution, which hinders their performance on multi-modal task distributions. To address this, the authors introduce a new strategy for heterogeneous meta-learning that modulates the routing between convolution channels in neural networks. This strategy, called the Gumbel-Benes layer, offers a more efficient parameter complexity compared to existing methods and shows improved performance in terms of accuracy and runtime on multi-modal meta-learning benchmarks.", "questions": "1. Given that $\\phi(D_1)$  produces the prototype for the entire task rather than for individual classes, I'm curious: is $L_{PROTO}$'s loss computed for each class's prototype? \n\n2. A more in-depth theoretical analysis would undoubtedly solidify the paper further and provide a more comprehensive view.\n\n3. It would greatly benefit readers if the authors could include an algorithm in the appendix, offering clearer insight into the paper's approach.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**: The paper innovatively tackles the homogeneous task distribution assumption in meta-learning by introducing a modulation mechanism between convolution channels, showcasing a fresh approach to the problem. \n\n**Quality**: The proposed Gumbel-Benes layer is robustly validated on multiple multi-modal meta-learning benchmarks, evidencing its effectiveness and superiority over existing methods.\n\n**Clarity**: The presentation of the novel strategy and its underlying concepts is coherent, making the methodology and results easily understandable.\n\n**Significance**: By addressing the limitations of previous meta-learning methods and offering a more efficient parameter complexity solution, this work holds substantial potential to influence future research and applications in the field of meta-learning.", "weaknesses": "Overall, the paper exhibits commendable novelty, particularly in utilizing meta-routing to learn task-specific activations for diverse tasks. However, I have several concerns related to the experiments:\n\n1. What is the impact of the lambda in front of the prototype loss on the model's performance?\n2. Did the authors experiment with adding routers only in shallow or deep layers of the network? I'm curious about the influence of routers on the model across different layers.\n3. Concerning the complexity introduced by the Router to the model, could the authors provide comparisons in terms of parameter count and training efficiency with HSML?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)