Towards Automated Knowledge Integration From Human-Interpretable Representations
Authors: Kasia Kobalczyk
Mihaela van der Schaar
Venue: ICLR 2025 Spotlight (2025)
Keywords: informed machine learning
knowledge integration
meta-learning
data efficiency
priors
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose a new perspective on meta-learning as a paradigm enabling automated and controllable inductive bias specification, establishing a bridge between human-interpretable representations of knowledge and the hypothesis space of ML models.

Abstract: A significant challenge in machine learning, particularly in noisy and low-data environments, lies in effectively incorporating inductive biases to enhance data efficiency and robustness. Despite the success of informed machine learning methods, designing algorithms with explicit inductive biases remains largely a manual process. In this work, we explore how prior knowledge represented in its native formats, e.g. in natural language, can be integrated into machine learning models in an automated manner. Inspired by the learning to learn principles of meta-learning, we consider the approach of learning to integrate knowledge via conditional meta-learning, a paradigm we refer to as informed meta-learning. We introduce and motivate theoretically the principles of informed meta-learning enabling automated and controllable inductive bias selection. To illustrate our claims, we implement an instantiation of informed meta-learning--the Informed Neural Process, and empirically demonstrate the potential benefits and limitations of informed meta-learning in improving data efficiency and generalisation.

Published as a conference paper at ICLR 2025
TOWARDS AUTOMATED KNOWLEDGE INTEGRATION FROM HUMAN-INTERPRETABLEE REPRESENTATIONS
Katarzyna Kobalezyk University of Cambridge knk250cam ac.uk
Mihaela van der Schaar University of Cambridge mv4 720 cam. uk
ABSTRACT
A significant challenge in machine learning, particularly in noisy and low-data environments, lies in effectively incorporating inductive biases to enhance data efficiency and robustness. Despite the success of informed machine learning methods, designing algorithms with explicit inductive biases remains largely a manual process. In this work, we explore how prior knowledge represented in its native formats, c.g. in natural language, can be integrated into machine learn- ing models in automated manner. Inspired by the learning to learn principles of meta-learning, we consider the approach of learning to integrate knowledge via conditional meta-learning, paradigm we refer to as informed meta-learning. We introduce and motivate theoretically the principles of informed meta-learning enabling automated and controllable inductive bias selection. To illustrate our claims, we implement an instantiation of informed meta- -earnnin -te Informed Neural Process, and empirically demonstrate the potential benefits and limitations of informed meta- meta-learning in improving data efficiency and generalisation.
INTRODUCTION
The importance of informed ML and challenges of conventional methods. The major chal- lenge of machine learning (ML), especially in small data regimes, lies in the selection of ap- propriate inductive bias. The hypothesis space of a model must be large enough to encompass a solution to the problem hand, while also exhibiting strong preference for solutions that closely align with the ground-truth data generating process (DGP) (Mitchell) 1980; Willson & Izmailov) 2020). Conventional approaches of inductive bias selection generally rely on external expert knowl- edge and its manual integration into the learning algorithm, e.g. via feature selection, specialised loss functions, architectures or data augmentation techniques. Despite their successes, such man- ual knowledge integration methods are limited by the extent to which expert information can be transferred to the learner. Preferences over competing hypothesis can be challenging to formalise and manually integrate into ML methods; with the knowledge integration step often forming the core contribution of many ML papers Goyal & Bengio (2020). Thus, rather than relying on human- engineered MI. pipelines, it is natural seek methods that automatically integrate task-specific wwwlledde-rprreenttee in its native formats, such as natural language-ing the learning algo- rithm. In this paper, we propose the development of such automated methods for controllable bias selection, meaning that the selected biases are contingent on human-interpretable knowledge repre- sentations, facilitating intuitive and steerable specification of inductive biases
Meta-lcarning for automating the selection of inductive biases. The problem of automatically learning the inductive bias has been previously addressed by introduction of the meta-learning paradigm (Thrun & Pratt 1998; Baxter 1997: 2000). In meta-learning, the learner is embedded an environment of related tasks and the problem of inductive bias learning is seen as the problem of learning prior over the hypothesis space that enables the learner to generalise to many tasks from this environment. It has been shown that, under certain conditions, a prior learned in this manner is guaranteed to perform well when applied to novel tasks originating from the same environment [axterr 2000: Guan & Lu 2022). However, despite the appeal of letanlea methods for au- tomated inductive bias selection. they may struggle with heterogeneous task distributions and their performance often drops significantly when faced with out-of distriiibbntiinn (OOD) tasks Chen et al.
2019): Zhang et al. (002T) Furthermore, priors acquired through meta-learning are largely black-

---

Published as a conference paper at ICLR 2025
box in nature, intractably dependent the choice of meta-training tasks. Once the neta-training phase is concluded, it is not possible to condition the learner on external information specific to the task at hand, and thus failing satisfy our desideratum of inductive bias steerability.
Towards automated and controllable inductive bias selection. Drawing from the ideas of conven- tional meta-learning, we propose a model of auto-
Leaamabiie te
This paser: Can mc lcorn bis map?
Jwll)
mated and controllable inductive bias specification via knowledge-conditiones
Palf (CC)
h
1/ Hypothesis space
meta-learning (Denevi jet al. 2020, 2022). Instead of learning fixed prior (f()) over the hypothesis space, we consider learn- ing a knowledge-conditioned prior that maps arbi- trary representations of expert knowledge, K., to pri- ors, (f]))) The learned mapping. K. ++ Po(]) translator between the human-
Krz Knew eogo space
Knowledge representations K, con- dition the heterogeneous learnable prior pf(f). condiiiii-o priors ps (f(..) more tightly concentrated around the ground truth data Sata-gener functions ft, facilitating stronger inductive hiases.
Figure I:
The
can be seen interpretable knowledge space (e.g. the space of nat- ural language) and the hypothesis space of model, assigning higher prior probabilities to solutions that are in agreement with the information conveyed in K. We refer to this model in short as in- formed meta-learning. Within this framework, representations of knowledge condition the distri- bution of tasks and thus inform about task similarity, mitigating the adverse effects of heterogeneous environments of tasks (Fig. D. As result, learning with the informed prior, i.c. inferring the pos- terior pe (f K.) based on an observed dataset, D, can lead to improved data data-efficiency, requiring fewer samples of the empirical data to successfully recover the true solution.
The aim of our work is not to present a new method that surpasses existing baselines on a benchmark dataset; rather, we propose a new viewpoint on meta-learning as means of establishing an interface between human domain knowledge and the hypothesis space of ML models.
Contributions. Section 2] We formalise the interplay between observable data, its underlying generating process and numanninterpretable knowledge representations. A Section 3 We introduce a new perspective on meta-learning as an approach enabling automated and controllable inductive bias specification based on the provided representations of knowledge. Sections 3.2. 3.2.2) We provide theoretical motivations for this approach and critically discuss the potential opportunities and challenges associated with developing robust informed meta-leamers. trate our claims, we implement an instantiation of informed meta-learning the Informed Neural Process. Section 5 Through empirical evaluation on both synthetic and real-world datasets, we demonstrate the feasibility of this approach to knowledge integration, as evidenced by improvements in predictive performance and data efficiency. Section 6 Finally, we discuss future opportunities for informed meta-learning in the context of emergent abilities of foundation models.
Section 4 To illus-
2 PROBLEM SETTING: THE RELATIONSHIP BETWEEN DATA & KNOWLEDGE
Let X. y be the input and output spaces, respec- tively. A single learning task
Observable data space
(Dc, Dr, is represented by: a context (aka training) dataset Do = [(2,, 3t) ) a target (aka test) dataset Dj {(x,, y1) and a representation of knowledge about the give learning task A. We assume that context and target datasets are generated according to an underly- ing function X - Y, sampled from an unknown stochastic process f ~ p(f), and the observable data is generated according to De, Dr P The ran- domness of (f captures the unpredictable variabil- ity of the data generation, given a particular sample function f. typically modelled with a Gaussian noise. The knowledge IC is represented in a numan-interpretabbe format, such as natural language, and contains truthful, but likely incomplete information about the underlying f. For instance, in the case of 1 1-dimensional regression, IC could describe the shape of f, e.g. K. "This function should be non-decreasing" ". We assume that knowledge about a fixed f is generated according to a distribution p p(C]f) which captures the variation of knowledge representations due to their semantic
m
D
D, D (DD))
K, $ p(X))))
Underfying function space
K1
Ki
KG
Knoosaaige space
Figure 2: The generating process of data and knowledge.
See Appendix A.1.1 for formal definition of the data and knowledge generating process.
2

---

Published as a conference paper at ICLR 2025
equivalence- a single property of sample function f can be described by many semantically equiv- alent expressions in natural language, and due to the incompleteness of information information-informatio information expressed in K. is insufficient to unambiguously determine the values of f (e.g. For K = "This function is linear", , we are missing the information about the slope and bias of f).
The goal. The goal of each learning task is make predictions on Dr, given De and the additional knowledge K. (see last pane of Fig. S With a single task, we would need to rely on a human ML engineer interpret the information represented in K. and translate into model endowed with ap- propriate inductive biases, penalising (or completely excluding) functions from the hypothesis space of the model that do not conform to the expert knowledge K. This paper considers the following:
Can we learn a generalisable mapping between human-interpreet and inductive biases of a model? That is, can we learn a map: K + Po(f(K), where po (o(f(C) assigns approximately zero mass to regions of the hypothesis space that do not conform to the knowledge expressed in KC?
knowledge representations
As detailed in the following sections, learning this mapping will necessitate access to additional, related learning tasks [T, represented as tuples Ty = (Das Dr.j,K)), with contexts, targets and knowledge representations generated according to the same process (first two steps in Fig 3)
p(f), Dc,j, Dr.- ~ p(D)ff), K, pK((f)
Knowledge vs. empirical data, Having in mind the key premise of this paper which is to allow domain experts inject their prior knowledge of given learning task in intuitive and flexible man- ner, we make the following distinctions between knowledge and empirical data: D1) The domain of knowledge should be human- -iterprrantereeeabllee and can be distinct from the domain of empirical data. D2) While empirical data is noisy, knowledge should contain only true information. In contrast, ad- ditional empirical data or meta-data from potentially alternative modalities, are not a priori known to contain truthful information that is relevant for the given predictive task. D3) The way in which information contained in is related to the underlying DGP of a task is assumed to be priori understood by the domain expert. See Appendix A.2. for more details and formalization.
3
INFORMED META-LEARNING
3.1 META-LEARNING PRELIMINARIES
In conventional meta-learning, to enable the learning of the inductive bias, we embed the learner in a distribution of learning tasks. We assume access to a training set of tasks (Ti JEJ, represented as tuples of context and target datasets Tj = (Dc.j, Dr.). In the probabilistic view, the aim is to learn posterior predictive map, As De + ps PD()). This map can be also interpreted as the learning algorithm. Ag is parametrised by some unknown θ and we seek that facilitates the best generalisation from context to target data points across all datasets from our training collection. Formally, we find I that maximise s:
Ep(De,Dr) (po(yr ØT, Dc)] = Ep((). /
where (yrTlET, f) evaluates the likelihood of target data 2T. E Dr for a fixed function f from the hypothesis space. The expectation in ① is approximated based on the context and target datasets covers cases where context data is observed and
from the training set {T Setting De the learnt distribution (f = (ff corresponds to the learnt prior over our hypothesis space, Le. the learnable inductive bias (grey distribution in Fig. [T]. After selecting 8, this prior, however, cannot be adjusted based on potentially available task-specific expert knowledge.
3.2 INFORMED = WWW.WLLEEE--ONDDIIIOONO META-LEARNING
To enable the learning of inductive biases that are dependent on external knowledge about the learn- ing task, analogously to the ideas of meta-learning, we embed our learner in a distribution of learn- ing tasks, with each task containing an additional representation of knowledge about the underlying DGP of the data, T (Dc.,. Dr.j, K.)). Analogously to meta-learning, the aim is to learn pos- terior predictive map that in this case is knowledge-dependen Ao : (K,, Dc) + Po(f]De. IC). By setting De Ø, Ao becomes our sought-after map associating arbitrary knowledge representations
3

---

Published as a conference paper at ICLR 2025
Date & knowledge generation:
Meta traininiing tasks: (Ty 1e.
Test timeet
1 [)mp(()
72
Pe1
Dy.
The funcion is shapp) derraning
De
flection wae
on unddrriynng function
D Do2
Dra
This fanction incrasingg and then raacer planes
(olos.De) uninformed predictions
Dog, Dry ((D obsarvational noise
polols, Po, Ai) informed predictions
De: contedd datapoints
Kg }
Meta tetaitraining:
exgressize
arg max Dr, (polar Jey, Ao)
cnosledde
Figure 3: Informed meta-learning Successful knowledge integration via meta-learning ensures that predictions obtained with the informed marginal: pe pe(y|z, Dc. Kc): a) improve upon the unin- formed predictions obtained with ( xz, Dc); b) qualitatively reflect our knowledge of the DGP.
K. with an informed prior distribution po (f(()). This learned prior should reflect the knowledge encoded in K. To learn Ag we should choose 0 that maximises:
BpcDe. [Po FT, Dc. KO) = Ep(D,,Dr.K) / Saies//Peeriie ( fre
(2)
Figure 3 summarises the three main steps involved in informed meta-learning. By learning the mapping KC po(f K.), informed meta-learning aligns the knowledge space with the hypothesis space of model (c.f. Fig. [ At test time, this map can be utilised by a human domain expert to communicate their prior knowledge about a new learning problem in an intuitive way (due to DI-D3). If knowledge is represented in e.g. natural language, this allows for informing the learner about the expected values of predictions in a more flexible way than it would be otherwise possible with additional empirical data. For instance, we may want to convey information about the overall shape of a regression function, rather than just its value at a particular input location. We also note the generality of equations ① and (2) enabling us to consider multiple incarnations of informed meta-Icarning based on the existing gradient-based or fully amortised approaches.
Example 1 (MAML). Meta-learning: The popular gradient- based meta- meta-learming, MAML (Finn
ee al. 2018), can be recast under the probabilistic formulation of (1) (Grant et al.) 2018). In MAML, 0 corresponds to the initialization of the weights of neural network fo and the map Ao is
alised by taking a step of gradient decent with respect to Dc. Given a loss function L. pa (f(cc a delta function centred at fer neural network with weights 0' = 0 VC(fe; De), and Informed meta-learning To make the learnable prior of
p p(yrler: f) & exp(- exp(-L(f; Dr)). MAML knowledge-dependen, we can modify its original formulation so that the initialization not common for all tasks, but is conditioned on K. Specifically, we may parametrise the neural network Sax with weights WK- = (K)), where go maps knowledge representations to the initialization wx. Therefore, po Po[f[De, KC) is a delta function centred at Luig with w/x wK V[[sux Pc) and wx = go (KC). During meta-training, inform the learner about the similarity of the task at hand with respect to the previously observed learning task; two tasks with the same knowledge are mapped to the same initialization.
learn the map, 9e I. + wk. Knowledge representations
Example 2 (Amortised meta-learners). Meta-learning: Another class of meta-learning methods that can be naturally formulated through (I) are amortised meta-learners for which Ag is reduced to a feed-forward map that embeds De into a fixed vector parametrising p(yr 12r, f). Examples include prototypical networks (Snell et al. (017) the family of neural processes (Garnelo et al.) 2018a|b). b). Informed meta-learning: Informed extensions of amortised meta-learners can be easily obtained by introducing IC as an additional inputs the feed-forward map Ag.
3.2.1 DOES KNOWLEDGE IMPROVE PREDICTIONS?
With informed meta-learning introducing additional inputs in the form of knowledge representa- tions, it is intuitive to think that conditioning the learner on the extra knowledge about the DGP should improve its final predictions on the target data. This is formalised with the following theo- rem, drawing from the results of Ashman et al. 2224g (see Appendix A.3 for details):
Theorem 1. Suppose that the generating process of datasets and knowledge representations is such that datasets 'D and knowledge representations K. are conditionally independent given the underly- ing f. Let ply)2, T) be the marginal posterior distribution of y, given x and additional information 1, where 1 E (Do,(D,,N))
(De, f3, Then, De) [KL (ply)æ, allpp,, Do,KK)) S E.pC.DD) (plyk,, Alp(y)>>, Dc))], (3)
4

---

Published as a conference paper at ICLR 2025
Proof. See Appendix A.I.2
In the above, the distribution plylæ, f) is the predictive marginal of y at a fixed location æ, given the ground-truth ʃ that generated Dc and that K. describes. In practice, ʃ is unknown and we must infer it based on the available De and optionally K. Theorem [ suggests that predictions based p(y|z, KC) should be closer to the true marginal p(y|z, J) than those based on the uninformed predictive distribution plyl, x. Dc). For further theoretical results, including conditions for the inequality (3) to be strict, refer Appendix A.1.3
3.2.2 REMARKS AND PRACTICAL CONSIDERATIONS
We make several remarks regarding the practical implementation of informed meta-learners.
Existence of the meta-training set. The required dataset of tasks consisting of pairs of datasets and their corresponding knowledge representations may seem difficult to obtain, especially at scale. At the same time, the importance of incorporating expert knowledge into ML predictions is greatest where data is scarce. If for the downstream application we are interested in obtaining predictions for small dataset D, given expert knowledge Λ., the meta- veeta-training set could be obtained from related learning tasks where data is abundant and knowledge easy to obtain. For instance, in medicine, D could correspond to classification task of rare disease and IC describe the typical symptoms as provided by a medical expert. The meta-training set could then consists of classification tasks for more common diseases, where plenty of medical records exist and knowledge about them is easily obtainable, e.g. from medical textbooks. Alternatively, viewing LLMs as databases of knowledge, can generate synthetic representations of knowledge with LLMs, mimicking the ones expected to be seen at test time. We will employ this technique in Section 5.2
Finite sample approximation. In practice, we model the ground-truth p(y|z, Kc with Po (z. K.). where the parameters 0 are found by maximising the expected posterior as in equa- tion ( which is approximated based on finite meta-training set (T, jee. While the inequality in equation (T) holds for the ground-trute marginals, it is no longer guaranteed to hold for po- When the number of training tasks is too small in comparison to the complexity of knowledge representations, empirical approximation of p(y xr Do, KC) may fail and the resulting predictions could be worse than those obtained with an uninformed meta-leareen trained just on {Dc.,, Dr.j) We empirically demonstrate this in Appendix A and A.8.2] However, size of the meta-training set being sufficiently large, predictions made with the informed posterior po po(y/ Kc) should result in an improved performance over uninformed predictions, as expected.
I will illustrate section 5 with the
Reduction to meta-learning. If for a given learning task knowledge is not available, should expect that the predictions made with an informed meta-learner trained tasks {(C,,, Dc., Drj)hje are no-worse than those obtained with an equivalent, uninformed model that has been trained just on {(Do,, Dra))ie.)
generalisation & distribution shift. While generalisation to previously unseen, and semantically novel knowledge representations is highly desirable, it remains significant challenge, due the is- sue of distribution shift. If the underlying knowledge generation process p(K])) remains unchanged test time, then knowledge representations that are semantically novel with respect to those ob- served during training must necessarily correspond to novel data generating functions f that are OOD in comparison to the training examples.
The next section introduces practical instantiation of informed meta-learning. This is followed by a set of illustrative experiments aimed at highlighting the above discussion.
4 INFORMED NEURAL PROCESSES
To illustrate the concepts discussed in the previous section, we introduce specific example of an informed meta-learner, Informed Neural Processes (INPs). The proposed class of methods builds on Neural Processes (NPs)-a a family of amortised meta-learners first introduced by Garnelo et al. 2018a b) and later extended with attention
architectures (Kim et al. 2019 Nguyen & Grover, 2023). NPs reduce the computational cost of learning to feed-forward operation, eliminating the need for costly gradient-base optimisation. NPs also model a distribution of functions, given the contextual inputs, rather than returning point predictions. This property of NPs is of particular inter-
5

---

Published as a conference paper at ICLR 2025
est to us, as, given the incompleteness of expert knowledge, a single knowledge representation may correspond to multiple underlying functions; a faithful informed meta-learner should adequately represent this form of uncertainty. We remind that the focus of this paper is not to demonstrate state-of-the-art performance on an existing benchmark dataset, but rather to put forward the idea of incorporating expert knowledge into MI predictions via informed meta-learning. Consequently, our implementation remains simple and lightweight, and its details can be found in Appendix [A.4]
The model. We extend the in-context inputs of conventional NPs with the representations of knowl- edge pertaining to each learning task and model the predictive posterior as:
po I, Dc, KC) = (y I, k)) = pes (3 2, 2)ge. (z rc, Ir, k)dz, 0 0= (0.,
(4)
In the above, Poα 2) and (x rC, k) model the conditional distributions p(y|æ, f) and P(f]Dc, K)) through a latent variable E Rd The variational distribution qe, (-Ire, depends on both the contextual datapoints De and expert knowledge K. and is modelled with encoder network ha. consisting of two sub-networks: the data encoder ho.. and knowledge encoder hg.... Fusion of knowledge with data is realised with an aggregation operator a, that he, K)),K) a(re, k), with k = h (KC) and re = he. (Dc). We let qo, (z|rc, k) = Mz Tg) with (12:03) a(rc, k) and and we find that choosing to be simple sum of the two vectors works well in practice. For regression tasks we model the outputs with a decoder network go. so that Pa, (y 2) = (y: flz an), where (2m.O2) goa (z, 2). If either De or K. are not available, set their respective representations, TC or k, to a zero vector.
Training. INPs are trained in an episodic fashion over distribution of learning tasks Tj (K,, Dc.j Dr.j). Omitting the dependence on j. we denote by re and rr the representations of context and target data, respectively, and by k the knowledge embedding vector of a single task. Parameters of the model are learned by maximising the expectation of ELBO over all training tasks:
log po (yr))
(yT| ØT, TC, k) (=Irr.)) [log po, (UT IT, =)] DRI. (go_ (= (r; qo_ (= [r,k)). k)).
During training, we randomly mask knowledge representations by setting k = 0. This allows for the possibility of knowledge being missing at test time. Further details the derivation and estimation of the ELBO loss can be found in Appendix A.5
5 EMPIRICAL STUDY
The experimental section is divided into two parts. First, we illustrate the benefits and challenges associated with informed meta-learning experiments with synthetic data, where knowledge repre- sentations are well-structured and there exists an analytic, closed-form expression linking knowledge with the true DGP. In the second part, we showcase possible applications real-world data where the underlying DGP is unknown and knowledge may be loosely formatted, particularly, represented in natural language. Full experimental details are described in Appendix AA.6]
5.1 PART I: ILLUSTRATIVE EXPERIMENTS
Based on the points made in section 3.2.2 we aim explore the following questions: Q1) Do INPs successfully learn to integrate know ledge into their predictions, and, as a result, improve the data efficiency of the learne? Q2) If knowledge is not available at test time, does the performance of the INP model reduce to that of an equivalent NP model? Q3) Does the additional knowledge about learning task help in mitigating the negative affects of task distribution shift? If yes, can the approach of conditional meta-learning enable generalisation to previously unseen and semanti- cally novel knowledge representations? Q4) What are the qualitative differences in the impact of knowledge vs. additional empirical data on our predictions?
5.1.1
Q1 Q2: KNOWLEDGE AND DATA EFFICIENCY.
Setup. For each task, context, and target data points are sampled according to the following process. A function f is sampled from the family of sinusoidal functions with linear trend and bias, f(z) ax + sin(bz) + c, for some randomly sampled values of the parameters a, b, c. We introduce a Gaussian observational noise, yi = f(x)) + 6, 6 AN(0, 0.2). The parameters a, b, c

---

Published as a conference paper at ICLR 2025
randomly sampled according to: U[-1,1], Ul. U[-1,1] We let KC encode the value of two, one or none (KC = 2) of the parameters a, b. or c. The number of context points n ranges uniformly between 0 and 10; the number of targets is set to
100. This setup simulates
scenario in which K, contains partial, incomplete information about f. By training over a distribution of tasks т; we expect the model to learn how put strong prior the function's slope, level of oscillations and bias.
Results. Fig. 4a shows the estimated log-likelihood on the test tasks against the number of context data points for both the original NP model and the INP. Results for INP are shown with knowledge pre- sented at test time (K * Ø) and when it is omitted (K = Ø). We observe that informing the model m significantly improves predictions. As the number of context points decreases, the performance gap between raw and informed predictions increases. Moreover, under K = 2, INPs performs on par with vanilla NPs. Thus, the ability to condition the prior on expert knowledge is not at the cost of reduced performance of purely data drtvenriven predictions.
Nagetive Log LogglellPood
-100
-150
-20
Kigh
Model:
NP
-300
10
15
Number comaxt datapolnts
(a) Avg. log- eigliikllhoodd vs.
To summarise the impact of knowledge on the predictive perfor- mance of INPs compute the relative AAUC score defined as the integral of the "A-likelihood against n:" (von Rueden et al.] (023). with " "A-likelihood"" is defined as: p (Dr (Dc, p (Dr (Dc). We report relative values with respect to the AUC of the uninformed pre-
number of context points.
AAUG
dictions. Fig. 4b shows the estimated AAUC depending on which of
the parameters a, b, or è have their values revealed at test time. Intu- itively, exposing more information about f should provide the model with stronger priors, simplifying the learning problem. As expected, the performance gains are larger than when [AC] =
[4]
(8)
[-] (6,0) (o.4) (6.))
Format of R.
Relative improvement by knowledge rype. Figure 4.
when |A||
5.1.2
Q3: DISTRIBUTION SHIFT AND GENERALISATION OF KNOWLEDGE
Setup. Performance of meta-learners often drops drastically in the presence of distribution shift between training and testing tasks [Chnn et al.) (2019). In this experiment, we simulate dis- tribution shift. Keeping everything else equal as in the previous setup, for the training tasks, we sample b ~ N22, 1), and for testing tasks we let b~ (3, 1). We let K. encode the true value of b.
-50
Noqutoe Los ba bood
-330
200
MP-.C KC
citraring toala
INP.A
-250
Tain
-de- lest
Results. Fig. 5a shows how the performance gap between training and testing tasks significantly reduced upon informing the model about the true value of b-the source of the distribution shift be- tween training and testing tasks. From Fig. 5b] we observe that as at test time, as we take the value of the parameter b further out of the range observed during training, the performance of the uninformed model degrades, while the INP main- tains its improved performance. Thus, in this in- stance, INPs successfully generalise to previously unseen knowledge representations.
10
Number of context contexttoooppooee cotopoltts
Valuo of
(a)
(b)
Figure 5: a) Average log-likelihood on training vs. testing tasks. b) Top: Log(loss) testing tasks. Bot- tom: Frequency of tasks a given value ob- served during training. Results presented for zero- shot tasks with De Ø. Bars represent standard deviation across the tasks within one bin of the b pa- rameter values. Providing knowledge about the pa- rameter helps in generalisation OOD tasks. INP generalises to previously unobserved values of b.
5.1.3
Q4: QUALITATIVE IMPACT OF KNOWLEDGE AND UNCERTAINTY REDUCTION.
The first two columns of Fig. show sample functions from the trained INP from experiment 5.1.1 given a single data point in De. and given the knowledge representation (see Fig. A. in the ap- pendix for more examples). We find that qualitatively, predictions obtained with the INP correctly reflect the semantic meaning of knowledge representations, with the sampled functions approx- imately representing the provided values of the parameters a, b, or As expected, knowledge
7

---

Published as a conference paper at ICLR 2025
provides information about the global behaviour of sampled functions while individual data points anchor the predictions in the x-y plane.
NPs possess key feature: the capabiliity to sample from the solution space, instead of provid- ing single point estimate. This enables us to measure the decrease in model uncertainty when incorporating expert knowledge. Our focus is primarily measuring epistemic uncertainty- the uncertainty stemming from a lack of knowledge about the true functional relationship, and not the inherent randomness of the data-generating process.
informed by diets For Po
Informed by kncleedge ~ (M)))
Reduction opisiemm uncoriaitty
To measure the impact of contextual inputs on uncertainty reduction of the INP, mate the predictive uncertainty as the condi- tional entropy, Hi(ps (y x, []] at fixed lo- cation x E X and 1 E (Dc, K,, (Dc, K.). Pre- dictive uncertainty is measured in the observa- tion space and therefore amounts for the uncer- tainty associated with observational noise. We can, however, decompose it as a sum:
esti-
mm
20sa eoouledge
I(y. f 2 22 prppe (1I)) [H[pe(y x, f)ll
Figure 6: Sample functions from a trained INP and re- duction in epistemic uncertainty.
opistemic
aleatooic
and approximate the predictive and aleatoric uncertaintics with MC samples. The epistemic uncer- tainty is then obtained the difference of the two quantities (see appx. A.7 for more details). The last column of Fig. 6 displays the reduction in uncertainty due to knowledge: T(y, x) - T(y, f r, XC), and due data: L(y. x) I(y, f xr, Dc), computed at all input locations As observed from the sample functions, we find that knowledge of the value of the parameter b, responsible for the oscillations, provides information about global characteristies of f. rather than local, in contrast to the single contextual data point in Dc. Future research could explore active- learning strategies (Rainforth et al.] (023) with the task-specific knowledge queried on the basis of the epistemic uncertainty (Kaddour et al. 2020; Astorga et al. 2024 Kobalczyk et al. 2025).
(-2, 2].
5.2 PART II: REAL DATA AND LOOSELY FORMATTED KNOWLEDGE
For illustrative purposes, representations knowledge in the previous part were highly struc- tured and with well-defined relationship to the underlying DGP. Naturally, in such scenarios would resolve to direct knowledge integration methods e.g., a Bayesian regression model (c.f. Ap- pendix A.8.4). However, the advantages of informed meta-learning become evident when: a) the functions to be learned lack known, closed-form expression; b) knowledge about the learning task loosely formatted, making manual integration of prior knowledge significant challenge.
5.2.1 INFORMED WEATHER PREDICTIONS
Setup: We use the sub-hourly temperature dataset from the U.S. Climate Reference Network, rep- resenting values of the air temperature measured at regular 5-minute intervals. For each task, target observations uniformly sampled from a 24h time range. Context data points are selected by sub- sampling at most 10, chronologically first samples. This setup enables us to assess extrapolation. We perform independent experiments with two formats of knowledge:
A: For each task, knowledge IC a vector encoding two values: the minimum temperature and the maximum temperature the day.
B: For each task, knowledge K. is synthetically generated "weather forecast" presented in natural language. We generate these with GPT-4 OpenAl et al. (2023) prompted write two sentences mimicking weather forecast, based on values from the ground truth temperature measurements,
Fig. shoshows representative examples of the daily temperature paths from test tasks alongside purely data-driven and informed predictions. NPs capture the general trend of the temperature rising during the day, and then falling down towards the night, but unsurprisingly, fail to accurately extrapolate beyond the observed regions. This is due to high level of heterogeneity present in the collection of meta- tetta-training tasks, which is reflected in the high variability of the sampled functions outside of the observed data range. In terms of the informed predictions, we observe that the information contained
8

---

Published as a conference paper at ICLR 2025
NP
(4)
NP
n
INP
INP
0
63.5 (1.)
60.7 (3.0)
21.1 (1.4)
17.3 (1.3)
3
14.0
16.3 (1.3)
Tempooin
30
5
11.1
13.8 (1.2)
10
4.1 (0.5)
1.1 (0.3)
15
0.6 (0.3)
0.1 (0.2)
AAUC
24.6
22.2 (1.0)
Time
(a) Sample predictions. NP: uninformed, INP (A): informed by knowledge about the minimum and maximum temperature on the day. INP (B): informed by knowledge about the tempera- ture presented in text format (see appx. A.6
(b) Relative performance gap (%) be- tween informed and uninformed predic- tions. Bootstrap standard errors brackets based on 110 testing tasks.
Figure 7.
in K. enables guided extrapolation beyond the observed range of values and reduces the variance of the sampled functions. Table 7b) compares the performance gap between informed and uninformed predictions. Notably, knowledge enables sensible, O-shot predictions with average improvement in log- logllikel 57.7% and 48. 4% for setups A and B, respectively. We also note that the rep- resentation of knowledge, as presented in setup A should, in the theoretically optimal case, impose hard constraints on the maximum and minimum values of the function's 's range. However, given that INP only a neural approximation of these constraints, the resulting curves may exceed the specified range as opposed to strictly adhering to it; as it could be possible with a custom-designe model that explicitly incorporates such constraints into its optimisation objective.
Take-away: In practical scenarios, predictive functions are difficult to model with closed-form mathematical expressions, making the process of external knowledge integration a challenging task. The benefit of neural, meta- approaches, is their functional flexibility. NPs learn non-trivial "kernels" from the collection of training tasks directly. INPs take this a step further, enabling the incorporation of non-trine con-trivially representable information about the underlying function into the model.
5.2.2 INFORMED IMAGE CLASSIFICATION
Setup: We apply INPs to few-shot classification the CUB-200-2011 dataset Wah et al. (201 We use 100 bird categories for training, 50 for validation, and 50 for testing and follow the standard N-way, k-shot classification setup. We adjust the INP architecture to suit the image classification task, employing CLIP vision and text encoders Fu et al. (2022) (details in appx. A.6.3).. We perform independent experiments with three formats of knowledge:
A: Knowledge represents features of a given bird class, e.g. wing span, feather color. Class-level attributes obtained by averaging the attribute vectors associated with each image from the dataset. Class-level attribute vectors are stacked together to obtain N' x 312 tensors.
B: Knowledge represents class-level textual descriptions of the N classes obtained by averaging sentence embedding of individual image captions belonging to the given class. We use human- generated captions collected in Reed al. (2016) and embed them with CLIP. Per-class aver- aged text embeddings are then stacked to form N x 512 tensor.
C: Here knowledge represents set of N individual descriptions of each class. We generate these with GPT-4 based on the captions from B (see appx. A.6.3 for examples).
Table D displays results for 5-way and 10-way classification. Across all settings, we observe higher classification accuracy when additional knowledge is utilised. This trend holds for 1, 3, 5, and 10-shot tasks, with the performance gap widening as the number of shots decreases. Moreover, the information about characteristic elements of each class contained in XC proves sufficient for relatively good zero-shot prediction performance. While the zero-shot performance for setup C is lower than that of setups A or B, it is nevertheless significantly higher than the accuracy of random guessing.
Take-away: INPs align the representations of images and knowledge about class-specific features to construct latent representations that contain the essential multi-modal information. This align- ment facilitates robust generalisation to new, previously unseen classes, enabling both zero-shot classification and improved few-shot classification accuracy.

---

Published as a conference paper at ICLR 2025
Table 1: Accuracy (%) on N-way, k-shot classification tasks for the CUB-200-2011 dataset. Num- bers in brackets represent the bootstrap standard errors of the estimates based on 60 tasks per each setting. Individual tasks are constructed with only previously unseen bird categories.
N N= 5
N 10
k
NP
INP (A)
INP (B)
INP (C)
NP
INP (A)
INP (B)
INP (C)
87.5 (0.7)
87.4 (0.5)
50.3 (0.6)
81.1 (0.4)
78.5 (0.4)
33.7 (0.3)
1
82.2 (0.6)
88.1 (0.6)
89.1 (0.5)
85.1 (0.5)
73.3 (0.4)
82.2 (0.4)
81.9 (0.4)
77.0 (0.5)
3
87.0 (0.5)
88.4 (0.6)
89.3 (0.5)
88.3 (0.5)
79.8 (0.4)
82.7 (0.4)
82.7 (0.4)
81.8 (0.4)
88.1 (0.5)
88.5 (0.6)
89.6 (0.5)
88.9 (0.4)
81.5 (0.4)
82.8 (0.4)
82.8 (0.4)
83.0 (0.4)
10
88.5 (0.5)
88.5 (0.6)
89.6
89.0 (0.4) 82.6 (0.4)
82.7 (0.4)
83.0 (0.4)
84.1 (0.4)
6 RELATED WORK, LIMITATIONS, & IMPACT
Manual knowledge integration. Conventional approaches to knowledge integration rely on the ML. engineer translating the a priori known property of the underlying function, into an inductive bias of the model. This has been successfully achieved by introducing specialised loss functions (Wu ctt al.] 2018 Karpatne et al. 2017), enforcing informative priors (Marêlle
& Schoot 2017 Constantinou et al. 2016), designing new model architectures (Cohen & Welling Butter et al. 2018). However, such manual approaches are limited by the extent to which expert information can be explicitly encoded in the learning algorithm. Even seemingly straight- forward tasks, such as enforcing monotonicity or other shape constraints often require significant engineering effort (Riihimkii & Vehtari 2010 Link et al., 2022). This restricts representations of expert knowledge to formal mathematical expressions (Karpatne et al. [2017] Qian et al. 2021)) knowledge graphs (Choi al. 2017 Zhang et al. 2019). or logical rules (Yang et al. 2023) Richard- son & Domingos 2006). In contrast, informed meta-learming offers a way to integrate knowledge from any source, including natural language, allowing for greater flexibility in the type of infor- mation that can be transferred to the learner. On the other hand, this approach lacks the guaranteed correctness that conventional methods enjoy. With model-based methods, properties like equivari- ance sparsity can be guaranteed by design. Informed meta-learning in its presented form can only approximate the true meaning of knowledge and its performance contingent on, i.a., the number and coverage of the training tasks, lack of spurious correlations, and complexity of knowledge (see Appendix (.8)). Finally, model-base methods do not require access to additional learning tasks that informed meta-learning assumes.
LLMs to automate knowledge integration. As LLMs continue to improve, they show potential to act out-of-the-box, in-context learners, capable of integrating expert knowledge into their pre- dictions. Recent works by |Requeima et al.] (2024); Jin et al. (2024) explore strategies for eliciting numerical predictions over time series data, given a set of training points and expert information pro- vided as text within the LLM's context window. Such purely LLM-based methods create an interface for human users to incorporate expert insights through language, while also leveraging problem- relevant knowledge embedded in the LLM. These approaches eliminate the need for training an informed meta-learner from scratch. On the other hand, they introduce the risk of incorporating unwanted biases and require transforming numerical datasets into tokens which may limit applica- bility and performance (Gruver et al., 2023). Future research should explore the mutual benefits of LLM-based learners and informed meta-learning for instance, by fine-tuning an LLM on the meta- training dataset, or presenting a sample of training tasks within the context window. Such strategies should enable to extend informed learning with LLMs beyond the current foeus 1-dimensional time-series data. LLMs may also serve as a useful tool for synthetic data generation, addressing the limitation of informed meta-learning due to potential difficulties in obtaining the training examples.
An extended discussion of the related work can be found in Appendix A.3
Impact. In this paper, have proposed new perspective automated inductive bias specifica- tion based on expert knowledge represented in human-interpretablle formats. We note that our work primarily focuses on putting forward the idea of informed meta-learning with the introduced class of INPs serving mainly as an illustration. We hope that this paper inspires future research to ex- plore new architectures improving learning efficiency and generalisation to novel representations of knowledge as well as real-world applications across a range of domains where expert knowledge is present (see A.2.3 for examples). The growing capabilities of LLMs present an exciting opportunity for informed meta-learning and we look forward to future developments in this area.
10

---

Published as a conference paper at ICLR 2025
7 REPRODUCIBILITY AND ACKNOWLEDGEMENTS
Reproducibility. Appendix A.4.A.6 contains all the necessary details needed to reproduce the ex- periments presented in the main body of this paper. We also provide access the anonymous reposi- tory containing the source code together with the instructions to reproduce the experiments from sec- tions 5.1 and 5.2.1 (code for the image classification experiments will be made available upon paper acceptance). We also release the synthetically gegerrated datasets with GPT-4 used in experiments 5.2.1 and 5.2.2 Code and data can be found https://githhub.. kwoaaaasaaaaoaayyyyaaaa //informed-meet. learning and at a wider lab repository: https://github.com
Acknowledgements. This work was supported by Azure sponsorship credits granted by Microsoft's AI for Good Research Lab. Katarzyna Kobalczyk is supported by funding from Eedi
REFERENCES
Zeynep Akata, Florent Perronnin, Zaïd Harchaoui, and Cordelia Schmid. Label-Fmbedding for Image Classification. CoRR, abs/1500.08667, 2015. httphtttp://aarvvv.oo//////////s
08677
arXiv: 1503.08677.
Ziad Al-Halah, Makarand Tapaswi, and Rainer Stiefelhagen. Recovering the Missing Link: Predict-
ing Class-Attribute Associations for Unsupervised Zero-Shot Learning. CoRR, abs/1610.04787, 2016.
16600 610.6.7777777
arXiv: 1610.04787.
Matthew Ashman, Cristiana Diaconu, Adrian Weller, and Richard E. Turner. In-Context In-Context
Learning with Transformer Neural Processes. In Javier Antorán and Christian A. Naesseth (eds.), Proceedings of the 6th Symposium Advances in Approximate Bayesian Inference, volume 253 of Proceedings of Machine Learning Research, pp. 1-29. PMLR, July 2024. URL https:
//proceedings mlr. wrw.sprsssp.......
html
Nicolás Astorga. Tennison Liu, Nabeel Seedat, and Mihaela van der Schaar. Active Learning with LLMs for Partially Observed and Cost-Aware Scenarios. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neu- ral Information Processing Systems, volume 37, pp. 10819-20855 Curran Associates, Inc., http:c/cpppppeeeeeeepeeeeeeeeeee files/paper/2024] fi fle224331100422433233444332e32e2
2024. URL httttp:://ppooceed.ggg....
J. Baxter. A Model of Inductive Bias Leariing Journal of Artificial Intelligence Research, 12: 149-198, March 2000. ISSN 1076-9757. doi: 10.1613/jair.731. URL /tattt:://xxdoo/ .oo..
10.1613/jair 731 Publisher: AI Access Foundation.
Jonathan Baxter. A Bayesian/Information Theoretic Model of Learning to Learn via Multiple Task Sampling. Machine Learning, 28(1):7-39, July 1997. ISSN 1573-0565. doi: 10.1023/A: 1007327622663. URL httlittts:://ee doi. org/ 10 0. 12.0233333 1 0000003332 7 6226633
Filipe de Avila Belbute-Peres, Yi-fan Chen, and Fei Sha. HyperPINN: Learning parameterized differential equations with physics-informed hyperntworkss, 2021. .eprint: 211 2111.01008.
Anja Butter, Gregor Kasieczka, Tilman Plehn, and Michael Russell. Deep-learned Top Tagging with a Lorentz Layer. SciPost Physics, 5(3), September 2018 ISSN 2542-4653. doi: 10.21 scipostphys.5.3.005 URL. UtI.http::////// www. / 0. 214 68 Set.:/ccPPPssso Post 5.3. Publisher: Stichting SciPost.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A Closer Look at Few-shot Classification. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https:
7/ //openevview.oo www./et//oooo um ? id HKxLXnAcFQ
Edward Choi, Mohammad Taha Bahadori, Le Song, Walter Stewart, and Jimeng Sun. GRAM:
Graph-Based Attention Model for Healthcare Representation Learning. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '17, pp. 787-795, New York, NY, USA. 2017. Association for Computing Machinery. ISBN 978- 978-1-4503-4887-4 doi: 10. 10.1145/3097983.. 7.79888399126 URL
3097983. 3098126 event-place: Halifax, NS, Canada.
11

---

Published as a conference paper at ICLR 2025
Taco Cohen and Max Welling. Group Equivariant Convolutional Networks. In Maria Florina Balcan and Kilian Q. Weinberger ), Proceedings of The 33rd International Conference on Machine Learning. volume 48 of Proceedings of Machine Learning Research, pp. New York, USA, June 2016. PMLR. URL https: ptttps://proceedingg ngs. p.psss//445
New York,
coohenc 6. html
Anthony Costa Constantinou, Norman Fenton, and Martin Neil. Integrating expert knowledge with data in Bayesian networks: Preserving data-driven expectations when the expert variables main unobserved. Expert Systems with Applications, 56: 16:79922 2016. ISSN 0957-4174. doi: http:///.ooooo.....0..ocooaaaaaa Beienne//arrccee///////571111111
URL https: http///wwiieeeeireeettttttteeeee com/
Giulia Denevi, Massimiliano Pontil, and Carlo Ciliberto. The Advantage of Condi- tional Meta-Learning for Biased Regularization and Fine Tuning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neu- ral Information Processing Systems 2020, NeurIPS 2020. December 6-12. 2020, vir- wttpscppppppccc.....////ppp htbber///200/hah// html
tual, 2020. URL
Giulia Denevi, Massimiliano Pontil, and Carlo Ciliberto. Conditional Meta-Learning of Linear Rep- resentations. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave. K. Cho, and A. Oh (eds.). Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 De- cember 9, 2022, 2022. URL
ni cc pttppp/peeeee.....eeeeeeeeeaeaee http:/apperriee ffrr__fiii_e www.fffleleee////e hash/Oleed3 9ca4 www.eedddecccceceeeeeee oec.aaacc......ee.e.e33773333333 -tw:::444771-1--tb--eeee Conference - htmm
Stefan Depeweg, Jose-Miguel Hernandez-Lobato Finale Doshi-Velcz and Steffen Udluft. Decom- position of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1184-1193, wwl..press/// /itess/v800 vtt http:////dddeeeeeee/e///
PMLR, July 2018. URL
html
Changxing Ding and Dacheng Tao. Robust Face Recognition via Multimodal Deep Face Represen- tation. IEEE Transactions on Multimedia, 17(1 7711))2049-2053 2015. doi: 10. 1095 2477042.
Mohamed Elhoseiny, Ahmed M. Elgammal, and Babak Saleh. Write Classifier: Predicting Visual Classifiers from Unstructured Text. IEEE Trans. Puttern Anal. Mach. Intell., 39(12):55992255; 2017. doi: 10. 10.1100PAMM...11..2 26643666777.. URL 1ttp:://o...../11111111111/1////
2016 2643667
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic Model-Agnostic Meta-Learning. In Samy Bengio, Hanna M. Wallach, Hugo I Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances Neural Information Processing Systems
31: Annual Con- ference Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, neurips............. cc/ .tmll
Montréal, Canada, pp. 9537-9548, 2018 URL ht http:tpp:://oceeeeeepeetet gepprr2208//ie https//eccccccaaaaoaooaa.32e23a0
Jinmiao Fu, Shaoyuan Xu, Huidong Liu, Yang Liu, Ning Xie, Chien-Chih Wang, Jia Liu, Yi Sun, and Bryan Wang. CMA-CLIP: Cross-Modality Attention Clip for Text-Image Classification. In 2022 IEEE International Conference on Image Processing (ICIP), pp. 2846-2850, 2022. doi: 10.1 0TT.0///CCCAA777722299999999
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional Neural Processes. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1704-1713. PMLR, July 2018a. URL |tttts: httpp://pxoceeddnng: mir. prepspeesssse.. /tzess/v80/qee http///vvgggaaeeell....
html
12

---

Published as a conference paper at ICLR 2025
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Tch. Neural Processes, 2018b. _eprint: 1807.01622.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative Adversarial Networks. CoRR, 1b6/1406.2661, 2014. URL
2661
arXiv: 1406.2661.
Anirudh Goyal and Yoshua Bengio. Inductive Biases for Deep Learning of Higher-Level Cogni- tion. CoRR, abs/2011..55t00 15099,,, 2020. URL.https//aaxxiv/ http://wwww.vvgaa/////aaapaaaa 2011 15091 2011.15091.
arXiv:
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting Gradient- Based Meta-Learning as Hierarchical Bayes, 2018. _eprint: 1801.08930.
Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large Language Models Are Zero-Shot Time Series Forecasters. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neu- ral Information Processing Systems 36: Annual Conference cessing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Neural Information Pro-
URL http::/ppppp......pppppp
html
Jiechao Guan and Zhiwu Lu. Task Selatedness-Baseee Generalization Bounds for Meta Learning. In International Conference Learning Representations, 2022. Utl/ittts::/opeerevrie
net/foc fortm/oorm?11 A3
Dan Hendrycks and Kevin Gimpel. Bridging Nonlincarities and Stochastic Regularizers with Gaus- sian Error Linear Units. CoRR, abs/ ftss/606.004115 2016. URL http:://arxiv..rr////a
1606. 08415
arXiv: 1606.08415.
Jonathan Ho and Tim Salimans. Classifier-Free Difftusion Guidance. CoRR, ab//2207,77225 2022. doi: 10.4 (0.48500/ARXIV.22000 12255. 12598. URL http:///d...///10.0.5555/////1..
12598
arXiv: 2207.12592
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Hugo Larochelle, Mare' Aurelio Ranzato. Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL 4c55ccecc888
nexrips /c/ http:eeapppp..cppaee///00000000 wccpppper/222//aak. 1twe6//f1111119994--b1 Abst Absttaaaaatbbtttttttttt....
Xiang Huang, Zhanhong Ye, Hongsheng Liu, Shi Ji, Zidong Wang, Kang Yang, Yang Li, Min Wang. Haotian Chu, Fan Yu, Bei Hua, Lei Chen, and Bin Dong. Meta-Auto- Decoder for Solving Parametric Partial Differential Equations. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference
Neural Information Process- ing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http: /tt:::/papppe http:/pppprrssssmm nips wwwpiissccppppppe cw/paper.pppppppccpppppe fi weeesrffilessspeeee wttpf/ss//aape//22////aaa. 5b 948552171302d33a:1t 5tw.499911551111114111 d7e Atteeettttt--tttttttt-------- Confrrrrac-oooooeeeeeeeee html
Tomoharu Iwata and Atsutoshi Kumagai. Sharing Knowledge for Meta-lcarning with Feature De- scriptions. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Ad- vances in Neural Information Processing Systems, 2022. URL httlhttps:/// /ttp::////peeeieeeeeeee net/ Eorum? 1d=kyd=ddoooo weokkyy4444ggtt
Ming Jin, Shiyu Wang. Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li. Shirui Pan, and Qingsong Wen. Time-LLM: Time Series Fore- casting by Reprogramming Large Language Models. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
URL http:ps:///pperevieeee
13

---

Published as a conference paper at ICLR 2025
Jean Kaddour, Steindór Sæmundsson, and Mare Peter Deisenroth. Probabilistic Active Meta- Learning. In Hugo Larochelle. Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tieun-Tien/aee Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. ef
neur
newr ips ct/ppaper//22///asiii Abst html
Anuj Karpatne, William Watkins, Jordan Read, and Vipin Kumar. Physies-guided neural networks (pgnn): An application in lake temperature modeling, arXiv preprini arXiv 1431, 2, 2017.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Gamelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive Neural Processes. In International Conference on Learning Representations, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URI. http:
// www.v.org/eaa....a........... abs 1412. 6980
Diederik P. Kingma and Max Welling. Auto-Ense huto-Encoiing Variational Bayes. In Yoshua Ben- gio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banjf, AB, Canada, April 14-16, 2014, Conference Track Proceedings. 2014. URL
http://arxivy wwwgg/bs//iaooo
6114
Kasia Kobalczyk, Nicolás Astorga, Tennison Liu, and Mihaela van der Schaar. Active Task Disam- biguation with LLMs, In The Thirteenth International Conference on Learning Representations, 2025. URL http:::::22.. http://ppppww.e..../ooo net/ www.moodd-nMMaeeeee....
Alexander Kovacs, Lukas Exl, Alexander Kornell, Johann Fischbacher, Markus Hovorka, Markus Gusenbauer, Leoni Breth, Harald Oezelt, Masao Yano, Noritsugu Sakuma, Akihito Kinoshita, Tetsuya Shoji, Akira Kato, and Thomas Schrefl. Conditional physics informed neural net- works. Communications in Nonlinear Science and Numerical Simulation, 104:106041, 2022. http://o.oorr//./..010oooos.ma.. 106041. URL https: /www. sciencedirect wwt.cccccccci.eeee
ISSN 1007-5704. doi: htt.:///we..t.cc...........eeeaa
Patrick Link, Miltiadis Poursanidis, Jochen Schmid, Rebekka Zache, Martin von Kurnatowski, Uwe Teicher, and Steffen Ihlenfeldr. Capturing and incorporating expert knowledge into machine learning models for quality prediction in manufacturing. Journal of Intelligent Manufacturing, 33(7):2129-2142, October 2022. ISSN 1572-8145. doi: 00010077/118445022-099755 URL
https://doi.co www.:/...rg1....
0122-01975-4
Yinhan Liu, Myle Ont, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. (BRRT)) Robustly Optimized [BERT] Pretraining Approach, 2020. URL https://openreviews wwtpeeewwwwwwweeeeeeeeeeeeeeeeee wwt.:orummfooiooooooooooooooo
Syx80I4tvS
Sarah Depaoli Mariëlle Zondervan-Zwijnenbuug, Margot Peeters and Rens Van de Schoot. Where Do Priors Come From? Applying Guidelines to Construct Informative Priors in Small Sample Research. Research in Humma Development, 14(4):305 320, 2017. doi: 10.1080/15427009.200
1001://0000.0010.111110//0//// 11101100000011111111111 040754426399 00...0.....1.2.... 2017 1370966. Routledge eprint: http:://oo..nrg//o0.o/////57777 00t.///000991111177799999
1370966. URL
Publisher:
Mehdi Mirza and Simon Osindero. Conditional C Generative 2014.
C Adversarial Nets. CoRR, abs/1411.1784, arXiv: 1411.1784.
Tom M. Mitchell. The Need for Biases in Learning Generalizations, Technical report, Rutgers Uni- versity, New Brunswick, NJ, 1980. ht http:::p:://mml/ http://a......p.peeeee/e//////ae m/lmm GcoLs///e.diin//eee22252o2EE222.
pdf
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y. Ng. Mul- timodal Deep Learning. In Lise Getoor and Tobias Scheffer (eds.), Proceedings of the 28th In- ternational Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28 July 2, 201 pp. 689 696. Omnipress, 2011. URL https: /wcml. wttpccc//apper........11/1 399 cml paper. .pd
14

---

Published as a conference paper at ICLR 2025
Tung Nguyen and Aditya Grover. Transformer Neural Processes: Uncertainty Meta Learning Via Sequence Modeling, 2023. URL https///aaxw...r////a////0...... 04179 .eprint: 2207.04179.
OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1ige Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavar- ian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadeut-hhaiiro Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai. Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte. Isabella Fulford, Leo Gao, Elie Georges, Chris- tian Gibson, Vik Goel, Tarun Gogineni. Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gor- don, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaf- tan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstan- tinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung. Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Man- ning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob Mc- Grew, Scott Mayer McKinney, Christine MeLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing. Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang, Cullen Keeffe Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Gi- ambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Ray- mond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr. John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Ben- jamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Eliza- beth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang. Ben Wang, Jonathan Ward, Jason Wei, C. J. Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin WU, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. GPT-4 Technical Report, 2023. _eprint: 2303.08774.
Vallone,
Tzuf Paz- Argaman,,,, Reut Tsarfaty, Gal Chechik, and Yuval Atzmon. ZEST: Zero-shot Learning from Text Descriptions using Textual Similarity and Visual Summarization. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pp. 569-579. Association for Computational Linguistics, 2020. doi: 11065///V12020.1INDD0
-EEML... EMNLP.
50. URL https: /do.. http:/:://///iiooo/110./..010 wtt.a//...rr/11111//////// /2000.2222222 ndi ngs- 1m.
15

---

Published as a conference paper at ICLR 2025
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. FiLM: Visual Reasoning with a General Conditioning Layer. In Sheila A. Mcllraith and Kilian Q. Artificial Intelligence,
Weinberger (eds.), Proceedings of the Thirty-Second AAAI Conference (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 3942-3951. AAAI Press, 2018. doi: 10.1609/AAAI V3211.11671, URL httph: /oi. org 10 169 aaaiaaaaaaa.........aaaa.....a v32il 11671
Zhaozhi Qian, William R. Zame, Lucas M. Fleuren, Paul Elbers, and Mihaela van der Schaar. Inte- grating Expert ODEs into Neural ODEs: Pharmacology and Disease Progression. In A. Beygelz- imer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Pro- cessing Systems, 2021. URL
net nwt/ id 6wF
Tom Rainforth, Adam Foster, Desi R. Ivanova, and Freddie Bickford Smith. Modern Bayesian Experimental Design, 2023. URL attps://arxiv..e httpxi/.orrg/bb//30000 2402.14545.
abs http://.//22222222 14545
.eprint:
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents. CoRR, abs/2204.06125,, 2022. doi: 10.48550/ ARXIV.2204.06 125. URL https: //dsi: .oaddoooorggo htt.o///o///...///3333.... Aww50//05551... 2204. 06125 2204.06125.
arXiv:
Scott E. Reed, Zeynep Akata, Honglak Lee, and Bert Schiele. Learning Deep Representations of Fine-Grained Visual Descriptions. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 49-58. IEEE Computer Society, 2016. doi: 10. N0.1199/CVR..211.155 13. URL https:: orgeli.rr////00 htt.g//..1111//901 1109/CVPR. 2016 13
James Requeima, John Bronskill, Dami Choi, Richard E. Turner, and David Duvenaud. LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language, 2024. URL
https::/aaxxie org/ /ww.////255.. 12856
eprint: 2405.12856.
Matthew Richardson and Pedro Domingos. Markov logic networks. Machine Learning, 62(1):
107-136, February 2006. ISSN 1573-0565.
doi: 10.10077:109944000-555777 URL https:
10.
100005833---
Jaakko Riihimäki and Aki Vehtari. Gaussian processes with monotonicity information. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Con- ference on Artificial Intelligence and Statisties, volume 9 of Proceedings of Machine Learn- ing Research, pp. 645-652, Chia Laguna Resort, Sardinia, Italy, May 2010. PMLR. URL
https: https::/proceeeingg wwwiiinnnsm.rr...mmm www..rr..... mwr.press ppress/vvae wwvpv/i/iiinnn wwwprriiiaaaiiaaattt....
Edgar Schönfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata. Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, doi: 10.1109/CVPR 2019.00844. http://cccocooooeeec_c__LLLL
pp. 8247-8255. Computer Vision Foundation / IEEE, 2019.
URL
http::/
9/
Variational Autcencoders CVPR 2019 wwwpr.apa.....
Xingjian Shi, Jonas Mueleer, Nick Erickson, Nick Erickson, Mu Li, Alexander Smola, and Alexander Smola. Benchmarking Multimodal AutoML for Tabular Data with Text Fields. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. Curran, 2021a. URL https: //aatssess beattaast--encccaaassssspsssss wwwp://wweooooeeeeee...e.eeeeeee orgpaaeeeeeeecctcccoooocnoneennn fiteaperrfiieeepppeee https//ppe////////// 2000:///0//EE1100000000000000010
pdf
Xingjian Shi, Jonas Mueller, Nick Erickson, Mu Li, and Alex Smola. Multimodal AutoML on Structured Tables with Text Fields. In 8th ICML Workshop on Automated Machine Learning
(AutoMI.), 2021b. URL ttt.httts:///peen wwwp:ewwwwwwweeeeeeeeeeeiiiieeee T
16

---

Published as a conference paper at ICLR 2025
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical Networks for Few-shot Learn- ing. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. bttpttps: / pwwp:pooooeedddoooooooooc ngs. nwwrips....... mwurips..cc/ppprr.... files/ paper/2017/F111
F281 ortaaaae6001.11111222200000000a0
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning Structured Output Repre- sentation using Deep Conditional Generative Models. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 28: Annual Conference on Neural Informa- tion Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp.
3483-3491, 2015, URL Bd55a249e 6baa5506bbaac0066 7w http:///ww....ooooooooo////////a
neurips. wwtpppper/211///aas.....e Abst html
Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Aché Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Informa- tion Processing Systems 32: Annual Conference on Neural Information Processing Sys- tems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 11895- cc/
11907. 2019. URL htpp:::/pprooeed.ngg..ee...
Nitish Srivastava and Ruslan Salakhutdinov. Multimodal Learning with Deep Boltzmann Ma- chines. In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Sys- tems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Pro- ceedings of meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pp. 2231-2239 2012. URL /t/p:stt:::/p/rreeedinng hew.:/ooeeeednn................. newrips. cc/ http/pappr/2211//eeeeee AFF2coocs/abb222e133322222333333
Absttaaa--bbbtaatttaaea .html
Sebastian Thrun and Lorien Y. Pratt. Learning to Learn: Introduction and Overview. In Sebastian Thrun and Lorien Y. Pratt (eds.), Learning to Learn, pp. 3-17. Springer, 1998. doi: 10.
978-1-4615-5529-22 URL lttpps//dd/
10. 100700002200000 1-4615 001-446-5529-222 5529-2_1
Yau- Yao-Hung ue Hubert Tsai and Ruslan Salakhutdinov. Improving One-Shot Learning through Fusing Side Information. CoRR, abs/1770.083444 2017. URL http:://arx.vv.orgaab/////
08347 arXiv: 1710.08347.
Gabriel Tseng, Hannah Kerner, and David Rolnick. TIML: Task-Informed Meta- Leatnnn for Agri- culture, 2022. _eprint: 2202.02124.
Laura von Rueden, Jochen Garcke, and Christian Bauckhage. How Does Knowledge Injection Help in Informed Machine Learning? In 2023 International Joint Conference on Neural Networks (IJCN), pp. 1-8, 2023. doi: 10. 20C.CCCNNN554002230000999999
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD
Dataset. California Institute of Technology, July 2011.
Andrew Gordon Wilson and Pavel Izmailov. Bayesian Deep Learning and Probabilistic Perspec- tive of Generalization. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- -Ten Lin (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020.
neur ips httpppaarr/220//aaee Abst html
Long Wu, Heng Xiao, and Eric Paterson. Physics-informe machine learning approach for augmenting turbulence models: A comprehensive framework. Phys. Rev. Fluids, 377)007460,,
July 2018. doi: 10.1 Fluio Publisher: American Physical Society. 10. 10.1113/Phyysee
URL httpthttt///nnkk. http://iikkapsoooo. aps..og/dooo///
luids. 3704622
Yuan Yang, Siheng Xiong, Ali Payani, Ehsan Shareghi, and Faramarz Fekri. Harnessing the Power of Large Language Models for Natural Language to First-Order Logic Translation, 2023. eprint:
17

---

Published as a conference paper at ICLR 2025
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhut- dinov, and Alexander Smola. Deep Sets. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httpss:/ ngs wwtprriissccppppppppeppppppppppe http:///pppaaeee///aaapeeeeeepee ffless/apprr/211. Fale:f/22e Paper.
Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive Risk Minimization: Learning to Adapt to Domain Shift. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Anmual Conference on Neu- ral Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual,
pp. 23664 23678, 2021. URL bash/c7000 110051122211 www.eddeeecc... 1aaa:676111111222272222222222433
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu, ERNIE: Enhanced Language Representation with Informative Entities. In Anna Korhonen, David Traum, and Lluis Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1441-1455 Florence, Italy, July 2019. Association for Computational Linguis- tics. doi: 10.18653/ 1/111/11119111111 1139. bttppp//aaclaattoooooooooo//////
18

---

Published as a conference paper at ICLR 2025
A
APPENDIX
Table A.1: Table Contents
Appendix section
Contents
A.1.1
Explaining the notation f p(f) and the data and knowledge generating process
A.1.2
Proof of Theorem
A.1.3
Further theoretical results following Theorem
A.2.1
Extended discussion on assumptions D1-D3 regarding data vs. knowledge
A.2.2
A short discussion the curation of meta- training datasets
A.2.3
Example applications informed meta-learning
A.3
Extended related work section
A.4
Details of the INP architecture
A.5
Training details and derivation of the ELBO loss
A.6
Details of experiments presented in Section 5
A.7
Details on uncertainty quantification with INPs
A.8
Additional experiments: A.8.1 the impact of trtt-training size on suc- cessful knowledge integration; A.8.2 the impact of knowledge complexity on successful knowledge integration; A.8.3p the impact of spurious corre- lations of knowledge and data on correct "knowledge interpretation" AA8.4 meta-learned vs. exact knowledge integration.
A.1 THEORY
A.1.1 FORMALISM
Let (2, Co, P) be an abstract probability space and ()), Σy) the measurable space of outputs. We define stochastic process F' as a collection of y-valued random variables:
E F := {F(x): X).
A single realization of F(-;w) : X' + y, is function between the input space X' and the output space y. We will denote with p P F-1 the law of this stochastic process, so that for subset FCX
p(F P((w E E F(::w) E F)),
Thus, the notation p(f) corresponds to sampling function F(::w) according the law p.
The data-knowledge generating model. We assume the following data and knowledge DGP First, a sample, not directly observable function f X' y sampled according to f p(f). Given f. the pair of context and targets datasets is generated according to Dc, Dr ~ p(D).) and the corresponding knowledge representation is generated according by ~ p(K]f)). Here p(D)j) and p(K]J)) are the data and knowledge generating distributions respectively. Note, this model assumes that knowledge representations and empirical data are conditionally independent given single realization f.
The somewhat unconventional notation of conditioning on functions f. through p(D).f) and p(C])) serves highlight the role of knowledge expressing information about the true underlying func- tional relationships between model inputs x E X' and its outputs E y
With slight abuse of notation, will denote by p(f. De, XC) the joint distribution on functions, datasets and knowledge representations. The assumption conditional independence tells us that factorise p(ʃ, Do, K.) p(Pcl/)p(K))
19

---

Published as a conference paper at ICLR 2025
A.1.2 PROOF OF THEOREM 1
Theorem 1. Suppose that the generating process of datasets and knowledge representations is such that datasets 'D and knowledge representations K. are conditionally independent given the underly- ing f. Let p(y) z. 1) be the marginal posterior distribution of y, given x and additional information I, where I E (Dc, K., f. Then,
De) [KL (p(y)a,, (llpy, Kc)K S EP(I.D)) (p(lo,, Dlptyll Dc))],
Proof of Theorem I (informal). The proof is analogous to that presented by Ashman et al. (2024) where K's take a particular form of an additional sample data generated according to the same stochastic process as Dc (see A.3 Meta-learning with meta-data for details).
We have that the LHS equal to:
de [KL (plylæ, ( Dc,).))
Ppp.. [p(/= f)] [ Pe) [log Do,
(6)
And that the RHS is equal to:
EoCIDD) [KL (p(y)x, M|pp, Pc))]
- (py f)] (.l::a))Ppuln.) [ogg(yylz,ec) Dc)]
We first consider the second term of 66p
Eptt..e..)) (lgppbyz, Dc,
(oti:ee.))lppilll..P..... [log plylz, Dc,
(by conditional independence)
-EpcDe Boull. [log p(u|,, Dc, K.)
= -) [log plylz, K))
(by Foubini)
= B ( KC)
S Ep(D,) [,) Dc)]
(conditioning does not increase entropy)
The last expression is equal to the second term of 7
[logp(ylæ,De) Dc)]
[log pwylz Pc)l
(by conditional independence)
ED (ppillp [log p(yl,, Dc)]
B [log p(yl,, Dc)]
(by Foubini)
E ((yl,,,
Finally, we notice that the first terms of and are equal:
Acy [p(yle v A = Ept [pyle,, [)],
as the entropy term H [p(ylar, f)] does not depend on K. This concludes the proof.
A.1.3 A FURTHER THEORETICAL DISCUSSION
Striet inequality in Theorem 1. By examining the steps in the proof of Theorem 1, we note that the inequality (5) can be made strict under the assumption of strictly positive mutual information between target values y and knowledge I the specified location X, formalised as:
I(y: K(rx)= == HI [p(y [=]] Ep(x [p(y|a, > 0
(8)
Note, we do not require that for any observed context dataset Dc. H [pyla,, Dc)] Eptxylii Jply|,, Dc)] > 0. We may imagine special cases in which given context dataset De. any additional knowledge about the underlying function would be redundant. For instance, in the noiseless setup of (..tion/5.111 with f(r) + sin(bz) + c, observing 3 distinct data points fully determines the underlying values the parameters a, b, and e; thus, any extra knowledge would be redundant in this case.
20

---

Published as a conference paper at ICLR 2025
Proposition 1. Under the assumption of S
(p) (p [p Do, (o) XC]] < H Dc)l
Proof. Note, we can decompose (p) ppll Do, [3] as sum of two terms:
By) x)) [1 Ø plyl,, Dc,
# Do.
For the first term, we have:
x))Do.) [1 0 De, [c]]
Ex(De)) [1 0) [p(y/=,
(plylæ, Ø,K) p(u|=, K.))
-Epr))Bppeeln [L = 2 [p(a|,, K]]]
(by conditional independence)
-Epc))((Dc = p [p(]a,,
-P(Dc Hx, (p(ll,,
(*)
<PP( 2) [1)]
(by the assumption (8)
= -p(D)) [D = Ø [py/==]]
The transition in (x) follows since the data generating process is such that the event 1(De = Ø} is independent of the particular choice of the data generating function f (e.g., in our experiments, the number of available data points for conditioning is sampled uniformly from a range of values regardless of the underlying function).
For the second term, since conditioning does not increase entropy, for any fixed dataset De we have:
E Ep(KPe) IHI [p(ulæ, De, K.)] < HI [p(yl Dc)].
Multiplying both sides by 1{De + Ø) and taking the expectation with respect to p(Dc) yields: (x)De.)) (Pe + Ø) [p(],,, Dc, K0]] - Ep(De) [1 H Dc)]] Combining the two inequalities gives us:
Ep(Do H [plylæ, Kc)
= K))(De.)) =
= 2) (plpl De, K)]l + Ep( [L + H)H EDD)) (1 = H [[y|=)]] + (DD) (Dc # @ p(y|., Dc)l]
EXDDE H [(yla,, Dc)]
Corollary Suppose that the generating process of datasets and knowledge representations is
such that datasets D and knowledge representations K. are conditionally independent given the underlying f. Let ply|z,
1,, I) be the marginal posterior distribution of y. given x and additional
information I, where I E (De, (Dc, K), Suppose that in addition. assumption 8 holds. Then,
Ep(..De.t)) [KL (plylx, f)ll, Dc, Kc)) < EpII.De) [KL flppllr Dc))l.
(10)
Proof. It suffices to replace in the proof of Theorem the inequality FPpxI ppyl Dc, S
H [p(y] Dc)] with the strict inequality from Proposition
When expect assumption (8) to hold true? Given the above results, it is natural consider when can we expect 1(y: X[[z) > to hold true. It may be in fact the case that K. provides localised information, e.g. IC = "The values for x smaller than O should not exceed ...". For this kind of knowledge, if a given x does not fall into the region that informs about (in the example, it is any 0), and without enforcing any further restrictions on the underlying process p(f), (8) is may hold. However, instead of considering the impact of K'ss at specified location x, it is reasonable assume that weaker conditions holds, i.e. that the conditional mutual information between the targets and knowledge I. is positive in expectation across the domain X':
Ep(x) (T;; K(x)) 0,
21

---

Published as a conference paper at ICLR 2025
in which case obtain that:
Epte. [KL folllll Dc, K.)] Bplx.f.De)
[KL (plylz, (yyy Dc))
(11)
We may also want to further break down the assumption of @ with respect to the information gain of K on the underlying y through f rather than directly, Due to the conditional independence of observed values and knowledge given f. we have the following:
I(y: K((=)) = I(CCzllæ)
H[[(KC) - Pellil)) (CK v)]
H[p(AC) EP() (p(( - B y)]
H[p(K]] BEPC) (x(([s)) Pe) H [K]],,,,,) Bp(ule) (KC()))) (t)
-Hp(())] E H (K(([)) Ep(ul=)
Hlp(K v)] Eep Hi (C) 2r,
=I(Xff)
= I(f; - Ep(ul=) [L(f: K22 y)],
where have used the conditional independence assumption in (f). Thus, Tly;K[z)) 0 if and only if:
[[ Ep(via) [[; w)] > 0,
(12)
where the above terms can be expressed as: K(f H[[(f)] - Ep(x)H[(((1)) and [[ ((f:K(,,,) = Hlpp(fl=, EIo(,,,,,, y)] Therefore, for 122 to be true we require:
1. Non-zero dependency between and K. We must necessarily have that I(f:K) O, which should trivially be true by the definition of knowledge as information about the ground truth data generating process.
2. Sufficient informativeness of (x.,) about f. Note, Eptyla) [T(f: y)] is always smaller or equal I(f: The condition of Ep(s/z) K[z, y)] < I[f: K[x),, im- plies that the observation noise of y's should be low. High noise leads to higher E (llla)) [T[[; AC|r, u)] bringing closer to [(J:K) and thus causing T(y; K((r)) to go to zero
More information leads stronger priors. the experiments, we have observed what is quite intuitive, namely, that providing the model with more information in knowledge representations results in stronger priors and thus improved predictions. To formalise this claim we can assume two competing mnowledge-generating distributions: p(Kl,f) and p(K[f), with their corresponding joints: p(f, Dc. = p(f))((ell))p((()) and p(f, Kc,K)) = p(f)p(Dclf)((011) In this case have that
plylz) - 1| ply. ((KK)aKKC
p(flpullz - p(u)z).
Then, saying that p(A).f) is (locally) more informative than ((K)f) can be formalised as:
H [p)(v=))
Do, Ep( Dc, H [p(][=)] Ex (p Dc, +
Ep(cDD [ De, Ep(, [pw]=, De,
(13)
Let,
((x) == B (KT (pT,
and
E(z) EpchncDelnsscin) [KL (p(
It is then straightforward to show that under the assumption of [1 c(z) ē(z), i.e. the posterior predictive is closer to the ground truth marginal p(y) x. f) for knowledge generated according to the more informative distribution.
22

---

Published as a conference paper at ICLR 2025
A.2 EXTENDED REMARKS, LIMITATIONS & FUTURE WORK
A.2.1 KNOWLEDGE VS. DATA
We remind the reader that the primary objective of this paper is to consider a learning method that allows domain experts to inject their prior knowledge of a given learning task in an intuitive and flexible manner. In this regard, we elaborate on what distinguishes expert knowledge from empirical (meta-) data.
In line with the goal of intuitive and flexible knowledge integration, D1) posits that the domain of knowledge should be human-interpretable and, as such, distinct from the domain of empirical data. Specifically, we argue that in many real-world applications, knowledge can be most naturally ex- pressed in natural language, which allows for the articulation of task properties that may be difficult to encode using strict mathematical equations, or additional data samples.
D2) asserts that, unlike empirical data, knowledge should consist only of true information. The assumption of truthfulness knowledge is a recurring theme throughout this paper and warrants further clarification. Referring our data and knowledge generation process, we assume the fol- lowing: first, an underlying function f ~ p(f)) is sampled. This function f generates both the observable dataset De p(Dl.) and the target dataset D/ ~ p(DIf), with p(D].) introducing the observational noise. An expert, with privileged insight into the data mata-generating process, provides a description of the expected properties of f with I. ~ p(K]f). Variability p(K]f) arises due to two factors: 1) For single function S, semantically equivalent representations may be used to express the same properties-this is particularly relevant when K. is represented in natural language; 2) Knowledge about specific function f is typically partial-only a subset of the properties of the underlying f may be reasonably expected to be known for a given task. The assumption of truth- fulness also implies that, up semantic equivalence, the process p(K]ff) should not be subject to distributional shifts, unlike p(Dlf) or p(f) itself.
Lastly, D3) states that the relationship between the information contained in K. and the underlying DGP of a task is assumed to be understood a priori by the domain expert. This implies that the process pf is, in itself, human-interprettale.
Assumptions DI) to D3) are in contrast with the use of additional data in hierarchical meta-learning setups considered in prior works, where contextual data Do is supplemented by additional meta- data about the task. For instance, (Iwata & Kumagai| 2022) condition the meta-learner on the fea- ture names of the dataset De. However, feature names alone do not convey meaningful, human- interpretable information about the learning task that informs about the underlying relationship between predictors and the target variable. While this relationship may be inferred during meta- training, it would remain unknown to the human expert, violating D3). The relationships between feature names and the underlying values of the variables is also likely to be subject to distribution shifts, violating D2). As result, even though conditioning on such additional data may lead to improved predictive performance, would not enable the intuitive steerability of inductive biases during deployment.
A.2.2 COLLECTION OF THE
DATASET.
In practice, a typical setup may begin with one or a few contextual datasets, each paired with ex- pert knowledge about the underlying functional relationships expressed in K. This raises a natural question: how can additional learning tasks required for informed meta-learning be obtained?
First, we acknowledge that the requirement of the additional training tasks may be a limitation of informed meta-learning in comparison to model-based strategies. In particular, the number of training tasks required for successful knowledge integration via informed meta-learning grows with the complexity of the knowledge representations, as illustrated in the synthetic experiment in Ap- pendix [..822 For the informed meta-learner to understand the underlying meaning of knowledge, it must be exposed to variety of datasets corresponding to semantically distinct representations of knowledge. Without semantic variation, learning a generalisable mapping between the knowledge space and the hypothesis space of model becomes impossible.
Given that interviewing a human domain expert to express knowledge about a large number of related learning tasks may be impractical, we propose that L.MMs, viewed as repositories of human
23

---

Published as a conference paper at ICLR 2025
knowledge, can be utilised for smart synthetic data and knowledge generation. While the exact methods for generating such data safely and reliably are beyond the scope of this paper, we suggest this as avenue for future work. Nonetheless, synthetic data generation augmentation is a viable and promising strategy, as suggested by experiments 5.2.1 and 5.2.2
A.2.3 EXAMPLE APPLICATIONS INFORMED META-LEARNING
The aim of the experiments presented in section 5.1.3 is not only to validate the empirical effective- ness of our approach but also to illustrate its potential in practical, knowledge-rich domains where natural language insights may improve low-data learning. We can draw parallels between these experiments and other domains. To inspire future directions of applied research we have provided additional example applications of informed meta-learning in Table A.2
Table A.2: Example applications of informed meta-learning.
Domain
Task type
Data
Knowledge
Finance
Economic / stock
GDP over time, or daily stock prices
Descriptions trends c.g. "The stock prices for APPL are expected to con- tinue the upward trend".
market forecasting
Healthcare
Disease classification
Patient health
Expected relationships between symp- toms and diseases e.g., "High blood pressure is often linked cardiovascu- lar issues."
measurements
Medical Imaging
Disease detection in X-rays
Images of lungs
Relationships between abnormalities and diseases e.g.. "Presence of shadows upper lung areas may suggest tuber- culosis." "
Energy
Demand forecasting
Hourly electricity consumption data
Expectations based on events or trends e.g.. "Energy demand is expected to spike in winter months due heating needs."
Logistics
Delivery time estimation
Delivery routes, traffic patterns
Expected effects of traffic and weather c.g., "Deliveries are typically delayed during peak hours in urban areas." Expected shopping behaviours e.g.. "Sales tend to increase during holi- day seasons, especially for electronic goods."
Retail
Demand prediction
Sales data, time series
A.3 EXTENDED RELATED WORK
The main body of this paper discusses prior work that is related with respect to achieving the goal of knowledge integration. Here discuss related work from the perspective of the proposed method of informed meta-learning and the concrete instantiation of INPs.
Conditional generative models. The goal of deep generative models (DGMs) is to learn neural approximation of the distribution of the data p(æ) over a space X. most commonly the space of images. Popular DGMs include, VABs Kingma & Welling (2014), GANs Goodffllow al. (2014), and diffusion models Ho et al.] (2020): Song & Ermon 2019). Their conditional versions, e.g. CVAEs Sohn al. (2015), CGANs Mirza Mirza Osindero (2014), and conditional diffusion models Ho & Salimans (2022); Ramesh et al. (2022) model the conditional distribution p(z c). where is an additional conditioning variable, e.g. a class label text sequence. A similar analogy can be drawn between NPs and INPs which, as meta-learners, bring the idea of (conditional) generative modelling to the space of predictive functions f X + y. The goal of NPs is to model the prior distribution over functions p(f)); as well the posterior predictive distribution p(f De). Similarly to CVAEs and CGANs, INPs introduce an additional conditioning variable the expert knowledge, and model the conditional distribution p(f K),, guiding the prior the space of functions, such that the informed predictions, dictated by p(f Dc: K)) are concentrated around the region of functions agreeing with both the observed dataset Dc and the expert knowledge K.
24

---

Published as a conference paper at ICLR 2025
Multimodal deep learning. Mulitmodal deep learning refers to deep learning methods that can process and relate information from multiple modalities simultaneously, such as image, audio, and text. Our framework assumes that the datasets D and knowledge representations K. may belong to two different data modalities (e.g. 'D contains input-outpur pairs for 1-D regression and IC contains natural language description of the expected shape of the regression curve). This places informed meta-Icarning in the area of multimodal methods. What makes it distinct is that standard multimodal strategies (e.g. Ngiam et al. (2011): Srivastava & Salakhutdinov (2012); Ding & Tao (2015); (Shi et al. (2021b consider finding predictive function J. where X' is multimodal input space X 1 x... x XM. with each j E [M]] corresponding to a different data modality. In informed meta-learning, the learned functions f are typically unimodal, but the learning algorithm to fit each function is conditioned on the knowledge representation AC, belonging to different modality.
Zero and Few-shot learning. As presented in the experimental section, informed meta-learning en- ables sensible zero-shot predictions guided by expert knowledge. For instance, in multi-class image classification, IC may contain list of characteristic attributes of each class or class- wide descrip- tions in natural language. Seemingly similar ideas of utilising side information about each class for zero-shot learning have been explored in works of Al-HHalhh et al. (2016); Elhoseiny et al. (2017); 7); Paz-Argaman et al. (2020). In contrast to these methods, informed meta-learning does not focus on zero-shot learning only, but on the process of integrating external knowledge (e.g. knowledge about what are the characteristic features of each class) with observed few-shot or zero-shot (Dc = Ø) data sample. In the image classification domain, the idea of combining sample images with zero- shot attribute information was considered by Tsai & Salakhutdinov (2017) in application to one-shot learning, and extended by [Schönfeld et al. (2019) few-shot learning. None of these works, how- ever, consider the meta-learning setup of N-way, k-shot classification and require that the class attribute information is always present at training and test time, as it implicitly defines class labels. In our setup, the role of class information contained in I. lies in enhancing model performance by emphasising which visual features are most distinctive for given class, enabling zero-shot classifi- cation by-product. Contrary to SSchönfell et al. (2019), the additional information contained in IC is not necessary for few-shot predictions on previously unseen classes.
Meta-learning with meta-data. Several studies considered applying meta-learning to hierarchical datasets that include empirical data alongside corresponding metadata, such as feature names Twata & Kumagai (2022). task-specific parameters Kaddour et al. (2020), labels Tseng al. (2022). In these works, metadata is used in conjunction with empirical data as additional inputs defining task similarities. While these works share similarities in terms of the data structure and the resulting method, their motivations and goals are distinct. For instance, Kaddour et al. (2020) extend active learning principles to meta-learning, allowing the algorithm to select which task learn next based on the task metadata. HHuang et al. (2022): [Belbute-Peres et al.] I2: Kovacs et al (022). apply meta-learning to solve differential equations given a specific parametrisation. Another stream of work utilise s additional data to improve performance and generalisation of meta-learners: [Iwata] & Kumagai (2022) leverage feature descriptions, and Tseng et al. (2022) use multi-class agricul- tural labels. Denevi et al. (2020) 2022) study theoretically the conditions under which conditional meta-learning is advantageous to condconiiii meta-learning in the specific case of gradient- based meta-lcarning and with hypothesis space restricted to linear functions of the input. A recent work by [Ashman et al. (204)) conditions the learner on an additional set of datasets, which are known priori to have been generated by the same stochastic process as the context and target data. This approach aligns with informed meta-learning-it leverages the known relationship between the knowledge representation, e.g. set of extra datasets, and the underlying DGP of De. Our work generalises to arbitrary representations of knowledge, moving beyond structured numerical forms, with a particular focus on human- interpeattteeeellle representations, such as natural language.
In contrast to the existing works, the motivation of this paper is not on designing new meta- learning method suitable for hierarchically structured learning tasks, Instead, we view meta-learning as means to enable human experts inject their prior knowledge about a given learning problem into the learning algorithm, an intuitive and more flexible way than it is possible with conventional, model-based methods. Further discussion on the differences between informed meta-learning and meta-lcarning with hierarchically structured datasets are discussed in A.2.I
A.4 INP MODEL ARCHITECTURE
The architecture of INPs consists of the following key components:
25

---

Published as a conference paper at ICLR 2025
A data encoder, he.. (X' x 3 R that takes in pairs (I4,3a) and produces order-invariany representation r Σ, he. no (z,, yr).
A knowledge encoder, he.. map from the knowledge representation space to the latent space R2 that takes in the knowledge inputs K. and extracts latent knowledge vector k he (K).
An aggregator, a, that combines the data representation, r, and the latent knowledge rep- resentation, k into representation that parametrises the latent distribution q0,- We take qo. (= k) :- N (2; 02), where (Pz, o2) = a(r, k).
A conditional decoder, 99a. that takes in samples of the global latent variable 2 ~ qo, (2 r. k) and the new target location x to output the predictions parametrised by Po. x. 2) == (w;; Miz, Ox)) with (x 90, 2).
In all experiments any MLP is implemented with the GELU non-linearity Hendrycks Gimpel 2016). Number and dimensions of hidden layers are reported individually for each experiment in section A.6 We experiment with different forms aggregation, a:
1. sum & MLP: ar, k) = MLP(r + k).
2. concat & MIP: (r, k) = (LP((r])))
3. MLLP & FiLM: a(r, k) (iLM(A) [MLP())]. We use the idea of modulation parameters introduced by Perez et al. (2018). Here a an MLP whose parameters are modulated with modulated with the outputs of ha,,x- x-
We find that in most cases, the first, least complex option performs the best.
A.5
INP TRAINING
INPs trained in episodic fashion distribution of learning tasks consisting of context and target datasets, and associated knowledge representations. Denoting by re and rr the context and target data representations and by ie the knowledge embedding vector of single task, we derive the evidence lower bound via:
/ po. po.(urlEr, 2) (z rc, rr,k)dd k))dz
po(UT FT, rc, k))
(14)
40. (2 rc, k)) / po, xT,
qo rr,, k) qo (a PT, k)dz
(15)
pa, (pog(ur wr, 2) 90, (2
rc, k:)
Bo,, (clrT,k)
(16)
qo. (z
PT, k)
And therefore, by Jensen we obtain:
log PoTT IT,TC, k) 2 Egg, (=lrr.k) Pl, Pe, (UT (TT 2)] DKL. (go, (z (rr,k) k) || qa. (z rc, kc)
The parameters of the model are learned by maximising the above ELBO for randomly sampled batches of tasks. During training, we use one sample of go, (= rr, k) to form a MC estimate of the ELBO. For evaluation, we use samples. Additionally, during training, we randomly mask knowledge by setting h = 0, the frequency of masking is hyperparameter of the model.
A.6 EXPERIMENTAL DETAILS
Throughout all experiments we use the Adam optimise Kingma & Ba (2015). During training, validation-based early stopping. All experiments Epyc Milan 7713 CPU, 120GB RAM, and using a single NVIDIA A6000 Ada Generation GPU accelerator with 48GB VRAM.
machine with an AMD
A. 1-D SINUSOIDAL REGRESSION (SECTION 5.1
For each task, context and target data points sampled according the following process. A function if is sampled from the family of sinusoidal functions with trend and bias, f(x) = az + sin(bz) + We also introduce Gaussian observational noise, 34 f(x,) + 6 0.2. The parameters a, b, are randomly sampled according to: a ~ U[-1, 11, U[0, 6). c 11 For each task, the context and target points uniformly sampled from the range 22 The
26

---

Published as a conference paper at ICLR 2025
Table A.3: Average negative log-likelihood for different knowledge rypes. Results averaged across 500 testing tasks with standard errors of the mean provided in the brackets.
Ø
{a}
{6}
{c)
{a, b}
{6, cf
{a, c}
n
0
292.7 (225.0)
196.2 (173.4)
80.9 (122.8)
222. (230.6)
43.1 (70.0)
49.3 (85.8)
106.8 (96.0)
1
218.1 (218.3)
165.9 (170.9)
47.8 (94.7)
192.6 (203.1)
30.3 (57.1)
40.6 (78.4)
145.1 (149.7)
3
97.4 (139.1)
75.3 (113.0)
10.8 (43.7)
88.7 (146.5)
3.1 (26.4)
8.9 (38.6)
64.8 (94.5)
5
47.0 (110.3)
34.6 (82.3)
(27.4
45.8 (104.6)
(214)
-1.5 (26.5)
31.7 (77.9)
10
8.5 (52.2)
7.2 (49.6)
-9.1 (12.8)
7.9 (54.8)
(11.)
-9.1 (12.5)
4.5 (40.6)
15
2.2 (33.6)
(30.0)
(15.4
1.5 (33.3)
-9.99
-9.3 (14.4)
-0.3 (28.5)
number of context points n ranges uniformly between 0) and 10; the number of targets, m = 100. We let K. to encode the value of two, one, or none (KC 2 of the parameters a, b, or c.
The data encoder, h is implemented as a 3-layer MLP. The knowledge encoder, he.. is imple- mented with the DeepSet architecture Zaheer al. (017)) made of two 2-layer MLPs. Each element of the set is represented by one-hot encoding of the parameter type with its value appended at the 128 and use the sum & MLP
end. The decoder is 4-layer MLP. We the hidden dimension, method for the aggregator, We use learning rate of le-3 and set the batch size to 64. During training, knowledge is masked rate 0.3.
In section 5.1 we use this setup to demonstrate and discuss the impact of expert knowledge on enhanced data-efficiency, reduction in uncertainty, and robustness to distribution shifts. Fig. A. shows sample predictions under 0, 1, or 3 observed data points and different formats of knowledge K. Table A.3 provides a detailed summary the negative log-likelihood by knowledge type.
Format X
(c)
(b)
(0.0)
(a.c)
(b.c)
Number of context points points
Figure A.1: Sample predictions under varying formats of knowledge. Knowledge about the value of the slope or frequency of oscillations provides global information about the overall shape of the function. Observing additional data points anchors the curves in the xy-coordinate system. Based on qualitative investigation we conclude that the INP successfully learned how to integrate prior knowledge with observed data points.
A.6.2 INFORMED WEATHER PREDICTIONS (SECTION 5.2.1
We use the sub-hourly temperature dataset from the U.S. Climate Reference Network (USCRN The data contains values of the air temperature measured at regular 5-minute intervals. For each task, the context and target datasets consist of measurements from one day. Training, validation, and testing collections of tasks are created by randomly selecting 507, 108, and 110 days, respectively, between the years 2021 and 2022 in Aleknagik, Alaska. For each task, the target dataset consists of all 288 measurements in the 24h range. Context observations are sampled by first uniformly sampling 10 data points and then selecting the chronologically first n observations where n U00 10). We perform independent experiments with two formats of knowledge:
27

---

Published as a conference paper at ICLR 2025
A: For each task, knowledge K. is vector encoding two values: the minimum temperature and the maximum temperature on the day. In this setup, the knowledge encoder, he.,x. is a simple 2-layer MLP.
B: For each task, knowledge IC is synthetically generated "weather forecast" presented in natural language format. For illustrative purposes, these weather descriptions were generated with GPT-4 OpenAl et al. (2023). In total, 726 descriptions, one per day were generated. The prompt used contains instructions to generate sentences mimicking a weather forecast, based on 48 values sampled 30-minute intervals from the ground truth temperature values. We use the following prompt:
System: You are given vector of values representing the temperature for the next 24h at 30-minute intervals, starting at 12 am. Your task is to present the weather forecast according to these values. Keep it to max sentences. Use descriptive words to refer to the times of the day, c.g. morning, afternoon, evening,
User: <<Temperature values>>
There are two reasons for using L.M-geenratee weather forecast instead real ones: 1) availability of such data for research purposes not requiring web scraping; 2) the alignment of weather forecasts with the observed values available as our task-specific data. Real weather forecasts may contain high degree of noise due to the predicted temperatures diverging from the recorded values. To learn a faithful mapping between the knowledge space and the function space that aligns with our human understanding, in agreement with the underlying data generating process. To ensure that the output of GPT- 4 indeed aligns with the observed values, we have manually examined a random sample of the generated weather forecasts and found them to be of high quality.
require that information contained in the representations of knowledge
In this setup, the knowledge encoder ha., is implemented with RoBERTa language model Liu ett al. (2020) with all weights frozen except for the layer norm weights, which are tuned during obtained as pooled sentence
the end-to-end training. The latent knowledge representation embedding. Here, we use the last hidden state of the CLS token.
For both setups and B, the data encoder he n is implemented as a 3-layer MLP and the de- coder as 4-layer MLP. We used the MLP & FiLM aggregator a. We set the hidden dimension, 128. We use a learning rate of le-3 and set the batch size to 64. The knowledge representation is randomly masked at a rate 0.3 by setting k = O. Vanilla NPs are known to underfit context obser- vations and underestimate the variance, which became apparent with this more complex and noisy dataset. To mitigate this issue, in this experiment, during the encoding of the data representation. r. as proposed by Kim et al. (2019). Precisely, Di=1- Att, (z;; Wi), where Att MultiHead Q, K, V), with Q being matrix of target inputs, K a matrix of context inputs and V a matrix consisting of individual data representations ri ha..p @ (Ni, yi). We use attention heads.
have employed multi-head cross-attention
See the main body of the paper for a discussion of the results. Figure A.2 shows sample tasks and their corresponding GPTT4 generated weather descriptions.
A.6.3 FEW-SHOT AND ZERO-SHOT IMAGE CLASSIFICATION WITH -0U-200-2011 (SECTION 5.2.2
We apply our model to zero and few-shot classification using the CUB-200-2011 dataset Wah al. (2011) It contains 11.788 images of 200 subcategories belonging to birds. Following Akata et al. 20155) we use 100 bird categories for training. 50 for validation, and 50 for testing. We generate the labels for N-way classification tasks by choosing N random classes at each training step and arbitrarily assigning the labels O,..., N 1 to each. For each task, the number of shots k, i.c. the number of example images per class ranges uniformly between 0 and 10. The target set consists of 20 images per class. We perform independent experiments with three formats of knowledge:
A: Knowledge IC represents attributes characteristic for given class, e.g. wing span, feather color, shape of the beak. This is obtained by class-wide average of the binary attribute vectors from
28

---

Published as a conference paper at ICLR 2025
NP
(N)
INP (D)
The night stort off cold with temperatures folling to 9CC by late morning, and then gradually rise to high of 1.6 C in afternoon. Temperatures will drop ugain the evening, renching -3. ICC by midnight.
-10
The night will start off chilly with temperatures around 015%C, but will drop to -17 by early morning. The day will gradually warm reaching high of 5.1"C in the afternoon before cooling off to 1PC C midnight.
Temperatoon
The night will start off cold with temperatures falling to -16.8"C down, and the day will chilly 23. OPC by midnight. Afteroonn semperotures will hover around 18,5", bundle up if you've heading
to get colder, reaching
The night will be bitterly cold with temperatures around -88 degrees, gradwallyy increasing -14 degrees by late afternoon. The temperature will slightly drop again to -15 degrees in the evening, warning up -14 degrees midnight.
Time day
Figure A.2:
Sample predictions and the GPT-4 generated "weather forecasts" for setup
the original dataset associated with each image. Knowledge representations, IC, are constructed by stacking all N class attribute vectors into a N x 312 tensor. In this setup, the knowledge encoder, he is simple 2-layer MLP.
B: Knowledge K. represents the average per class, natural language descriptions of the N classes. These are obtained by averaging sentence embedding of individual image captions belonging to the given class. We use human-generated captions collected in Reed al. (2016) and encode them using CLIP embeddings Fu et al. (2022). Averaged per class text embeddings are then stacked to form N x dmodel where d mnodel MLP.
512. In this setup, the knowledge encoder, he., is 2-layer
C: We use GPT-4 to generate individual descriptions of each class based on the human- generated image captions. We present randomly sampled image captions pertaining to one class and prompt GPT-4 to generate short descriptions of features characteristic of the given bird breed. To generate the class descriptions, we use the following prompt format:
You given descriptions of bird breed. Based this information generate one comprehensive description of the bird breed. Keep it short and informative.
System:
User: <<list of randomly sampled image captions>>
In this setup, the knowledge encoder, he. is the CLIP text encoder. The embeddings of class descriptions are obtained the average of all outputs from the last layer of CLIP. After stacking them together in a N x model tensor, they are passed through a linear projection layer.
For all setups, A, B, and C, the data encoder, he_.p is implemented with a frozen CLIP vision model. followed by linear projection layer. Following the approach of Garnelo et al. (0)18)). we only aggregate over inputs of the same class. The aggregated class-specific representations are then concatenated form the final representation of size N We set d 512. We use the sum 2- layer MLP aggregation We modify the decoder return the logits of the categorical distribution. For a N- W-way task with class labels C1, cI,...,C0 ... we define p(y x, 2) as:
exp(-wF)
Pa. (y Cj 22 Ej exp( wf.a)
[W wr go, (2), Nxd
where x is a CLIP image embedding from the target set and go, is a 2-layer MLP. In experiments, the Hugging Face implementation of the CLIP ViT-B/32 model
29

---

Published as a conference paper at ICLR 2025
sit-base-pate pasee-atchh3). We use learning rate or le-4, batch size of 32 and knowledge is randomly masked at rate 0.5. For setups A and B, the INP model is trained end-to-end. For setups C, the weights of the INP model from the trained weights of the already trained, plain NP, and all model components, including the CLIP text encoder, are fine-tuned. As opposed to setup B, in setup C fine-tuning of the CLIP text encoder necessary ensure align- ment between the class-wide descriptions and image representations. Empirically, the two-stage training resulted in improved convergence.
For the empirical results and short discussion, refer to the main body of the paper. In Table A.4 we present sample Juman-generain captions (used in setup B) and their corresponding GPT-generated class descriptions (used in setup C).
A.7 UNCERTAINTY QUANTIFICATION WITH INPs
One particularly appealing property of Neural Process, which motivated their choice for the basis of our informed meta-learner, the ability to sample from the posterior instead of returning a single point estimate. This allows us to measure the reduction in model uncertainty given prior expert knowledge and/or observed data. We are mostly interested in measuring the epistemic, rather than aleatoric uncertainty.
Aleatoric uncertainty refers to the notion of randomness seen as the variability in the outcomes which is due to inherently random, unpredictable effects. As opposed to this, epistemic uncertainty refers to uncertainty caused by the lack of knowledge about the true relationship between model inputs and outputs. By observing data, or by inserting prior knowledge into the model, the epistemic uncertainty is reduced.
A natural choice for measuring the epistemic uncertainty would be the (conditional) entropy. By comparing H[)(f)) with H[p(f KC)] or H[p(f Dc)] we can measure the impact of prior expert knowledge observed data on the reduction in the epistemic uncertainty for single learning task (for notational convenience, we omit the dependence of on 8, making the expressions applicable both the ground truth DGP and the approximate po). However, in INPs, only have access to samples from the variational distribution and since the decoder is implemented as a neural network, evaluating the distribution over functions f is not possible directly. Instead, we need resolve to measure the uncertainty in the observation space. Thus, we are interested in computing
Hpp(y x. D), I KUD,CDDDDD0)
(17)
at a particular location x E X' in our input space, which can be then, for instance, averaged across uniformly distributed points in X. The quantity in (17) known the predictive uncertainty. To approximate (17) for an input x, we rely on Monte-Carlo estimation by sampling S functions based on our variational decoder.
Hypl ar [)] = - / ply x, 1) logply | a, TJdy
/ (ʃ plu x. syrs nar) 1og log (ʃ piu x., DP(s Tyas) dy
plu I xr, p(o))) dy - / Σ plu x, f(o)) log Σ
S
s
(18)
For each sample f(*), p(y I, f(v)) has a closed-fom expression-in the case of regression it is modelled with a normal distribution. Thus, (18) can be computed by numerically approximating the integral in the last line. Note that, since predictive uncertainty is measured in the observation space, also encompasses the uncertainty associated with the observational noise. Depeweg et al. (2018) suggest that (7) can be decomposed as:
Hip(y x, T)] T(y, f 1T, Eysperin [Fpp( a. f)]
(19)
epistomic
alentoric
The second part, Ey-pern () [py x, f)ll, is the average entropy when the predictive function is known, thus can be interpreted as the aleatoric uncertainty. If we model p(u x., f) with
30

---

Published as a conference paper at ICLR 2025
Table A.4: Example images, image captions and GPT-generated class descriptions.
Sample Images
Sample image captions
GPT-ggenerate class description
1. A large bird with a white belly, black and white wings with long beak.
This bird breed is medium to large size, characterised by its grey body feathers, contrasting white belly and face, black back and wings, distinctive black eye rings, and long, downward- curving peach bill.
2. This bird white and grey in colour with a curved beak, and black eye rings.
3. A large bird with a white belly and face, black back and wings, and peach bill.
4. Bird has gray body feathers, white breast feather, and long beak
5. A medium sized bird with black wings, and a bill that curves downwards
1. This big bird has shurp beak and hus black covering its body.
This bird breed is large and entirely black with sparse plumage, characterised thick brown beak, long tail, and wide rectrices relative to its body size.
2. An all black bind with distinct thick, rounded bill.
its
3. This entirely black bird has long and wide rectrices relative to the size of its body.
black bird with a long tail and large
beak.
This black bird has sparse plunage and a thick brown beak.
1. This goofy looking bird sports webbed feet and a bright orange bill, with piercing white eyes and dull coat of gray.
This bird breed is characterised by its black body, webbed feet, bright orange bill with in- verted feather curl the base, piercing white eyes with a dis- tinctive stripe, and a dull grey coat.
black bird with a small, orange beak and inverted feather curl at the base of the beak.
black body, white eye with stripe next to it, and an orange bill are on this bird.
4. This black bird has a orange bill with hair coming out of it, small pupils, and white line across its face.
5. This bird has wings that are black and has orange bill
1. This is black bird with white spotted belly and white eye.
This bird breed features black body with a white and black spotted underbelly, a white and grey speckled chest, black crown, bright white eyes with very small pupils, and short, pointed, black and orange bill.
This bird black with white and very short beak.
This bird has wings are black and white and has a small bill
This small bird white with black spots, a white neck, and black around its eyes.
5.
This is short stocky bird with webbed feet, it is mostly white with black wings and black specklex throughout.
31

---

Published as a conference paper at ICLR 2025
normal distribution, Hyp(y x, f.) has closed-form expression, 2 (x)(o()))Te), where o² (x))) is the variance at location 22.
The first part, 1(y, f 1), representing the information gain can be interpreted as the epistemic
uncertainty of interest. This quantity can be computed as the difference of H æ, f)] and By-o(fin [H[p( x, f)]], where both quantities are easy estimate, discussed above.
A.8
ADDITIONAL EXPERIMENTS
A.8.1 MODEL PERFORMANCE VS. NUMBER OF TRAINING TASKS
Setup: We follow the same setup as in the illustrative experiment from section 5.1.1 We cre- multiple training collection of tasks with varying number of total training tasks, Ntrain (25, 50, 75, 100, 200, 1000), with the upper limit being the number of tasks used in the original experiment. For each training collection of tasks, we train independent INP and NP models. The INP models receive information about two, one or none of the parameters a, b or с via knowledge representations, K. All models are validated and tested on the same collection of validation testing tasks.
Performance on training tasks
Performance on test tasks
A
50
Log Likeelooo
100
-150
200
2001
-250
madd
-900
NP
-35
3
10
3
10
Number context points
Figure A.3: Log likelihood of target data vs. number of context data points (higher is better). Comparison across varying number of all training tasks, Attrain Left model performance on training tasks, Right model performance on test tasks.
Figure A.3 shows the performance of all models on training (left) and testing (right) tasks. We observe that: 1) Both for the NP and INP models, as the number of training tasks decreases the performance gap between training and testing tasks increases. We note that this performance gap already at a (subjectively) reasonable level with only few 75 training tasks. 2) For all INP models trained with Aftrain 2 50 tasks we the additional knowledge presented for each task improves the performance over the plain, uninformed NP. When the number of training tasks is too small, here N Atrain = 25, we observe a "knowledge overfitting" effect. With insufficient number of training tasks the INP is unable appropriately capture the relationship between knowledge and empirical data, and thus fails to generalise to new, previously unseen tasks and their corresponding, also previously unseen, knowledge representations.
Take-away: We tested the robustness of the INP model to the reduction in the number of training tasks. We showed that in the experimental setup of section 5.1.1 adding external knowledge con- tinues to deliver noticeable performance gains over the uninformed NP when dropping from 1000 to as few as 50 training tasks. We also noted that with too few training tasks, the INP may fail to generalise. To prevent this effect from occurring in real-world deployment, we advise testing the model on held-out validation tasks and comparing its performance against an uninformed baseline, monitoring the knowledge overfitting effect. We note that in real-world applications, more training tasks can be obtained by (semi- )yynheecy data generation, augmenting the size of the meta-training set to improve model generalisation.
32

---

value Published as a conference paper at ICLR 2025
A.8.2 MODEL PERFORMANCE AND KNOWLEDGE COMPLEXITY
Setup. To assess the impact of knowledge complexity on the efficacy of learning the relationship between knowledge and the model hypothesis space, we again follow the same setup in section
5.1.1
We create multiple training collection of tasks with a varying number of total training tasks, Nerann E (25, 50, 75, 100, 200, 1000). J All models are validated and tested on the same collection of
validation testing tasks. For each setting of N train uninformed NP and 3 independent INP models with different knowledge representations used during training:
Make is a model where for each task its corresponding knowledge encodes, random, one of the three parameters, a, b, or c;
Mat is model where for each task its corresponding knowledge encodes, at random, one of the two parameters: a or (the value of e is never revealed);
MM is model where for each task its corresponding knowledge encodes the value of (the values of parameters a and e are never revealed).
constructed by one-hot encoding the type of the revealed parame- ter with its value appended at the end. We note that for the INP models M,, Mab. Mabes the complexity of knowledge representations gradually increases; the knowledge space is 2 and 3 di- mensional, respectively. We hypothesise that as the complexity of the knowledge space grows, more training tasks are needed to effectively learn the mapping from knowledge representations to prior distributions over functions. Given the same number of training tasks, the INP model Mas needs to learn how to disentangle the information about the function's 's oscillations (parameter b) from the information about the function's s slope (parameter a). Model Mabe additionally needs to discover the meaning of knowledge about the intercept (parameter c). Therefore, we expect that, given the same number of context points and the same information contained in K., the relative performance gains of the INP models Mas Mab. Mabe over the uniformed NP model should decrease as the complexity of knowledge space increases.
Knowledge representations
M₁
Ma
Mae
-50
25
75
Libelbood
-150
100
-00
1000
model
INP
-300
-300
10 35
D.
10
15
11
30
15
Number context points
Figure A.4: Log-Likelihood of target data vs. number of context data points (higher is better). Comparison across varying number of tasks used for training Nttran Complexity of knowledge space grows from left to right. All INP models are presented with the same knowledge about each the value of the parameter b.
Figure A.4 shows the log-likelihood of the target data evaluated on 500 testing tasks. For every INP model at test time we reveal the same information via knowledge representations- of the parameter b. Firstly, we observe the same two effects as experiment A.8.I With more training tasks, model performance improves. 2) An insufficient number of training tasks may lead to the "knowledge overfitting" effect; here Ntraan Secondly, we look at the performance gap between the INP and the NP (the gap between solid and dashed lines). We observe that as the complexity of the knowledge space grows (left to right) the performance gap between the INP and the NP decreases. This is summarised through the AAUC metric, presented in the Table A.3 From Figure A.4 we can also conclude that the more complex the the knowledge space is the more training tasks needed to effectively train an INP model. For instance, performance of the INP model M,, trained with Atrain = 75 tasks is comparable to the performance of the INP Mahe trained with Atrain 100 tasks.
25 the INP performs worse than the NP.
33

---

Published as a conference paper at ICLR 2025
Table A.5: Average relative performance improvement (%) between informed and uninformed pre- dictions. The performance gains become smaller the complexity of the knowledge space grows (top to bottom). Attran the number of training tasks used. INP performs better than the NP for all settings of Ntrein 2 50 indicating effective transfer between knowledge representations and functional priors. INP overfits with not enough training tasks, here at Vfrain 25.
Ntrain
1000
500
100
75
25
model
Ms
81.74
83.51
64.52
79.1
44.78
39.71
-8,52
Mab
65.3
67.45
43.46
54.66
22.41
6.67
-18.67
Mabc
71.76
57.51
2.02
26.46
9.48
8.05
-5.6
Take-away: The above experiment confirms our hypothesis about the complexity of the information conveyed in knowledge representations and the hardness of learning the mapping between knowl- edge representations and the model hypothesis space. As the complexity of knowledge increases, more training tasks are needed effectively learn relationship between knowledge representa- tions and the functional priors.
A.8.3 CORRELATION IN TRAINING DATA AND KNOWLEDGE DISENTANGLEMENT
Setup: For each task, context and target data points are sampled according to a similar process as in the experiments from section 5.1.1 A function f is sampled from the family of sinusoidal functions with linear trend. [(x = az sin sin(bz). As previously, we also introduce a Gaussian observational noise, s.t. y/ fxx,) + 6, e, (O, 0.2). In this experiment. simulate scenario in which the training data exhibits potentially spurious correlation. We sample the parameters a and 6 from a multivariate Gaussian,
2 2
~ M
We create 6 training and validation collection of tasks, one for each value of the covariance between a and b, a E {0.0, 0.3, 0.6, 0.9, 1.2) 1.4). We then train independent INP and NP models. For the let IC encode the value of one of the two parameters
INP models points n ranges uniformly between and 10; the number of targets is set to m 100. The testing collection of tasks is created by sampling functions where a and b are independent (i.e. This setup aims to test the robustness of the INP model to spurious correlations in the training data. We want investigate whether the INP model is able disentangle the meanings of parameters and b. In this setup, p(KC)f) remains the same at training at test time-it always provides partial information about the value of one of the parameters. It is the underlying process p(f) that changes between training and test time.
b. The number of context
0.0).
Results presented table A.6 show that when the correlation between the parameter a and b creases, the test-time performance of both the INP and NP models downgrades. This is due to the train-test distribution shift. Moreover, when the correlation is moderate (p I 0.64), the INP model outperforms mathces the performance of the NP. We note, however, that for p >= 0.42, the zero-shot predictions (n 0) are better for the uninformed model than the INP. This is also true for all values of n. at higher correlation levels (p 0.85). We hypothesize that this is because the INP has overfitted to the correlation between the parameters a and b. In the training dataset, revealing the information about the value of one parameter gives information about the value of the other, unrevealed parameter. INP exploits this dependency.
Take-away: INPs learn the meaning of knowledge based on its relationship with the empirical data. If this relationship changes at test time, good performance of the INP can no longer be guaranteed. This characteristic may be especially dangerous when there are spurious correlations in the dataset. The INP is prone to overfitting to these correlations, "misunderstanding" the true meaning of knowl- edge, and thus failing to generalise to new knowledge representations and their corresponding tasks, where the spurious correlations are no longer present.
A.8.4 META-LEARNED VS. EXACT KNOWLEDGE INTEGRATION
34

---

Published as a conference paper at ICLR 2025
Table A.6: Average log-likelihood on test tasks vs. correlation in training data (higher is better). the correlation coefficient between random parameters and b. π number of context data points per task. Model results for which the log likelihood is higher by statistically significant margin highlighted in bold. Values in brackets stand standard errors estimated with bootstrap.
P
0.00
0.21
0.42
model
INP
NP
INP
NP
INP
NP
0
-139.1 (10.2)
-209.2 (8.3)
-174.4 (13.7)
-196.5 (7.7)
-266.1 (19.6)
-221.7 (9.2)
-9..0 (10.6)
-102.1 (6.5)
-73.1 (6.2)
-120.4 (5.4)
995.0 (9.4)
-108.6
-16.9 (1.7)
-30.9 (2.6)
-34.8 (3.9)
-41.0 (3.4)
-29.2 (3.8)
-38.2 (3.4)
n = 5
-12.1 (2.0)
-11.9 (2.))
-15.7 (1.8)
-17.8 (2.6)
-12.4
-21.7 (2.9)
T 10
1.3 (L.)
1.84
-0.5 (1.2)
2.1 (0.9)
0.2 (0.9)
-2.5
15
3.5 (0.7)
(144).
0.8 (0.8)
(1.6
3.1 (0.6)
-2.2 (2.1)
P model
0.64
0.85
0.99
INP
NP
INP
NP
INP
NP
0
-356,4 (22.2)
-214.0 (9.2)
-795.6 (38.9)
-321.8 (14.8)
-1367.3 (63.6)
-410.7 (15.2)
-108.1 (7.1)
-160.3 (7.4)
-234.4 (9.3)
-200.0
-830.9 (40.9)
-527.3 (19.2)
n1
-26.2 (2.2)
(4.46
-149.0 (6.6)
-118.6 (6.3)
-551.5 (32.5)
-360.1 (10.6)
n
18.3 (2.0)
(3.03
-101.4 (4.9)
.9..0
404.2 (13.2)
-319,9 (9.0)
10
-6.2 (1.2)
(2.6)
-81.2 (4.4)
-70.6
-342.7 (1L.1)
-324.9 (9.3)
15
-4.3 (1.2)
(2.6)
-74.1 (4.3)
-66.2 (4.3)
-332.2
-313.0
The primary advantage informed meta-learning lies in its universality and applicability to scenarios where model-based approaches to knowledge integration are infeasible or require significant human efforts. Unlike informed leaanne,,,,ng, model-based approaches require knowledge representations to be easily translatable into explicit inductive biases, such as functional representations, parameter priors, or regularisers for the loss function. On the other hand, model-based approach may guarantee the correctness of knowledge integration by design instead of relying on a neural approximation learned based on finite number of meta-training tasks. However, the universality of informed meta-learning necessarily comes with trade-off in performance. In this section we investi- gate the efficacy of the data-driven approach of informed meta- learning knowledge integration in comparison based approach-fitting an explicit parametric model encoding the available task-specific knowledge exactly.
Negative Log-Liko Likallhood
100
-150
model_name
INP
200
Bayes Reg.
knowledge 2
-200
K.
K.
300
3
5
10
Number of Context Points
model-
Figure A.5: Knowledge integration via the INP vs. exact inference with Bayesian regression.
Setup. We again follow the setup of section 5.I with sinusoidal functions. We compare the INP and NP models, against an exact Bayesian regression model with az + sin(bæ) ) + c and we put priors on the parameters a, b, and that correspond precisely to the ground-truth process p(.f) used in your setup, i.e. ~ U[-1, 1], b U00 6), U1-1, 1]. We fix a to its ground-truth value of 0.2. For given task with knowledge A. about the value of one or b, we fix the value of the respective parameters and only estimate the posteriors (()) for the remaining, unknown quantities. Given the non-linear transformation of the parameter through the sine function, find the posterior of the parameters a, b, with MCMC estimation with the NUTS algorithm. For each task, we run 4 chains with 1000 burn-in samples and 2000 samples used for estimation.
observations modelled according to: p(y|=) N Nyy; (();, o²), where μ(z)
of the parameters a,
Results. Figure A.5 compares the performance of predictions made based according to an exact regression model and the meta-learned predictive posterior with INPs. We first note that already for the the uninformed case, K = 2, the INP performs worse than the Bayesian regression model, specifically when data becomes available. This is expected, as the regression model is
35

---

Published as a conference paper at ICLR 2025
plicitly based the ground-trun -ruth data generating process Meanwhile, the Neural Process only ap- proximates this distribution over sample functions by learning on finite collection of meta-training tasks and is also known underfit contextual observations. In terms of knowledge integration, observe that even thought the informed predictions of the INP significantly improve upon the non- informed INP (which is effectively the plain NP), there is still noticeable gap between the INP and the thcoretically achievable upper bound dictated by the informed Bayes regression. It is also worth comparing the inference times of Bayes regression with MCMC estimation vs. that of the (I)NPs. After training, predictions with (I)NPs simplify to forward pass through the trained neural network, which for single task in this instance takes less than 0.004 seconds on a machine with the AMD Epyc Milan 7713 CPU, 120GB RAM and the NVIDIA 48GB A6000 GPU. In comparison, estimating the parameters of the regression model with MCMC took us on average 8.6s per task. For reference, training an INP model in this instance takes around 20min.
Take-away. The major advantage of informed meta-learning its applicability across diverse knowledge and data formats, as illustrated with the real-world and natural language data experi- ments section 5.2, However, as demonstrated with the above experiment, this universality comes with trade-offss in performance. When model-based approach is available, it should be preferred, as it may guarantee the correctness and exactness of knowledge integration by design instead of relying neural approximation learned based appeal of informed meta-learning becomes evident when the underlying functional form of the data generating processes not known explicitly and the task-specific knowledge difficult represent in form of an exact mathematical equation or a constraint. In such cases, model-bate approaches knowledge integration become challenging if not infeasible.
finite number of meta-training tasks. The
36


Reviews:
[{"review_id": "H4RqlmlShq", "reviewer": "Reviewer_EZNr", "summary": "The paper provides a theoretical framework around conditional meta-learning, oriented toward conditioning neural processes on natural language info about temporal processes.", "questions": "Detailed comments by section:\n\n- I would add \"particularly\" to the first sentence of the abs\n - almost very nice first fig, but caption could be improved (what are the different indices on Ks and how do they relate to K\n\n2.  \n- nice concrete examples of K in the problem setting\n- not sure exactly how to phrase this, but it seems worth mentioning how strong the assumption of an underlying stochastic process is. e.g. in the nice explanatory figure, in the observable data space you have a sine and a line, and these are clearly different, but what we actually have in real life is often a bunch of high-D points that you could fit a sine or a line to with equal error and it doesn't mean that either of them is properly the \"underlying\" function. I'm not trying to say this assumption isn't fine to make, but I think it's worth emphasizing how abstracted it is from \"real\" data\n  - similarly for the assumption of conditional independence of K and D that is necessary for the proof; it seems to me like it's a lot of math to say something pretty obvious; that if we add new necessarily true and relevant information that will improve things ... it would be nice to give an idea of how much the proof relies on those properties.\n\n3. \n - nice concrete examples but man the formatting is hard to read; please make it not all in italics!\n - the remarks section is a bit random and take up a lot of room relative to what they provide; suggest moving to appendix to make more room for easier to read formatting and extra concrete examples/explanations of experimental results and connections from theory to experiments.\n\n4 \n- the arrangement of the sections makes sense, but it does mean all the theory and acronyms about neural processes are really far from the empirical results; it would be nice to tie them together better with cross-references\n- unclear how the aggregator operator works\n\n5 \n- definitely nice to have the empriical results but because of the synthetic setting it's hard to take anything from the results without the discussion being more concrete -- it would be nice to continue the concrete example pattern you had from the theory section (e.g. for the dist shift experiments, what was the K you provided?). \n - why not use real weather forecasts?? introducing a LLM adds an unnecessary layer of complexity and potential for hallucinations \n- again unclear how the K is actually informing the NP -- are you encoding it with a pretrained LLM and appending that to the input of the neural process? adding? something else? these details are the technical core of the paper and it's important to make them clear.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Nice concrete examples through the theory; good explanations and clear notation.", "weaknesses": "Overall an interesting and well-written paper, but  seems like it's maybe missing references to conditional metalearning, and doesn't compare to the baselines it discusses early on. The theory seems like a lot of math to say things that are kind of vacuous, but such is math, and it's well explained with nice concrete examples. The biggest criticism I have is that the way K is given to the model, concretely, is very unclear, and in my experience (conditioning physics-informed neural nets, which come to think of it is another area that should be cited as relevant lit) this kind of 'technical detail' makes a huge practical difference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KQGNDWdSKz", "reviewer": "Reviewer_66xM", "summary": "The work proposes a viewpoint on how to integrate prior knowledge into the learning process via informed meta-learning. The idea is to incorporate knowledge that is conditioned on the generative function, but is not a unique representation of the function. The meta-learning algorithm can then be applied to incorporate this knowledge. Neural Processes are used for empirical studies of the approach, demonstrating the advantages of prior knowledge in three specific cases.", "questions": "For the MAML Example 1, how similarity between tasks is measured?\n\nCould you please comment of how number of shorts or context points should be chosen while using the framework?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper can potentially be applied to a lot of domains, where the expert knowledge and priors are presented.\n\nThe paper is overall clearly and well-written.\n\nThe proposed viewpoint is clearly illustrated based on a set of examples.", "weaknesses": "One of the core ideas of the paper’s perspective is to use prior knowledge K to guide the learning system. To assess how applicable this viewpoint is to different problems and domains, more discussion is needed on how to construct K for different scenarios, as the examples provided in the paper are very specific and not necessarily realistic. More discussion is also needed on how the properties of K influence the viewpoint. For example, in 5.1.1, more knowledge corresponds to a stronger prior. Can this be formalized for a general viewpoint?\n\nCould you please comment on whether it is possible to have a stronger statement in Theorem 1 with a strict inequality?\n\nMinor:\n\nLine 245: \"mete-training\" -> \"meta-training\"\n\nPlease name the axes in Figure 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TooamIVpA7", "reviewer": "Reviewer_gpLz", "summary": "This paper studies a meta-learning setup in which a learner tries to adapt to a new environment given only a few samples. \n\nThe paper's main idea is to endow the learner with additional domain knowledge about the specific task (\\mathcal{K} in the paper).\n\nThe key research question is whether, in the era of LLMs, we can provide the learner with human-interpretable domain knowledge in the form of text, e.g. the human provides the learner with the information that the function \"is monotonically decreasing\". \n\nThe paper first develops a formal framework for meta-learning with additional information (Section 3).\n\nThe paper then provides different empirical applications of the new paradigm using Neural Processes (Section 5).\n\nThe paper states that *\"our work primarily focuses on putting forward the idea of informed meta-learning with the introduced\nclass of INPs serving mainly as an illustration.\"*\n\nOverall, this is an interesting and well-written paper with limited empirical evaluation (which makes the currently submitted paper a marginal paper in my view).\n\nI'm not very familiar with the meta-learning literature, and I decided to give this paper the benefit of the doubt with a score of 6.", "questions": "**Question 1:** Figure 3 contains nice examples, e.g. K = \"This function is sharply decreasing.\" However, there seems to be no exmpirical application with K = natural language for a regression problem in the paper. Why not? Did the authors not try this, or did they try it, and it did not work? I would be quite interested to see if this can work well in practice. \n\n**Question 2:** Figure 4(b): I do not understand how the plot value for {a,b} can be in between {a} and {b}? should {a,b} not be strictly better than {a} and {b}?\n\n**Question 3:** What exactly are the research contributions in Section 3? To me, this Section mostly reads like a summarization of different possible approaches, laying out the strategy the authors want to employ in the paper. \n\n**Question 4:** Concerning the general motivation, it is a bit strange to read some of this in 2024. For example, from the introduction: *\"with the knowledge integration step often forming the core contribution of many ML papers\"*. Do the authors actually believe that \"the knowledge integration step\" will be a core part of many ICLR'25 papers?\n\n**References:** Some of the references need updating. e.g. \n\n\"Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic Model-Agnostic Meta-Learning, 2019.\neprint: 1806.02817.\"\n\nwas published at neurips 2018.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I find the research question quite interesting. It connects a classic topic in machine learning research—how to integrate human domain knowledge into the algorithm's inductive bias—with recent developments (LLMs). \n\nThe paper is well-written, and the appendix documents the different experiments in detail. The paper illustrates key ideas and results with well-designed figures. \n\nThe paper's theory seems sound (though I did not check the details), and the experiments are well-designed. \n\nOverall, this paper makes some interesting suggestions for how novel developments in LLMs can inform existing work in knowledge integration / meta-learning.", "weaknesses": "The paper's main weakness is the empirical applications. Specifically, it does not provide a real-world application in which providing additional information in natural language improves upon other approaches (running somewhat contrary to the motivation as outlined, for example, in Figure 3).\n\nThe paper is aware of this and states that *\"The aim of our work is not to present a new method that surpasses existing baselines on a benchmark dataset; rather, we propose a new viewpoint on meta-learning as a means of establishing an interface\nbetween human domain knowledge and the hypothesis space of ML models.\"*\n\nHowever, I'm uncertain how interested the ICLR audience will be in the methodological approach alone.\n\n**Section 5.1:** The experiments in section 5.1 are all for the synthetic function y_i = ax_i + sin(b x_i) + c + epsilon_i. For this reason, the results in this section (while convincing) read like a case study, not a systematic assessment of the pro's and con's of the proposed method.\n\n**Missing baselines:** I find the usefulness of the proposed method hard to gauge because the paper does not compare it with other existing methods. For example, the authors mention that Bayesian linear regression would be a natural method to use in the setup of Section 5.1. Then why does the paper not compare performance with Bayesian linear regression?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)