A Metalearned Neural Circuit for Nonparametric Bayesian Inference
Authors: Jake Snell
Gianluca Bencomo
Thomas L. Griffiths
Venue: NeurIPS 2024 poster (2024)
Keywords: Nonparametric Bayes
metalearning
amortized inference
Primary area: probabilistic_methods
TL;DR: We introduce a metalearned neural network that captures the inductive bias of nonparametric Bayesian models.

Abstract: Most applications of machine learning to classification assume a closed set of balanced classes. This is at odds with the real world, where class occurrence statistics often follow a long-tailed power-law distribution and it is unlikely that all classes are seen in a single sample. Nonparametric Bayesian models naturally capture this phenomenon, but have significant practical barriers to widespread adoption, namely implementation complexity and computational inefficiency. To address this, we present a method for extracting the inductive bias from a nonparametric Bayesian model and transferring it to an artificial neural network. By simulating data with a nonparametric Bayesian prior, we can metalearn a sequence model that performs inference over an unlimited set of classes. After training, this "neural circuit" has distilled the corresponding inductive bias and can successfully perform sequential inference over an open set of classes. Our experimental results show that the metalearned neural circuit achieves comparable or better performance than particle filter-based methods for inference in these models while being faster and simpler to use than methods that explicitly incorporate Bayesian nonparametric inference.

A Metalearned Neural Circuit for Nonparametric Bayesian Inference
Jake C. Snell
Gianluca M. Bencomo
Department of Computer Science Princeton University jsnell memeel@@rrnneeoon.o edu
Department of Computer Science
Princeton University
gb5433@princeton.edd edu
Thomas L. Griffiths
Department of Computer Science
Department of Psychology
Princeton University
tomgoprinceton. edu
Abstract
Most applications of machine learning to classification assume a closed set of bal- anced classes. This is at odds with the real world, where class occurrence statistics often follow a long taig-tailed power-law distribution, rarely revealing the entire prob- lem domain in single sample. Nonparametric Bayesian models naturally capture this phenomenon, but have significant practical barriers to widespread adoption, namely implementation complexity and computational inefficiency. To address this, we present a method for extracting the inductive bias from a nonparametric Bayesian model and transferring it artificial neural network. By simulating data with a nonparametric Bayesian prior, we can metalearn sequence model that performs inference over an unlimited set of classes. After training, this "neural circuit" has distilled the corresponding inductive bias and can successfully perform sequential inference that the metalearned neural circuit achieves comparable or better performance methods that explicitly perform Bayesian nonparametric inference while being faster and simpler to use.
open of classes. Our experimental results show
than particle
Introduction
Standard machine learning approaches to classification assume that the set of possible classes is known priori. Classification in this setting thus involves selecting the most appropriate class label from closed set. However, this is not the case for human learners. Imagine European explorers in Australia seeing kangaroo for the first time. Rather than trying classify this observation into an existing class - is a deer or rabbit? - they recognized that new class needs to be created. Although this easy for humans, current machine learning systems struggle identify novel classes and use them in predictions [46]
Bayesian statistics offers elegant solution to the problem of novel classes: define model in way that does not make a commitment to an upper bound on the number of classes. This idea is expressed in nonparametric Bayesian models, namely the Dirichlet process mixture model (DPMM) 3. In this model, a new data point assumed to be generated from an existing class with probability proportional to the number of previous observations from that class and from new class with probability proportional to a, a hyperparameter of the model. This makes it possible to both postulate
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1. Meta- Train
2. Attach
Dirichlet Process Mixture Model Data
Recurrent Neural Network
Nonparametric Inference Network
Our proposed nonparametric inference network first internalizes the desired nonparametric Bayesian prior via metalearning recurrent neural network (RNN) to model its posterior distribution over class assignments. Afterwards, the metalearned RNN, or neural circuit, has captured the corresponding inductive bias and can be used perform sequential inference over potentially unbounded number classes.
Figure 1:
that a new datapoint might come from new class and capture long-taiie distributions of class frequency commonly found in real-world classification problems.
Despite their elegance, nonparametric Bayesian models have fallen out of favor in machine learning, as they are difficult to reconcile with the current focus on large-scale models defined over complex objects such as images. The scalability of non-parametrie Bayesian models is limited due to the high computational cost of Bayesian inference, typically requiring the use of sampling algorithms such as Markov chain Monte Carlo requires creativity in defining generative models that are sufficiently expressive without making inference intractable. Furthermore, most standard Bayesian inference algorithms are not designed to perform sequential inference, instead assuming that all data are presented in single batch.
44] particle filters [X Applying these models to complex objects
In this paper we pursue different approach inference in nonparametric Bayesian models: training a recurrent neural network (RNN) to approximate the posterior distribution over classes from a of metalearning, repeatedly sampling sequence of
DPMM. We formulate this problem class memberships and observations from the DPMM and training the model to predict the class of each observation conditioned on the class labels of those preceding it. The resulting neural circuit (Figure T internalizes the inductive bias of the nonparametric Bayesian model used to generate the training tasks. It can then be used component in deep neural networks, extending nonparametric Bayesian methods to complex objects such as images.
Our approach combines the elegance of Bayesian nonparametric models with the predictive power of deep learning in principled and practical manner. The discriminative nature of the RNN allows it to classify complex inputs without making the restrictive distributional assumptions required by standard Bayesian inference algorithms. Since RNNs are sequence models, the neural circuit can efficiently make predictions for sequence of observations each successive predictive distribution is computed in constant time. We apply our neural circuit to a challenging open-set image classification tasks from the ImageNet [51] and iNaturalist [6]] datasets and show it achieves excellent predictive performance at a fraction of the computational cost of standard DPMM inference algorithms.
2 Background
Consider the task of learner classifying objects new environment. The learner encounters After item x, is observed, the learner attempts to predict its class label. The learner is then presented with the true class label 2,, updates internal representations as necessary, and repeats the procedure for timestep & + 1. The predictive distribution over the entire sequence labels 21.7 = Z1...., 2T can be written as
sequence of items X1, x.,.... Xy.
T p(Z1 NxT) = p(21 x1) II p(2 (111)) 2=2
(1)
We first review Dirichlet process mixture models, which provide a framework for expressing posterior inference over #1.7 without restricting the labels to belong to closed set. We then

---

review the particle filter of Fearnhead [23], which provides method for sequentially computing the predictive distribution through the use of weighted particles representing assignments of class labels.
2.1 Dirichlet Process Mixture Models
The nonparametric Bayesian solution to the prediction task presented above involves infinite mixture models, the most notable of which is the Dirichlet process mixture model (DPMM) 3 22 47 There are two components to the DPMM: a distribution over class memberships and a class-conditional distribution over observations. In this model, observations the distribution over classes places no limit on the number of classes. The joint probability follows a Markov process &1:T and assumes fass-conditional independence for X1.7:
generated from latent classes, where
p p(z))p(x1 II
p(= 1( 2=
The conditional distribution on class memberships p p==((21e-1) relatively simple:
x {
n previously observed
p( k 11
k new class,
where n& denotes the number of occurrences of class k in 21...... 1. and by convention the value of l: for new class is taken to be one greater than the number of classes observed so far. This process is known as the Chinese restaurant process (CRP) [1.
The conditional distribution over observations is p(x; 2 k) gxx((pxx). where g-1lp) is some probability function with parameter ş. Each ęP: is in turn distributed according to a shared prior (4px) for k 1, 2, .... The predictive distribution over class assignments (1) can be derived by
simple application of Bayes' rule to the joint distribution (2)
Direct computation of the posterior in DPMM is intractable, and therefore several methods have been developed to perform approximate inference using methods such as MCMC (44) and varia- tional inference [9] However, these methods typically assume that all observations are presented simultaneously and do not attempt to handle the sequential nature inherent to our problem formulation.
2.2 Particle Filter for the DPMM
One notable method that does aim to perform sequential inference of class labels in the DPMM is the particle filter proposed by Fearnhead [23.. Particle filters maintain a of weighted particles each timestep that approximate the posterior distribution over latent variables [14] 19]. At each timestep, a particle filter propagates a set of partieles forward in time by first sampling a new set of states from a transition distribution and then assigning weights according to a potential function chosen such that the set of weighted particles approximates the target posterior distribution.
In order to make inference tractable in the DPMM setting, Fearnhead (3 assumes that gx((p))
belongs to the exponential family and T(p)) is the corresponding conjugate prior. In this case, is possible to marginalize over when computing the posterior predictive distribution for a class. Suppose that x; (xx1(4) for i 1,, .1.,...... , 1. Since the posterior p(yp|zr (x...) has the same form as the conjugate prior x (sp),, there exists a closed form expression for the posterior predictive distribution
p(x,+ (Xim)
p(x.+ (()ppp)))((()))))))))))))))))))) dyp.
represents an entire trajectory of class labels up to the current timestep. Suppose that after timesteps there are J/ particles =0) each with weight wy 2 0 for j 1, 1.....JJ J such that Σ Wj- 1. For each particle, the transition distribution is:
Each particle in the method of Fearnhead 23
p(=+11
X1æ+1, (1)) p(z++1 k|=(0) ((x | 200, 2++1 k),
(5)
where the likelihood term p(x++1 X11, 2 2++1 = k) can be computed in closed form according to If k represents a new class, this will be the prior predictive probability instead.
()
Two major shortcomings of the particle filter are that it makes restrictive distributional assumptions (the class conditional distribution needs to be exponential family) and requires many particles in order to sufficiently approximate the posterior. the next section, we present our metalearned neural circuit which is aimed at addressing these issues.
3

---

3 Metalearning a Neural Circuit
We propose a novel amortized inference approach for class inference in the DPMM, based on metalearning recurrent neural network (RNN) to predict class memberships. By applying met- alearning tasks defined by sampling data from a DPMM, the RNN can effectively internalize the corresponding inductive bias in reusable neural network, hence the name neural circuit.
Our approach inspired in part by the observation that the updates of the particle filter in Section 2.2 can be implemented by accumulating sufficient statistics of observations. Recall that an exponential family distribution and its conjugate prior can be expressed in terms of natural parameters ny:
p(x m) h(x) exp (77 t txx) A(n))
(6)
p T, 1/) = exp [(T, 7)) - =A(n) - B(T,))}
(7)
where t(x) are the sufficient statistics and A(n) is the log normalizer. The posterior after observing Xin is of the same form as the prior, namely p(n T where
t(x,)
v' =v=
(8)
From this perspective, the particle filter can be implemented by first initializing the representation of each class to be T. u. Then the sufficient statistics t(x,) for each observation are extracted and used update the corresponding class's after the true label revealed.
An analogous computation is carried out by a recurrent network, which given some input x and previous hidden state h computes an updated hidden representation h, and an output u,:
h; U, +- RNN hę-1)
The differences with respect to the particle filter are twofold: the representation of each cluster is no longer separate but shared in h, and the representations are learnable end-to-end.
We also recognize that the output Ug can be the basis for predicting the current class label, and h, can be made to capture the updated current state of all classes simultaneously. To predict the current class label, the RNN's output u, is mapped to a provisional logit a, using a learnable weight matrix W and bias vector b. The logits then additively masked by m,, which preserves the logit long as predicting the corresponding class would be valid (i.e. the class label is at most one greater than any previously seen class label). The input to the RNN each timestep is the concatenation of the current observation X, and a one-hot representation of the previous label zt-1 (all zeros when = 1). The predictive distribution therefore defined to be:
Py (2e X1,t 21:-1) SOFTMAX(a; m:)
(10)
a, = Wu, +b
k < + max 21:t-1 otherwise,
mik DO
11 h < RNNo([X,, ON hz-))
where y A (0, W, are learnable parameters.
In order learn these weights, we turn metalearning. In metalcarning, a system is presented with set of tasks sampled from a distribution over tasks. The goal is to leverage the shared structure of these tasks not only to become better at solving each individual task but also to solve future tasks better, effectively "learning to learn" g 30 56]. This is done by estimating set of hyperparameters shared across tasks. In our case, these are the weights of the neural circuit, which we metalearn by minimizing negative log- likelihood of item-label sequences generated by a DPMM.
Let 21. be generated according CRP (3) and X, 2t k be generated according to some (possibly unknown) distribution (x. (pxx) for 1, T. (21T, X1:T). We define
Let 'D denote this joint distribution over
>" arg min Xir) (
log Pay (xt(x1
(1))
T

---

where the minimization is performed via gradient-based optimization procedure. This metalearning approach can be applied equally well whether the the form of the class-conditional distributions known or not, since all that is required are samples from the joint distribution. If the class-conditional distribution unknown, taken to be the empirical distribution constructed from dataset by placing an evenly-weighted point mass on each sample within a class.
Relationship to Amortized Inference Our method can be viewed as form of amortized inference, wherein a function (e-g. represented by deep neural network) is learned to directly map from inputs to an approximate posterior distribution over latent variables [5 49] Amortized inference is similar to discriminative classification in that it directly maps from inputs to class labels, but it can also be viewed as approximate Bayesian inference within the generative modeling framework. Networks trained to perform amortized inference are sensitive to the choice of prior, since different priors will lead to different posteriors, the KL-divergence to which will be minimized during learning. In similar fashion, our method is also affected by the choice of DPMM prior as different priors change the distribution over sequence that the neural circuit is trained on.
Relationship to Metalearning A variety of methods have been used to learn shared hyperparam- eters to solve a set of related tasks problem for deep neural networks [2] [24] 48 [5]] 69], but our approach is most similar to those in which recurrent neural network is trained perform multiple
tasks 200 65]. This approach is more typically used in reinforcement learning, defining a system that learns a global meta-policy that supports efficient learning on specific tasks. The resulting RNNs have been shown to encode information equivalent to a Bayesian posterior distribution [40] making
them good candidate for metalearning amortized Bayesian inference.
4 Experiments
We apply the neural circuit on three data settings: a synthetic dataset where the form of the DPMM is known, sequences of labels generated from CRP on ImageNet [51], and sequences sampled directly from the long-tailed iNaturalist 2018 species classification dataset [6]] The goal of our experiments is compare both the predictive performance and computational efficiency of the neural circuit to standard sequential inference techniques for the DPMM.
Our main experimental point of comparison is the particle filter of Fearnhead 233 discussed in Section 2.2 For a non-Bayesian baseline, we compare to a softmax- classifier that augments the distribution over K known classes at time Ê with an additional logit representing the possibility of new class. The logit representing the new class is derived using the maximum entropy principle [33] by introducing the constraint that the marginal probability that 2, is new class be equal to a (t-1)++ (please refer to Appendix B for more details). We refer to this baseline as "Softmax + Energy."
We consider two main evaluation scenarios. The first, referred the sequential observation setting, measures the quality of model's predictions when the true class label is provided after each prediction. We report the average predictive negative log-likelihood (NLL) averaged across timesteps and corresponding perplexity. The second setting quantifies how well the model predicts when there is feedback about the true class labels (we refer to this as the fully unobserved setting). We compare the maximum a posteriori (MAP) prediction of the model against the true sequence of class labels using standard clustering metrics, including the adjusted Rand index (ARI) [1] and adjusted mutual information score (AMI) [62]. In both settings, we compare computational efficiency by evaluating the wall clock compute time per sequence to make predictions. Please refer to Appendix D for additional experimental details.
4.1 Modeling Synthetic Data from a DPMM
We first evaluate performance of the neural circuit on a synthetically generated DPMM dataset. Our aim is determine whether the neural circuit can match performance of the particle filter when the class-conditional distributions belong to the exponential family with a known prior. Specifically, we use ceamaaliiveere-gammme prior and Gaussian class conditional distributions with unknown mean and variance. For each dimension d = 1,..., D, the form of the class-conditional distribution is:
0 1-11 1 (a,
pa - N(m,0/////1 Id NN oz),
5

---

where X, a, and b are known hyperparametees. We set the length of the sequences to T = 100 and chose D m 0. A 0.01, and a = 2. Several sequences drawn from this distribution are We directly ser the hyperparameters of the particle filter to their true values.
shown in Figure 2
60-
20
-20
0-
Figure 2: Visualization of sample synthetic sequences generated from the normal-inverse prrsoe--rrorroooo used in Section 4.1 Classes are sampled from Chinese restaurant process (CRP) with 1.0 and sequences consist 100 timesteps. Clusters colored by true class label.
The results of our evaluation can be found in Table I The Softmax + Energy baseline is significantly slower than the other methods due to solving for weights after each timestep. Although its NLL is better than the CRP, the average perplexity is worse due to the presence of some sequences where NLL quite poor. These sequences are accentuated due to exponentiation in the formula for computing perplexity. We find that although the particle filter performs best negative log-likelihood, the neural circuit provides better clusterings in the fully unobserved setting. We hypothesize this is due to the particle filter's insufficient exploration of the posterior over labelings stemming from its reliance on particles. We also find that neural circuit inference roughly 5x faster than the particle filter in the sequential observation setting and roughly 10x faster in the fully unobserved setting.
Table 1: Results two-dimensional data synthesized from DPMM. Evaluation computed as average over 10,000 held-out sequences of length 100. Negative log-likelihoods are expressed in nats per timestep.
Seq. Observation
Fully Unobserved
Inference Time (ms/seq.)
Method
NLL (L)
Perp. (1)
ARI (t)
AMI (t)
Seq. Obs. (L)
Fully Unobs. (1)
CRP
1.006
2.978
0.010
0.010
0.019
0.179
Softmax + Energy
0.929
24.742
0.388
0.392
1679.716
1691.403
Particle Filter
0.048
1.053
0.769
0.814
1.617
4.432
Neural Circuit
0.076
1.086
0.921
0.928
0.059
0.421
4.2 Open-set Classification on ImageNet-CRP
Next we consider a challenging open-set image classification task where the input features are activations from ResNet (29). Our goal is to determine whether the neural circuit can effectively scale up to high dimensional space where the form of the class-conditional distribution is unknown.
We downloaded the weights of pretrained ResNet-18 from TIMM [66] and extracted the 512- dimensional penultimate layer activations from the entire ILSVRC 2012 dataset [51] We split the 1,000 classes into 500 reserved for training imeta-train classes) and 500 for testing (meta-test classes). We generate sequences by sampling N from CRP with = 1.0, assigning each distinct value of zn class uniformly random, and then sampling the corresponding observation x, uniformly from the images belonging to that class. We call this data-generating procedure ImageNet-CRP.
We metalearn the neural circuit with the same architecture and optimization procedure in Sec- tion 4.T For the particle filter, additional care needs to taken to model the sparse nonnegative ResNet features that are produced by a ReLU activation. We modeled this using hurdle [16] model with log- normall distribution over nonegative values. This model posits a log- normal distribution over the nonnegative values and point mass at zero:
Ed Fz 3d
Zd LogNormall(u 0Z)
Dd Bern(pa)
(12)

---

1. Sample class sccording
1:4
Sample image without replacement
Image Dataset
CRP Sequences
Diagrammatic representation of generating Chinese restaurant process (CRP) using image data. At every sequence step a class is sampled according to a CRP and then an image from that class is sampled without replacement.
Figure 3:
can be shown (see Appendix[Affor details) that za can be expressed as being drawn from exponential family with a Beta prior on the hurdle probability Pa and normal-inverse-gen gamma prior on the log- normal parameters μl and o. We apply this exponential family model independently the 512 dimensions. Since the optimal hyperparameters for the particle filter are a priori unknown, we metalearn the hyperparameters of the conjugate prior using the same minibatch setup as the neural circuit and Adam with learning rate of 0.1. Note that gradient estimation with respect to hyperparameters possible the particle filter since the NLL of sequence with sequential observations can be computed without the use of particles.
each of
The results of this experiment show in Table 2 Despite the effort adapt the particle filter this setting by carefully selecting the exponential family model, the neural circuit outperforms the particle filter by a large margin, both terms of predictive performance and computational efficiency, Here, fully unobserved inference in the neural circuit is over 100 faster than the particle filter, since exponential family inference in the particle filter must be performed separately over each of the 512 dimensions. As expected, predictive performance of the neural circuit drops when evaluating on novel classes drawn from the meta-test set, as these novel classes represent patterns of activations the eircuit has not encountered during metalearning. However, the neural circuit still significantly outperforms the particle filter even this difficult setting. Importantly, the architecture and training procedure of the neural circuit is identical to our setup in the experiment with synthetic data (Section 4.1). which speaks to the versatility of our method.
Table Results on Imagenet- CRP with ResNet- 18 activations as features. Evaluation computed as average over 10,000 held-out sequences of length 100. Negative log- liglliielsooods are expressed in nats per timestep.
Meta-train Classes
Meta-test Classes
Inference Time (ms/sequence)
Method
NLL (L)
ARI (f)
NLL (L)
ARI (t)
Seq. Obs. (1)
Fully Unobs. (J)
CRP
1.005
0.010
1.003
0.009
0.019
0.185
Softmax + Energy
3.196
0.006
3.471
0.004
1883.066
1907.395
Particle Filter
0.848
0.070
0.933
0.048
2.407
73.896
Neural Circuit
0.255
0.749
0.680
0.271
0.067
0.412
4.3
Robustness to Distribution Shift: Open-set Classification on iNaturalist 2018
In order to test the robustness of our neural circuits to a strong form of distribution shift, we apply circuits trained using ImageNet-CRP to open-set classification on the long-tailed iNaturalist 2018 18 dataset [61]. iNaturalist consists of 437,513 images, each having labels at 7 taxonomic levels: kingdom, phylum, class, order, family, genus, and species. We therefore consider versions of the task, each treating a different taxonomic level the class label. This setting represents distribution
7

---

shift not only in the images but also in the label statistics (though long-tailed, they are no longer drawn from a CRP).
Due to the fine-grained nature of iNaturalist 2018, for this section we modify the training of the neural circuits on ImageNet-CRP in several ways. First, we extind the sequence length 500 during metalearning. The lower levels of the taxonomy have a large number of classes, meaning that the expected number of images per class low for sequences of length 100. Second, train range of neural circuits, each with different value of a E {1, 5, 10, 20, 50, 100, 200}. This is because the coarser levels of the taxonomy may be better represented by low values of and the finer levels better represented by large values of a. Third, we use an ImageNet class split of 350 meta-train 350 veta--alidation, and 300 meta-test in order to mitigate the potential risk of overfitting.
Our baseline is a CRP with a tuned to provide the hest performance on that task. For each taxonomic level, we evaluate the neural circuit trained with the most similar the CRP-tuned oracle a. This meant to emulate the setting in which learner has only a rough estimate of the level of diversity expected when encountering a new environment. In order to help bridge the input distribution shift, we also apply an affine transformation and ReLU activation on top of the ResNet-18 features extracted from iNaturalist. The weights of this layer trained using a small number (n 10) of training
sequences and early stopping is performed on the basis of validation sequences (n = 10). The weights of the neural circuit remain frozen. neural circuit that performs better than the CRP means that transfer from ImageNet-CRI to iNaturalist has successfully occurred. We construct test sequences from iNaturalist by sampling 1,000 randomly permuted sequences of length 100.
Table 3: Dataset transfer from ImageNet-CRP to iNaturalist 2018. The a of the CRP base- line is tuned to provide optimal performance, whereas the neural circuit is selected within (1,2,5, 5, 10, 20, 50, 100, 200} to be closest to the tuned CRP a. Evaluation is performed on 1,000 randomly permuted sequences of length 100 from iNaturalist 2018 at each of 7 taxonomic levels. The average NLL per timestep is reported in terms of minimum, mean, and maximum across 5 runs with different random seeds. Best results are highlighted in bold.
CRP
Neural Circuit (Ours)
Taxonomy
Tuned
NLL
Pretrained
Min. NLL
Mean NLL
Max. NLL
Kingdom
0.6
0.70
0.37
0.42
0.55
Phylum
2.1
1.30
0.80
0.84
0.88
Class
5.3
1.82
1.27
1.31
1.35
Order
33.1
2.24
20
2.15
2.28
2.49
Family
144.5
1.56
100
1.58
1.62
1.74
Genus
758.6
0.53
200
0.58
0.59
0.60
Species
1584.9
0.28
200
0.38
0.39
0.40
Our results (Table show that positive transfer indeed occurs for three the seven levels, indicating that the neural circuit is able to successfully transfer to novel open-set classification tasks in spite of distribution shift both label and image statistics. Interestingly, the levels which the neural circuit performs best are the higher levels, which there are likely to be multiple images per category.
We additionally perform an analysis to examine the effect of mismatch a between meta-tres train and meta-test. Our results (see Appendix C.2) indicate that for low-moderate levels of a, it may be beneficial train on a slightly higher than anticipated, whereas for larger values of a, it is best to match these statistics at meta-train time.
Related Work
Approximate solutions for nonparametric Bayesian models have historically used Markov chain
Monte Carlo (MCMC) [4] or variational inference [64], with examples including Gaussian processes
660, Indian-Buffet processes [18] [5], and the Dirichlet process models we consider here 9 47).
These approaches, however, assume batch setting that is not efficient for sequential inference.
Inference can be made more efficient in sequential settings by considering online variational inference
with fixed update costs [10] various forms of variational particle filtering [37] 42 52 However,

---

overly restrictive distributional assumptions (e.g. exponential family) can hinder performance when working with complex feature spaces such as neural network representations.
Amortized inference 135 49 is better suited to complex problem domains since it learns a function that maps directly from inputs to an approximate posterior distribution, effectively amortizing the cost of variational inference. Several works have considered amortized inference over Dirichlet priors 21 34 43). Gaussian process priors [_] and nested Chinese Restaurant Process priors [26]. Similar to the classic approaches mentioned above, these methods belong to the family of batch inference solutions. Sequential variational autoencoders [5] 25
28 38 offer an online solution but none of these approaches are suitable for sequential inference in a DPMM.
Our neural circuit approach offers scalable solution to sequential nonparametric Bayesian inference in a DPMM. We do so by using metalearning to incorporate the inductive bias of a nonparametric Bayesian model into the learned network rather explicitly instantiating such model. This is reminiscent of previous metalearning approaches that train with "episodes" [63] to learn a general prior distribution over weight initializations (24 [27], languages [39], or supervised regression
and classification problems [41]. Similarly, our method can be viewed as learning an implicit application of Bayes' rule in a sequential setting [3] where meta-learning defines Bayesian updates over complicated distributional assumptions 445) We use sequences of observations and labels that
can also be viewed as a kind of episode, but our goal is different: we aim to perform sequential inference over an unbounded number of classes. Compared to the previously mentioned approaches using variational inference or amortized inference, we leverage metalearning to directly learn a distribution over class labels rather than jointly learning encoder and decoder networks to maximize a variational lower bound on the log likelihood of the observations.
Related to our aims, the goal of open set recognition (OSR) is, broadly speaking, to detect previously unseen classes while accurately predicting known classes [54]. Several early approaches to OSR focus solely on detecting previously unseen classes, either using traditional machine learning methods
[3 32 55
68 and, more recently, creating neural networks with open-set capabilities 17 (7).
Bendale and Boult [6] first treated OSR as an incremental learning problem by using extracted image features to perform metric learning over known classes initially, and performing incremental class learning thereafter. This method employs thresholded distances from the nearest class mean as its decision function. Rudd et al. (50) advanced this approach by introducing distributional information into the thresholding process. Both methods rely on large datasets of known classes. [36 introduced a
metalearning formulation of OSR based on thresholding prototypical embeddings (58] to address this
limitation. Similarly, Willes et al. [67] proposed a method called FLOWR that combines prototypical embeddings with Bayesian nonparametric class priors. These methods operate in the sequential observation setting, observing the true class label after every prediction, and explicitly learn a metric space end-to-end instead of performing inference over arbitrary input features (including possibly representations extracted from pretrained network) compare against closely resemble a variant of FLOWR modified for our setting.
do. The particle filter baselines (23
Conclusion
We have proposed neural circuits for sequential nonparametric Bayesian inference: metalearning recurrent neural network to capture the inductive bias of DPMM that generates the training sequences. Our approach outperforms particle filter baselines in predictive performance while being more computationally efficient. Neural circuits are simple to implement and flexible enough to handle complex inputs with minimal modifications to their training procedure and architecture. In future work, we plan to use the neural circuit approach to capture inductive biases of more complex nonparametric Bayesian models with richer latent spaces.
Two limitations of our neural circuit approach are i) the difficulty of metalearning as a -> oo, and ii) misspecification of the base distribution when there is mismatch between the data generating distribution of meta-train and meta-test sets. Classical DPMMs specify these quantities explicitly, making modeling assumptions easier to identify and reason about. The process of meta-learning learning the model itself from data brings about different challenges. Care should be taken when deploying the model in scenarios where the diversity in labels differs greatly from metalearning. Additionally, large values may be difficult to learn due to the infrequency of repeated class instances

---

in meta-traiin metattraining sequences. In future work we plan to explore how these challenges can be addressed through curriculum learning.
Neural circuits are broadly applicable to the growing use of foundation models and pre-trained networks. They bridge the gap between Bayesian nonparametric methods, which often make more appropriate assumptions for real-world scenarios, and the powerful representations of neural networks. This integration allows existing neural networks to be adapted for tasks such as open-set recognition within a principled and efficient Bayesian framework, without the need to retrain the base model. We anticipate that this approach can be used in wide range of settings to be able to deploy interpretable models with more explicit inductive biases build on top of expressive representations for complex datasets.
Acknowledgments and Disclosure of Funding
This work was supported by grant N00014-23-1-255 from the Office of Naval Research. The authors would like to thank Liyi Zhang for useful discussions.
References
[1] David J. Aldous, Exchangeability and related topies. In David J. Aldous, IIldar Ibragimov, and Jean Jacod. editors, École d'Ééé de Probubilités de Saint-Flour XIII 1983. pages 1-198. Springer Berlin Heidelberg, 1985.
[2] Marcin Andrychowicz, Misha Denil, Sergio Gómez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. Learning to leam by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, 2016.
[3] Charles E. Antoniak. Mixtures of Dirichlet Processes with Applications to Bayesian Nonparametric Problems. The Annals of Statistics, 152:115211774 1974.
[4] Baxter. Theoretical models learning to learn. T. Mitchell and Thrun, editors, Learning to Learn. Kluwer, 1997.
[5] Gianluca Bencomo, Jake Snell, and Thomas I. Griffiths. Implicit maximum posteriori filtering via adaptive optimization. In The Twelfth International Conference Learning Representations, 2023.
[6] Abhijit Bendale and Terrance Boult. Towards open world recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015.
Abhijit Bendale and Terrance F. Boult. Towards open set deep networks. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition, 2016.
[8] Samy Bengio, Yoshua Bengio, Jocelyn Clouticr, and Jan Geesei. On the optimization of a synaptic learning rule. In Proceedings of the Conference on Optimality in Artificial and Biological Neural Networks, 1992.
[9] David M. Blei and Michael Jordan. Variational inference for Dirichlet process mixtures. Bayesian Analysis, 1(1):121 143, 2006.
[10] Andrew Campbell, Yuyang Shi, Thomas Rainforth, and Arnaud Doucet. Online variational filtering and parameter learning. Advances Neural Information Processing Systems, 2021.
[11] Francesco Paolo Casale, Adrian Dalca, Luca Saglictti, Jennifer Listgarten, and Nicolo Fusi. Gaussian process prior variational autoencoders. In Advances in Neural Information Processing Systems, 2018
[12] Hakan Cevikalp. Best fitting hyperplanes for classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, (166):077-1088. 2016
[13] Hakan Cevikalp and Hasan Serhan Yavuz. Fast and accurate face recognition with image sets. In Proceedings of the IEEE International Conference Computer Vision Workshops, 2017.
[14] Nicolas Chopin and Omiros Papaspiliopoulos. An introduction to sequential Monte Carlo. Springer. 2020.
[15] Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Gocl, Aaron C. Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In Advances in Neural Information Processing Systems, 2015.
[16] John G. Cragg. Some statistical models for limited dependent variables with application to the demand for durable goods. Econometrica, 39(5):829-88)) 844, 1971.
[17] Akshay Raj Dhamija, Manuel Günther, and Terrance Boult. Reducing network agnostophobia. In Advances in Neural Information Processing Systems, 2018.
[18] Finale Doshi, Kurt Miller, Jurgen Van Gael, and Yee Whye Teh. Variational inference for the Indian buffet process. In International Conference on Artificial Intelligence and Statistics, 2009.
10

---

[19] Arnaud Doucet and Adam M. Johansen. A tutorial on particle filtering and smoothing: Fifteen years later. Handbook of Nonlinear Filtering, 12(656-704):3, 2009.
[20] Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett Ilya Sutskever, and Pieter Abbeel. RL": Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:161I 1116611.077799 2016.
[21] Ehsan Abbasnejad. Anthony Dick, and Anton van den Hengel. Infinite variational autoencoder for semi- supervised learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[22] M. D. Escobar and M. West. Bayesian density estimation and inference using mixtures. Journal of the American Statistical Association, 00:577-588, 1995.
[23] Paul Fearnhead. Particle filters for mixture models with an unknown number of components. Statistics and Computing, 14(1):1 2004.
[24] Chelsea Finn, Picter Abbeel, and Sergey Levine. Model-agnostic meta- meta-Icarning for fast adaptation of deep networks. In Proceeedings the 34th International Conference on Machine Learning, 2017.
[25] Marco Fraccaro, Søren Kaac Sønderby, Ulrich Paquet, and Ole Winther. Sequential neural models with stochastic layers. In Advances in Neural Information Processing Systems, 2016.
[26] Prasoon Goyal, Zhiting Hu, Xiaodan I Liang. Chenyu Wang, and Erie P. Xing. Nonparametric variational auto-encoders for hierarchical representation learning. In Proceedings of the IEFE International Conference on Computer Vixion, 2017.
[27] Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient-based meta-learning as hierarchical Bayes. In International Conference on Learning Reprexentations, 2018.
[28] Karol Gregor, George Papamakarios, Frederic Besse, Lars Buesing, and Théophane Weber. Temporal difference variational auto-encoder. In International Conference Learning Representations, 2019.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Rea, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2016.
[30] Sepp Hochreiter, A. Steven Younger, and Peter R. Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Nenworks, 2001.
[31] Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of Classification, (1):193 193-218, 1985.
[32] Lalit P. Jain, Walter J. Scheirer, and Terrance E. Boult. Multi-class open set recognition using probability of inclusion. In European Conference on Computer Vision, 2014.
[33] Edwin T Jaynes. Probability theory: The logic of science. Cambridge university press, 2003.
[34] Weonyoung Joo, Wonsung Lee. Sungrae Park, and Il-Chul Moon. Dirichlet variational autoencoder. Puttern Recognition, 107:107514, 2020.
[35] Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
[36] Bo Liu, Hao Kang, Haoxiang Li, Gang Hua, and Nuno Vasconcelos. Few-shot open-set recognition using meta- leta-lea In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[37] Chris J. Maddison, John Lawson, George Tucker, Nicolas Heess, Mohammad Norouzi, Andriy Mnih, Amaud Doucet, and Yee Tch. Filtering variational objectives. In Advances in Neural Information Processing Systems, 2017.
[38] Joseph Marino, Milan Cvitkovic, and Yisong Yue. A general method for amortizing variational filtering. In Advances in Neural Information Processing Systems, 2018
[39] R. Thomas McCoy and Thomas I.. Griffiths. Modeling rapid language learning by distilling Bayesian priors into artificial neural networks. arXiv preprint arXiv:2305.147700 2023.
[40] Vladimir Mikulik, Grégoire Delétang, Tom McGrath, Tim Genewein, Miljan Martic, Shane Legg. and Pedro Ortega. Meta-trained agents implement bayes-optimal agents. In Advunces in Neural Information Processing Systems, 2020.
[41] Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. In International Conference on Learning Representations, 2021.
[42] Christian Naesseth, Scott Linderman, Rajesh Ranganath, and David Blei. Variational sequential monte carlo. In International Conference on Artificial Intelligence and Statistics, 2018.
[43] Eric Nalisnick and Padhraic Smyth. Stick Stick-breaking variational autocncoders. In International Conference on Learning Representations, 2017.
[44] Radford M. Neal. Markov Chain Sampling Methods for Dirichlet Process Mixture Models. Journal of Computational and Graphical Statistics, 92):249 2000.

---

[45] Pedro A Ortega, Jane X Wang, Mark Rowland, Tim Genewein, Zeb Kurth-Nelson, Razvan Pascanu. Nicolas llesss, Joel Veness, Alex Pritzel, Pablo Sprechmann, et al. Meta-learning of sequential strategies. arXiv preprint arXiv:/905.038:
2019.
[46]
Jitendra Parmar, Satyendra Chouhan, Vaskar Raychoudhury, and Santosh Rathore. Learning: Applications, Challenges, and Opportunities. ACM Computing Surveys, 55(10):1 37, 2023.
Machine
[47] Carl Rasmussen. The infinite gaussian mixture model. In Advances in Neural Information Processing Systems, 1999.
[48] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference on Learning Representations, 2017.
[49] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approxi- mate inference in deep generative models. In International Conference on Machine Learning, 2014.
[50] Ethan M. Rudd, Lalit P. Jain, Walter J. Scheirer, and Terrance E. Boult. The extreme value machine. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(3):762-768, 2017.
[51] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211-252, 2015.
[52] Ardavan Saeedi, Tejas Kulkarni. Vikash K. Mansinghka, and Samuel J. Gershman. Variational particle approximations. The Journal of Machine Learning Research, 18( (8(11:2328-223333 2356,, 2017.
[53] Adam Santoro, Sergey Bartunov, Mattew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory- auggenttd neural networks. In Proceedings of the 33rd International Conference on Machine Learning, 2016.
[54] W. J. Scheirer, A. de Rezende Rocha, A. Sapkota, and T. E. Boult. Toward Open Set Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(7):1757-1772, 2013.
[55] Walter J. Scheirer, Lalit P. Jain, and Terrance E. Boult. Probability models for open set recognition. JEEE Transactions on Pattern Analysis and Machine Intelligence, 36(11):2317-232) 2014.
[56] Jürgen Schmidhuber. Evolutionary principles in self- refereneflll learning. PhD thesis, Institut für Informatik, Technische Universität München, 1987.
[57] Lei Shu, Hu Xu, and Bing I. Liu. DOC: Deep open classification of text documents. In Conference on Empiriaal Methods in Natural Language Processing, 2017.
[58] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems, 2017.
[59] Yee Whye Teh, Dilan Grür, and Zoubin Ghahramani. Stick-breaking construction for the Indian buffet process. In International Conference on Artificial Intelligence and Statistics, 2007.
[60] Dustin Tran, Rajesh Ranganath, and David M. Blei. The variational gaussian process. In International Conference on Learning Representations, 2016.
[61] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The iNaturalist Species Classification and Detection Dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[62] Nguyen Xuan Vinh, Julien Epps. and James Bailey. Information theoretic measures for clusterings comparison: Is a correction for chance necessary? Proceedings of the 26th Annual International Conference on Machine Learning, 2009.
[63] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, 2016.
[64] Martin J. Wainwright and Michael I. Jordan, Graphical models, exponential families, and variational inference. Foundations and Trends@ in Machine Learning, 1(1-2):1-305, 2008.
[65] Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement leam. arXiv preprint arXiv: 1611016611.07663. 2016.
[66] Ross Wightman. Pytorch image models. 2019.
[67] John Willes, James Harrison, Ali Harakeh, Chelsea Finn, Marco Pavone, and Steven Waslander. Bayesian embeddings for few-shot open world recognition. IEEE Transactions on Puttern Analysis and Muchine Intelligence, 2022.
[68] He Zhang and Vishal M. Patel. Sparse representation based open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(8): 1696, 2016.
12

---

A Derivation of Hurdle Model
First we recall the form of exponential family likelihood:
p(x m) - (xx exp {(n, t(x)) A(n))
(13)
The corresponding conjugate prior takes the following form:
p(n T. 1/) = exp ((T, n)) -A(A)) B(T,1)}
(14)
A.1 Deta-Bernoulli
We first show how Bernoulli likelihood and Beta conjugate prior can be expressed in terms of (13)
and (14) This is useful stepping stone towards the hurdle model, which contains the Bernoulli distribution as a component. Recall that for a Bernoulli likelihood (ern())0),
p(z 8 (1 0)2- p(z))) = x log 0 + (1 x)) log(1 0),
(15)
(16)
from which we recognize
0
17 log 1
(17)
t(x) x
A(n) = log(1- - 0).
Similarly, recall that for Beta conjugate prior Beta Beta(O)u, b).
T(a ga-l(1 (1 0)6-1
p(fla, b)
(18)
(b)
1
V = a + b b-2
B(T, v) log ((a) log [(b) log [(a + b)-
A.2 Hurdle Model
Now suppose have an arbitrary exponential family likelihood p(z[n) of the form (13) and conjugate prior p(n|T, v)) of the form (14) Now define a hurdle model based on this likelihood to be:
1-0
((x n)
(19)
0p(zln)
+ 0.
Intuitively, this is an application of a Bernoulli variable that gates whether the underlying exponential family likelihood is active or not. Observe that (<>l))) may be written as:
qx= (i) = (1 - 0) (1- (1=1 (0h(r) exp ('n t(x)) A(n)))
(20)
log q< ri) = log(1 - 0) + 1 + 03 ((x) + (n. ((t)((((1))))))) (- + 00) + log
1-0 A(m) 1 fx + 0}
0
(21)
We therefore recognize ((x) 7) itself exponential family:
(x = h(x) exp (n, t(x)) A())
(22)
log 1-0 A(n)
7/1
n
7/2
(x*0)
(=(2) 0)
(x
i(x) =
log h(x) = 1 (x + 0) log M(x)
Á(σ) = log(1 0)
13

---

We also posit a conjugate prior for g(z
q D) exp D) iA(m)) B(F
(23)
log (ū] - fu + (T. n) DA(1)) - B(F,
Now let (ern(x))) = h(z)) exp (7, [(x) - A with definitions following from (17) In this case, n log I-8 and thus A(r) = A()). Moreover, î: = fj A(1). and we can then write
log g 77 . .) = - ( - A(n)) + (T. - - - B(F, P) (T, 51 A(n) + - (A( B(F,D),
(24)
from which we recognize that
B(7, B(T,V= f1) + D).
(25)
Therefore, the hurdle applied on top of an arbitrary exponential family likelihood can be expressed in terms of the underlying base model and the Beta-Bernoulli presented in Section A.
B Softmax + Energy Model
The softmax energy baseline is derived by viewing standard logistic regression model through the lens of the maximum entropy principle (2). Suppose that Wi...., w are the vector of weights corresponding to each of K known classes (it is assumed that the input x has been augmented with an dummy feature to capture a bias term). The predictive distribution then:
exp(w) W x)
ply = k|x) Σ exp(w.
(26)
This can be viewed as a maximum entropy distribution where each weight Widd is a Lagrange multiplier representing a constraint on E[Tly= k]]aa] under the model. Now in order to allow for
to take on the value K + 1, representing new class, we introduce an additional constraint that the marginal probability that y K + 1 follows the Chinese restaurant process 3 o where t is the current timestep. The corresponding Lagrangian then becomes (discretizing and setting D 1 for simplicity of illustration):
Q
Σ
Pei log miki
Pki
+No
1
K+1 Σ
Σ
Pki
Av
Me
Σ T:pki +dx+1 3
Σ PK+1A
1=
(27)
Setting aQ_ 0 for k = =1, 1,..., K and similarly for
aQ 0px+1.1
yields:
Spm.
1
Pks Z
Mkre AAas
(28)
1
PK+11 Z
m K+1see" AAX1
(29)
K
z= Σ MK+1.e AK+1
myee
Ayzi
(30)
Thus, we can compute the probability of a new class occurring as:
Axx+1
P = K + 1/=;) = PK+1. 6- c-Ax11 + Ek e-Au
(31)
which we recognize as a softmax with wik -Ax for = 1,..., K and - playing the role of aQ
an additional logit, which we will presently solve for. By setting ūAK+1
0, it can be shown that
-AK+1 BZ. Substituting into (30) and solving for Z yields:
Z
mare Auzi
(32)
14

---

Therefore, we have that
B
K
AK41 = log
+
log
mase Auzi
(33)
1 8
k=
log - -log(t 1) + log
myje "k
(34)
k=1
The derivation extends in the straightforward manner with D >
1. By considering infinitesimally
small discretization and specifying a uniform measure m(x) over -M, MI. can be shown that:
K
D
sinh Sinh(Mwkel
AK+1 = log a log(t - 1) + log
Σ
exp
Σ
log
(35)
MWkk
k=1
Similarly, choosing a Gaussian measure m(x) = exp
(-
x2
produces
20²²
1
log o + of ow
AK++1 log a log((t 1) + log
exp Σ
log(2n) (2x)
(36)
We use (35) as the augmented class logit in the ImageNet experiments this better represents the state of knowledge for the ResNet-18 activations and similarly we use (36) for the 2-D synthetic
DPMM augmented logits. We learn the weights of the logistic regression classifier after every timestep by using gradient descent with the Adam optimizer.
C Additional Experimental Results
In this section, we include additional experimental results.
C.1 Visualization of Predictive Distribution
In Figure we visualize the evolution of the neural circuit predictive distribution across timesteps.
50
190
Clamir
Duatar Sun Oudli
t 10
50
1- 100
son Okitz
20
Figure 4: Visualization of the predictive distribution over cluster assignments across timesteps for a neural circuit model trained on the two-dimensional data synthesized from a DPMM. Colors are determined by mixing each cluster's color proportional to the output softmax probabilities. Yellow indicates new, as yet unobserved class.
15

---

C.2 Effect of Varying
In this section, we evaluate the effect of mismatch in a between meta- meta-train and meta-test. We take the neural circuits trained on ImageNet- CRP from Section|..3 and and evaluate all possible combinations of meta-train and meta-test a. The results (Table show that for low-moderate levels of a, may be beneficial to train slightly higher than anticipated. This is likely due to their increased level of diversity. However, when evaluating on large values of a, it tends be best to match these statistics at meta-train time.
Table 4: NLL on ImageNeet-CRP Meta-test Classes under different meta- veeta-trainin conditions. Evalua- tion is performed on 1,000 sequences of length 500.
Meta- Veta-Training
Meta- Meta-Test a
10
20
50
100
200
1
0.76
0.75
0.76
0.78
0.87
1.10
1.25
2.30
2
1.24
1.19
1.18
1.21
1.31
1.51
1.83
2.80
5
2.26
2.02
1.90
1.89
1.97
2.16
2.41
3,09
10
3.37
2.94
2.62
2.52
2.53
2.67
2.88
3.30
20
4.13
3.78
3.40
3.23
3.06
3,08
3.26
3,56
50
4.60
4.44
4.22
4.58
3.83
3,49
3.55
3.77
100
4.79
4.74
4.63
5.06
4.34
3.97
3.47
3.53
200
4.94
4.94
4.91
5.37
4.69
4.51
3.45
3.08
C.3
Additional Results on Synthetic DPMM Data
In T show additional metrics collected in the synthetic DPMM setting from Table Here we show the results in terms of min, mean, and max across 5 training runs with different random initializations.
Table 5: Results two-dimensional data synthesized from DPMM for 5 neural circuits each trained with a different random initialization. Results are summarized as the min, mean, and max of the models. Inference times reported in units of ms/sequence.
Metric
Min.
Mean
Max.
NLL (L)
0.075
0.076
0.078
Perplexity (L)
1.085
1.086
1.089
ARI (t)
0.920
0.921
0.922
AMI (f)
0.927
0.928
0.929
Inf. Time (Seq. Obs.) (4)
0.056
0.059
0.064
Inf. Time (Fully Unobs.) ())
0.394
0.421
0.445
C.4 Additional Results on ImageNet-CRP Data
Similarly, we show in ohe min, mean, and max results of neural circuits trained with 5 different random initializations.
D Additional Experimental Details
For all of our experiments neural circuit RNN cell was chosen to be a 2-layer gated recurrent unit (GRU) [I] with hidden size 1024. The maximum number of output logits was sequence length used for metalearning, as this is the maximum number of classes that could possibly be encountered. Training was performed over 10,000 minibatches each of size 128 sequences of length 100 (256 sequences of length 500 for the iNaturalist experiments). The CRP coefficient was set to a 1.0 for the synthetic and ImageNet-CRP experiments, and was set range of different
be equal the
values {1, 2, 5, 10, 20, 50, 100, 200} in the iNaturalist experiments. The neural circuit was trained
16

---

Table 6: Results on ImageNet-CRP for 5 neural circuits each trained with a different random initialization. Results are summarized as the min, mean, and max of the models. Inference times are reported in units of ms/sequence.
Metric
Min.
Mean
Max.
Meta-train NLL ())
0.239
0.255
0.272
Meta-train ARI (1)
0.728
0.749
0.770
Meta-test NLL (L)
0.668
0.680
0.696
Meta-test ARI (f)
0.260
0.271
0.286
Inf. Time (Seq. Obs.) (L)
0.066
0.067
0.068
Inf. Time (Fully Unobs.) ())
0.407
0.412
0.420
using Adam with learning rate 0.001 and the particle filter using learning rate 0.1. For the synthetic and ImageNet-CRP results, the tuning hyperparameters (e.g. model architecture, learning rates) were selected by monitoring training loss over the first several hundred minibatches and modifying as necessary. For the iNaturalist 2018 results, overfitting was monitored on a separate meta-validation class split. MAP prediction is implemented in the neural circuit by simply selecting the class label with highest predicted probability for each timestep.
One random seed used for training each method. The clustering metries computed using the adjusted_rande sereoddrrnd__oorrrd ssore and adjusted_mttuse _eal__nnf____________ score functions from Scikit Learn [5]. Experiments were performed using NVIDIA A100 GPUs with 40 GB of GPU memory and 64 GB of CPU memory across threads. Training of the neural circuits took approximately 6-8 hours per run.
For the particle filter, resamples according to a multinomial distribution over the particles whenever the effective sample size drops below 50. In the fully unobserved setting, we take the MAP prediction to be the particle with the largest weight after running the filter on a sequence.
100 particles for inference and utilize adaptive resampler that
For the experiments in Section 4.2 the meta-training classes were taken to be the first 500 and the meta-test classes were taken to be the last 500 classes, sorted by class ID, For Section 4.3 the meta-training classes were the first 350 classes, the meta-validation classes the next 350 classes, and the meta-test classes the final 300 classes, as sorted by class ID.
All methods are implemented PyTorch 4 and are GPU-enabled. Our code is available online
References
[1] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv: 1409, 1259, 2014.
[2] Edwin Jaynes. Probability theory: The logic of science. Cambridge university press, 2003.
[3] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations, 2015.
[4] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, 2019.
[5] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer. R. Weiss, Dubourg, Vanderplas, A. Passos, D. Cournapeau. M. Brucher, Perrot, Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-285 -8300, 2011.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our proposed approach is discussed in Section 3 and experimental justification for its effectiveness is contained in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
This submission does not contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Experimental details are discussed briefly Section 4 and in detail Ap- pendix D
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
Code is made available at oil.tter//ghhhh....///aaa//////r
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee //nips.com /ttc://lllllliiiiee////eeeaaaass ssiniiiiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Experimental details may be found Appendix D Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The number of sequences used for evaluation is stated alongside each experi- mental result. Results across multiple training runs are reported in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Computational resources are discussed Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the code of ethics and agree abide by them.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts are discussed in Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not anticipate high risk of misuse for our proposed model. Limitations and societal impacts are discussed in Section
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Both the ImageNet [5]] and iNaturalist 2018 [61] datasets are cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets released with this paper. Code and documentation included in the supplementary material.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No human subjects were used during the preparation of this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human participants used during the preparation of this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "l3UnKycPWS", "reviewer": "Reviewer_grnU", "summary": "This paper proposes to use recurrent neural network (RNN) for classification with an open (non-fixed) number of class labels. This is motivated by non-parametric Bayesian models such as Dirichlet process mixture model and the iterative update on exponential family sufficient statistics in its particle filter methods. The latent state of RNN would resemble the iteratively updated sufficient statistic, and can be used for classifying data into previously seen or a first time occurring class label. Experiments on synthetic data and image data are provided, along with an extended study on robustness to distribution shift.", "questions": "1. Have you tried visualizing the learned weight matrix W? It would be interesting to see if the probability of introducing a new cluster decays with the number of data and the number of observed clusters like Chinese restaurant process.\n2. How does the expected number of clusters scale with the number of data? Does the curve resemble some Pitman-Yor process?\n3. How could I use this approach as an efficient approximation of Bayesian nonparametric methods and make inference (e.g. getting posterior distributions or some summary statistics of it)? I could imagine that we repeatedly draw random samples from the softmax probabilities in Eq.(10) instead of taking argmax, but this would just become another version of particle filter methods. Is there some better ways of doing this? If so, then weakness 1 wouldn't really be a weakness.", "limitations": "See weaknesses and questions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well organized and written, with enough background and clear motivation.\n2. The proposed method is interesting and innovative, bridging the two largely non-overlapping fields of Bayesian nonparametrics and deep learning.", "weaknesses": "1. It seems that the proposed method only does the classification task, but looses the inference side of Bayesian nonparametric methods.\n2. The ability to assign label to a class occurring for the first time is through the mask m_t in Eq.(10), whereas the dimensions of weight matrix W, mask m_t have to be pre-fixed. That is, we are not really assuming there can be infinitely many classes as in Bayesian nonparametric methods, but instead picking a large enough number and assuming that the number of classes is smaller (this is not necessarily a weakness, just some differences I noticed).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments. Visualizing the weight matrix W and examining the relationship between the expected number of clusters and the number of observations are both excellent suggestions. The probability of predicting a new cluster can be estimated in the synthetic data case via Monte Carlo sampling, as the marginal distribution in input space is known. We will update the paper to add this analysis.\n\nRegarding posterior inference, our method is primarily intended to produce a posterior distribution over cluster assignments. Although this does not provide sufficient statistics directly, from the visualizations in the above PDF (see the general rebuttal above), it is evident that this information is being represented in the hidden state of the RNN as it adapts over time to accommodate new clusters. Posterior per-cluster statistics could be estimated by sampling cluster assignments from the model and then averaging the statistic over observations assigned to the same cluster. We will update the paper to add discussion about this point.\n\nThe reviewer is correct in that the proposed method is not truly infinite in that there is an upper bound on the number of classes, determined by the output layer of the RNN. Care should be taken to choose an appropriate maximum output size with respect to the anticipated sequence length and growth rate of classes. As CRPs have a logarithmic relationship between the number of observations and the number of occupied clusters, we do not view this as a major drawback, but we will nevertheless make this point clearer by adding it as a limitation in the discussion.\n\nWe thank the reviewer for their insightful questions which have helped to improve our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zEuKLbzkEk", "reviewer": "Reviewer_QfVx", "summary": "This paper presents an approach for nonparametric Bayesian inference using metalearning to train a recurrent neural network (RNN) to perform sequential inference over an unbounded number of classes. The key contributions are: (a) method to extract the inductive bias from a Dirichlet process mixture model (DPMM) and transfer it to an RNN through metalearning, (b) \"neural circuit\" architecture that can perform fast sequential inference over an open set of classes after training and (c) experimental results showing the neural circuit achieves comparable or better performance than particle filter methods while being faster and simpler to use. The proposed approach consists of first generating training sequences from a DPMM prior. These sequences are then used to train an RNN to predict class labels sequentially. When trained to completion the RNN captures the underlying structure from the sequences (DPMM) and consequently can be used for fast inference. The authors include empirical results on synthetic data from a known DPMM, as well as more realistic experiments on ImageNet features with classes sampled from a Chinese Restaurant Process and a long-tailed iNaturalist species classification dataset. The neural circuit outperforms particle filter baselines in predictive performance and computational efficiency across these tasks.", "questions": "Please see weaknesses section above.", "limitations": "The authors discuss the limitations of the method in the conclusion, but I would also encourage them to discuss some of the aspects I highlight in the Weaknesses section.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The proposed method is a neat way of combining the strengths of nonparametric Bayesian models and deep neural nets. By using a RNN to amortize inference in a DPMM, the paper leverages the flexibility of DPMM and RNNs. The paper falls in a line of recent work using NNs to amortize Bayesian inference, e.g. [1]. So while the idea of approximating inference with NNs is not completely new, this paper is the first to study it in the context of DPMMs. \n* While not perfect as I will discuss below, the empirical analysis is quite thorough. The authors present experiments on a relatively simple synthetic domain as well as more challenging image-based experiments. The amortized approach perform better than or at least on par with the particle filtering baseline while being much faster at inference. \n* The method also appears to demonstrate some robustness to distribution shifts as the experiment of tranferring the ImageNet-trained model to iNaturalist shows.\n* The authors also provide code to aid reproducibility.\n\n[1] Transformers Can Do Bayesian Inference. Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, Frank Hutter. ICLR 2022.", "weaknesses": "* While the results are certainly significant, I find the experimental analysis lacking in several aspects. Firstly, the experiments in the more realistic open set image classification task are limited by fixing a pretrained featurizer to embed the image into a lower dimensional vector. This removes significant complexity from the inference task and also means the method already has access to some information about the underlying inference task. This also makes the experiments of transfer to the iNaturalist task a bit less impressive since the ResNet featurizer already embeds the images in a \"similar\" space.\n* The authors also perform all the experiments using a relatively small RNN (2-layer GRU). There isn't much of a discussion on how this particular architecture was chosen and more importantly there are no experiments to understand the effect of this choice. As this is a largely empirical paper, it is important to study these design choices. For instance, how does the performance vary if we use a larger RNN or if we use a transformer instead? Other such design choices are also not accounted for.\n* The comparison is somewhat limited to particle filter baseline, and despite the focus on open-set image classification, there is no consideration for other standard baselines for the task [2]. \n* Another limitation of the approach is robustness to model misspecifcation. Distilling the structure of a (fixed) DPMM into a RNN can make it harder to generalize to data coming from different (unknown) data generating process.\n\n[2] Large-Scale Open-Set Classification Protocols for ImageNet. Andres Palechor, Annesha Bhoumik, Manuel Günther. WACV 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and constructive feedback. The reviewer is correct that in our experiments we assume the use of a pretrained featurizer. Our initial motivation was to demonstrate that neural networks originally trained on a fixed number of classes can easily be adapted to the open-set setting by replacing the penultimate layer with our meta-trained circuit. Backpropagating through the pretrained feature extractor or training it directly from scratch are interesting avenues to explore and we are grateful to the reviewer for raising this point.\n\nRegarding design decisions of the network, we have run preliminary experiments exploring the role of hidden size in [512, 1024, 2048] and number of RNN layers in [1, 2, 3]. We trained these using the class split discussed in section 4.3 and found that all models had a similar final validation loss, with each falling in the range [0.7582, 0.7791]. Thus the performance is not very sensitive to this choice. We will add this ablation to the paper.\n\nWe also thank the reviewer for suggesting other baseline methods in the open-set literature. The primary barrier of previous open-set approaches is that they typically assume a considerable amount of training data for each of the known classes so that e.g. a prototype representation can be estimated for each class and distances computed to these in order to estimate the probability of a new class. This is problematic in the settings we consider here, as there may be only one or two class instances available for several of the classes. We thank the reviewer for the reference and will add another suitable open-set baseline to the paper accordingly.\n\nRegarding robustness to model misspecification, it is true that our method may struggle if the data-generating process is significantly different from that during training. We have explored this in Appendix C, Table 4 of the paper which provides some guidelines on choosing a good value of alpha that is able to handle variations to this at test time. This is a drawback shared by particle filters as well, which require specifying a prior assumption over alpha. However, one advantage of our proposed method is that in principle the metalearning procedure could incorporate perturbations to the alpha used to construct training minibatches, whereas it is not straightforward to adapt particle filters similarly. We will add more discussion about this point to the paper.\n\nWe thank the reviewer again for their insightful comments which have greatly strengthened our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qdy3yEbAqY", "reviewer": "Reviewer_aqs6", "summary": "This paper proposes a novel method for inference for Bayesian nonparametric models through an amortized approach. After simulating data from a Dirichlet Process Mixture Model, a Recurrent Neural Net is trained to characterise the relationship between the simulated data and the parameters, and consequently characterise the posterior distribution. Comparisons are made with a particle filter method for sequential inference for a DPMM, and experiments are run on simulated data, and image classification tasks using pre-trained neural net activations as inputs.", "questions": "Can this method be adapted for non-streaming data? Would a more general class of neural network architecture work for a different data paradigm? \n\nHave you considered a comparison with finite parametric mixture models, or overfitted finite mixtures? These are being given greater consideration now after some inconsistency results for DP clustering models, and the inference is substantially easier.\n\n\nWould this inference method work with a finite parametric mixture model, or a truncated nonparametric model?\n\nHigh dimensional (in the observed covariate space) model-based clustering is in general hard. Would this method have an advantage here, as neural nets normally adapt pretty well to high-dimensional spaces?\n\nLine 32: “nonparametric Bayesian models have fallen out of favour in machine learning” Really? Give more justification to this statement, or provide some examples of how they have been replaced. If they have fallen out of favour, why are you working with them? Is it just the inference that is holding them back from flourishing, or is it something else?\n\nLine 203: For a more general audience: What exactly is a ResNet? What has it been trained on? What kind of properties do you expect its activations to have relevant to the task? Can you explain this more thoroughly before going further?\n\nLine 227/228: where are the computational efficiency results exactly? They are not visible where you mentioned them.\n\nLine 248: is there no way of tuning the alpha within the neural net framework? If not, please explain why. The additional analysis looking at the transferability of alpha (line 262) is good, however!\n\nLine 272: Can you explain more clearly why none of references [10,37,42,52] are appropriate comparisons? Is there nothing more usable than the fiddly particle filter model?\n\nCan you do better than MAP prediction? What kind of posterior uncertainty does this method provide, if any?\n\nLine 551: “One random seed was used for training each method”? Really? Is this why there is no errors on the results? If so, this is a major weakness of the results here hidden in the Appendix. From the code of conduct, it seems that this is the case, and will be “updated to multiple runs in the next version of the paper” - does that include the revised version for Neurips?\n\nLine 553: “Experiments were performed using NVIDIA A100 GPUs” why the plural here? How many GPUs per experiment? This is unclear and an important detail.", "limitations": "Table 1/2/3/4: there is no uncertainty/errors on the losses. This is because everything was done with a single random seed, so maybe you’re not in a place to add uncertainty easily.\n\nLine 227-228: claims are made about the computational efficiency of the method that aren’t really backed up, and are possibly undermined by the inability to use more than one random seed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This is an interesting article taking a novel approach to inference for DPMMs. The motivation and theory are adequately explained, and the application examples chosen are interesting. The use of amortized inference in this context is novel: there are some interesting links to make here with Likelihood Free Inference (LFI) methods, where using a neural net to model the relationship between simulated data and the parameter space is becoming more common. See https://www.pnas.org/doi/10.1073/pnas.1912789117 for a relevant review. The quality of the scientific writing is in general high and concepts are explained clearly.", "weaknesses": "The experiments are not thorough: they each use one random seed per method, which gives no indication as to the uncertainty or robustness of the results. This is especially confusing, since the authors make a point of describing the new method as being faster than existing methods.\n\nThe computational resources needed for training the neural net are quite expensive, so maybe tone down the enthusiasm about this method being so computationally efficient. At least make clear that this method makes effective use of a generous budget of compute resources.\n\nThe examples used are interesting but quite limited in scope: the first example is (reasonably enough) a simple mixture of Gaussians, and the subsequent examples are practical but involve somewhat esoteric knowledge of using pre-trained neural nets as training data and similar. The article would benefit from a clustering example based on real observational data, as hinted at in the introduction. This would make the methods and article much more approachable for a more general audience\n\nLine 17/57: I wouldn’t call a mixture model a “classification model” exactly: that word is normally associated with supervised learning and known discrete labels. This is more generally considered “clustering”\n\nLine 76: “Chinese restaurant process” might need a little explaining for people who haven’t heard the analogy before. Otherwise good!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and constructive comments. In particular, we are grateful for the suggestion to improve our analysis by training models with multiple random initializations. Please refer to the PDF under the general author rebuttal above, where we show the results of training 5 different random initializations. These results show that there is relatively little variation across different runs of the model. We will update the paper to include these measures of uncertainty.\n\nWe also thank the reviewer for raising the point about the computational cost of training the network. Our perspective had been focused primarily on the situation at inference time after the models have already been trained. However, the reviewer is quite right that there are applications where a trained network is used for inference only a few times, in which case the amount of computation required for training may become burdensome. We will add discussion to the paper about this point and adjust our claims regarding computational efficiency to reflect this.\n\nThe suggestion to consider a clustering example based on real observational data is a welcome one. As our method can be applied to any input modality where a deep neural network is able to provide meaningful representations, we anticipate that our method would also perform well in this setting. We will add experiments to demonstrate this.\n\nIn terms of non-streaming data, our method could be adapted to this scenario by assigning an artificial order, or by sampling several random permutations and averaging over the predictions. Similarly, our method is compatible with any distribution defined over the cluster assignments, including finite and truncated nonparametric distributions. The only requirement is that it is tractable to draw samples from this distribution. In preliminary experiments, we had tried a uniform distribution over a fixed number of K classes, and found that the neural circuit was able to handle this case as well. We will add a section to the experiments that explores this setting.\n\nOur hope is that by focusing directly on the mapping from input to cluster assignment, our proposed method can better handle high-dimensional input spaces. This may be contrasted with particle filters, where an explicit cluster conditional distribution must be posited in the input space. This may be relatively straightforward in lower dimensions but for higher dimensions it becomes difficult to specify a suitable distribution. Neural circuits circumvent this issue to some degree by learning to directly predict a distribution over clusters. We invite the reviewer to please refer to the attached PDF under the general author rebuttal, which visualizes the predictions in the two-dimensional setting. It can be observed that the neural circuit is somewhat more flexible than a Gaussian cluster-conditional distribution and can instead learn an appropriate mapping to the cluster assignments directly.\n\nRegarding nonparametric Bayes, our intent was not to disparage NPB. On the contrary, our work is aimed at marrying the unique advantages of NPB with deep neural networks. We meant to say that large-scale deep models have become relatively more popular than nonparametric Bayes in recent years. We will update the paper to make this point clearer.\n\nWe will explain more clearly our use of ResNets as a representation, briefly stating that they are a popular choice for image classification and it is straightforward to download and use pretrained ResNets. The cost of training a large-scale image classifier from scratch can be prohibitive, so we see the ability to perform inference over fixed embeddings as a strength.\n\nThe choice of alpha can be viewed as a modeling assumption that reflects any knowledge that the practitioner has regarding the rough growth rate of classes. Since our method utilizes metalearning, there is a great deal of flexibility here. For example, if a practitioner is unsure about the exact sort of growth rate but is confident that the alpha will be in the interval [1, 10], then e.g. a uniform distribution could be placed on alpha and used when sampling the minibatches for training. The resulting neural circuit would thus be trained to be robust to selection of alpha within this range, perhaps at the cost of performance for a specific alpha.\n\nThank you for the suggestion to add more particle filter baselines, including those with a fixed number of clusters and variational approaches. We will update the paper to add these baselines.\n\nIn terms of posterior uncertainty, posterior cluster assignments may be sampled from the neural circuit and then used to estimate statistics as appropriate. We chose to use MAP prediction for the sake of simplicity in our fully unobserved evaluations as it should represent the model’s best single guess about the cluster assignment.\n\nThe connection to likelihood-free (simulation-based) inference is highly relevant as the CRP sampler plays this role by selecting the sequence of cluster assignments to be used in each minibatch of training. We will add discussion about this to the related work section.\n\nWe thank the reviewer again for their many insightful comments, which have greatly improved our paper. We hope that the reviewer would consider increasing their score based on these changes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vKXng2hCiT", "reviewer": "Reviewer_SHmK", "summary": "The authors present a recurrent neural net (RNN) that learns to mimic a Bayesian non-parametric (BNP) approach to classification with (potentially) heavy-tailed data and an a priori unknown number of classes. The method is evaluated on three experimental setups.", "questions": "- Only a single run is reported for each result. Was the seed to generate the same for all methods? E.g., has every model in Sec 4.1 access to the same training data?\n- In sec 4.2 the choice is to pick the representation space post-relu activations, placing an extra hurdle on the particle filter baselines (the hurdle model). Why was this choice made instead of sticking to common practice for taking the pre-activation representation features from pre-trained neural nets?  Which outcome would the authors expect without this artificial introduction of sparsity?", "limitations": "The limitations of the approach are adequately discussed by the authors.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper itself is well-written and motivated. It provides a clever approach to combining BNP with the straightforwardness of an RNN.  \nConditioned on the experimental weakness mentioned below, the improvements seem significant. However, as I am not familiar with the literature I cannot properly judge the significance, as I am not sure (but somewhat doubtful) whether the state of the art on this problem has not progressed in the last twenty years as the chosen baselines.", "weaknesses": "The weaknesses lie primarily in the experimental evaluation. \n- As discussed, the only baselines are a publication by Fearnhead from 2004, a vanilla Chinese restaurant process, and a self-designed non-Bayesian model. This hardly provides a proper evaluation of the approach. \n- All results are reported from a single seed without any significance analysis. (And no, promising multiple runs in a future revision of the paper is not sufficient to answer _yes_ to this question in the checklist. _No_ would have been the correct answer.)\n\n## Minor\n- The abstract claims better performance than _\"particle filter-based method**s**\"_ while only a single method is being compared against\n- Given that there is a single seed, reporting four decimals of significance seems unjustified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments. Regarding reproducibility, please see the attached PDF (under general author rebuttal), where we have added results for neural circuits trained using 5 different random initializations. Please note that the variation in results across these random seeds is low and the paper will be updated to reflect this measure of uncertainty. Each model has access to the exactly the same training data and is evaluated on identical precomputed test sequences. Thank you also for the suggestion to add more baselines. We will update the paper to add these. In terms of activations, our method is agnostic to the input representation. We chose to use post-relu activations as this is analogous to the final projection layer of standard neural networks. However, we will also try the pre-relu activations and update the paper accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank each of the reviewers for their feedback. In response to the suggestions provided by the reviewers, we have prepared several new results in the attached PDF.

We appreciate the suggestion to improve intuitions about the approach through visualizations. We have added plots visualizing the predictions for synthetic data, in which it is possible to clearly see the predictions of unobserved classes.

We have also added additional results to support the reproducibility of our findings. We have added results with five random seeds, and show that our findings hold up across this sample. We also note that each method has access to the same training and test data, and clarify that for each run both training and evaluation was performed using a single A100 with 80GB of GPU memory. Please note that the inference times have improved somewhat since the previous neural circuit inference times had been inadvertently collected on a less capable GPU, and this has now been remedied.

We hope the reviewers find these additions to our results convincing, and we appreciate the time and effort that went into generating such constructive feedback. We provide a detailed response to the points raised by each reviewer in the individual responses below.

Meta-review: This paper proposes a novel method for performing inference with Bayesian nonparametric models, specifically Dirichlet Process Mixture Models (DPMMs), by leveraging recurrent neural networks (RNNs). The authors achieve this by first simulating data from a DPMM and then training an RNN to classify this data. The trained RNN, called a neural circuit, can then be used to perform inference over an unseen sequence of data, potentially containing new classes not observed during training. The paper presents the strengths of this method to be its ability to handle open-set classification tasks (classifying data with potentially new classes) and its efficiency compared to traditional particle filter methods for BNP inference.

Strengths:

The paper is well-written and clearly explains the proposed method, its motivation, and its background.
The idea of combining Bayesian nonparametric models with neural networks is original and with potential for impact.
The paper presents experiments on synthetic data and real-world image classification tasks demonstrating the effectiveness of the proposed method.
The method offers advantages in terms of efficiency compared to traditional particle filter methods for BNP inference.

Weaknesses:

The reviewers raised concerns about the limited scope of the experiments, particularly the use of a pre-trained featurizer on the image classification tasks.
The lack of exploration of different network architectures and hyperparameters was also noted by some reviewers.
The paper could benefit from a more thorough comparison with existing open-set classification methods.
The limitations of the method, such as its potential sensitivity to model misspecification, should be discussed in more detail.

Recommendation:

All the reviewers voted for acceptance. I therefore recommend acceptance and encourage the authors to use the feedback provided to improve the paper for its final version.
Decision: Accept (poster)