dataesr/scientific-paragraphs-categorization huggingface dataset:
Additionally, RBMs can be stacked into multi-layer neural networks, which have played a historically fundamental role in pre-training deep network architectures  [15, 16]. These constructions, known as deep belief networks, were the first truly deep neural architectures, leading to the current explosion of activity in deep learning  [17]. While access to vast training datasets has made such pre-training dispensable for certain tasks, RBMs remain a fundamental tool in the theory of unsupervised learning. As such, a better understanding of RBMs can be key to future developments in emergent machine intelligence. To date, the most popular and effective approaches to training RBMs have centered on differing flavors of short-chain Monte Carlo sampling  [9, 18] , which we cover in detail in the sequel.
While such techniques can yield trained RBMs which produce sampled configurations very arXiv:1702. 03260v3 [cs. LG] 9 Oct 2018 similar to the target dataset, and can be used in a number of applications as detailed previously, they do not bridge the gap in understanding what the RBM has learned. Furthermore, understanding the modes, or internal representations, of the RBM with sampling-based frameworks have mostly consisted of subjective comparisons of sampled configurations as well as a subjective analysis of the couplings themselves, often referred to as receptive fields in the machine learning literature. Additionally, comparing two trained models, or even monitoring the training of one model, becomes problematic when using sampling-based investigative tools. For example, annealed techniques  [19]  can provide estimates of the log-likelihood of a model, but only at a large computational cost  [20, 21].
TAP machines also naturally handle non-binary variables as well as deep architectures. While Deep Boltzmann Machines' (DBMs)  [26]  state-of-the-art training algorithms mix both Monte Carlo sampling and "naïve" mean-field approximation, a deep TAP machine relies entirely on the TAP mean-field approximation. Under this interpretation, a TAP machine is not a generative probabilistic model, but a deterministic model defining a set of representational magnetizations for a given training dataset. Advantageously, this learning output can be computed exactly in finite time by converging a fixed-point iteration, in contrast to the indeterminate stopping criterion of Markov-chain Monte Carlo sampling. This is a major distinction between the TAP machine and the classical RBM, for which the true probability density function is intractable.
At its core, the TAP machine training consists of arranging the minima, solutions, in the proposed TAP-approximated free energy so as to maximize the correlation between these solutions and the dataset. In our experiments, we demonstrate how to track the growth and geometry of these solutions as a novel way to investigate the progress of unsupervised learning. We also show how to use a trained TAP machine as a prior for inference tasks. The paper is organized as follows. In Sec. II we formally describe the classical binary RBM and review the literature on RBM training and analysis. Subsequently, in Sec. III, we describe our proposed modification of the binary RBM to a model with arbitrary real-valued distributions with bounded support. Next, in Sec. IV, we briefly describe how to apply belief-propagation to perform inference in the setting of real-valued spins.
The details of this approach are pedagogically described in Appendices A & B. In Sec. V we derive the TAP approximation of the real-valued RBM via a high-temperature expansion of a two-moment Gibbs free energy. Then, in Sec. VI, we detail how to convert this approximation to a practical training algorithm. In Sec. VII, we conduct a series of experiments on real datasets, demonstrating how to use the properties of the TAP machine interpretation to provide insight into the unsupervised learning process. We additionally show how to use a trained model for bit-flip correction as a simple example of leveraging a TAP machine for inference tasks. Lastly, in Appendix C, we detail the derivations of necessary distribution-specific functions.
This approach was demonstrated in the contrastive divergence (CD) of  [9] , where very short-chain block-Gibbs sampling was shown to be sufficient for adequate RBM training. The CD approach consists of a sampling chain alternating between samples drawn from the conditional probabilities of each layer, which are dependent on the conditional expectations at the previously sampled layer. Specifically, the conditional probabilities for the hidden and visible units factorize as P (x|h) = i sigm   b i + j W ij h j   , (3) P (h|x) = j sigm c j + i W ij x i , (4) where sigm(y) = (1 + e -y ) -1 is the logistic sigmoid function. In order to learn the parameters of the RBM for a given training dataset, one looks to maximize the following loglikelihood, ln P (x; W, θ) = ln h e Wij xihj + i bixi+ j cj hj -ln Z [W, θ] , (5) via gradient ascent on the parameters W and θ.
g. image in-painting, label recovery  [11] , or collaborative filtering  [12]  by reconstructing missing data with a single visible-hidden-visible step. In truth, the CD-k training algorithm which popularized RBMs, first with binary units  [9] , then Gaussian units  [10, 31]  and finally with arbitrary units  [32] , does not use thermalized samples to evaluate means and correlations. Instead, it focuses on the region of the configura-tion space nearest to the training dataset  [28]  by using short block-Gibbs Markov chains, starting from training data points, to get fast and low variance estimates of moments. However, CD-k is prone to learn spurious minima in configuration space far from the data as it does not explore this region during training  [28]. It also does not systematically increase the true likelihood of training data  [33].
However, this training strategy has been found to be very efficient in the applications mentioned above, which consistently remain close to the dataset in configuration space. One finds that CD falls short for applications which require long-chain MCMC sampling from the trained RBM, as this represents a fundamental mismatch between the training and application of the RBM. In order to address some of theses shortcomings of sampling-based approaches, we now turn our attention to deterministic mean-field approximations of the RBM. The TAP approximation  [23]  for disordered systems relies on the deterministic inference of approximated magnetizations, from which one can obtain estimators of all kinds of observables, starting from the log-partition or free energy. TAP is derived from a small weight expansion of the variational approach and can be considered as an extension of the naïve mean-field (NMF) method (see  [34, 35]  for the original derivation, and  [36, 37]  for pedagogical expositions).
Previous works which have attempted to make use of the NMF approximation of the RBM have shown negative results  [18, 38]. The TAP approximation was first considered for Boltzmann machines in the context of small random models without hidden units in  [39]. In the recent work of  [40] , this approximation was extended to a practical training algorithm for full-scale binary RBMs which was shown to be competitive with Persistent Contrastive Divergence (PCD)  [18]  when applied to real-world datasets. In parallel, other works have used TAP, and the related Bethe approximation, to perform inference on binary Boltzmann machines  [41] [42] [43] [44]. In the next sections, we detail how to re-write the RBM model in the non-binary case, for generalized distributions on the visible and hidden units, similar in spirit to  [32].
V. TAP APPROXIMATION FOR PAIRWISE MODELS While one could utilize the r-BP approach in order to estimate the free energy of a generalized real-valued spin model, as detailed in the earlier section, such an approach might not be desirable in practice. Specifically, if one wishes to solve the inverse learning problem, estimating model parameters from a given dataset, it is necessary to estimate the gradients of the model parameters w. r. t. the model likelihood for each parameter update. Using a steepest-ascent approach, as detailed in Sec. III, requires one to estimate these gradients many thousands of times. For systems of large size N , the r-BP scales quite poorly. Estimating a gradient requires the iteration of the r-BP equations on O(N 2 ) messages. Additionally, one must distinguish between cavity terms (i → j) and marginal terms (→ i).
This task turns out to be closely related to the TAP approach to low-rank matrix factorization  [52] [53] [54] [55] [56]. While it is possible to derive the stationarity conditions for the inferred marginals from the r-BP messages directly by Taylor expansion, we rather focus on the free energy directly that will provide the gradients we require for training the GRBM parameters via a high-temperature expansion we present below. Lastly, we point out that the TAP free energy secondorder (TAP) term depends on the statistical properties of the weight distribution. The derivation presented below assumes independent identically distributed weights, scaling as O(1/ √ N ). This assumption is a simplification, as in practice the weight distribution cannot be known a priori. The distribution depends on the training data and changes throughout the learning process according to the training hyper-parameters.
The range of TAP free energies (gray box ) gives a boundary on the location of F β. Averaging the TAP free energies (dash-dot) provides an estimate of F β. in this approach are proportional to the exponents of each solution's TAP free energy, placing much stronger emphasis on low-energy solutions. However, such an approach is not well-justified in the our general setting of P i , where we expect large deviations from the expectations derived for the SK model. Additionally, while this weighting scheme is shown across the entire set of solutions for a particular random SK model, in our case, we are interested in the solution space centered on the particular dataset we wish to model. Since the solutions are computed by iterating the TAP selfconsistency equations, we can easily probe this region by initializing the iteration according to the training data.
We note that rather than updating over the entire system at each time step, fixing one side at a time has the effect of stabilizing the fixedpoint iteration. For clarity, Alg. 1 is written for a single initialization of the visible marginals. However, as noted in Sec. V. 2, there exist a large number of initializationdependent solutions to the TAP free energy. Thus, in order to capture the plurality of modes present in the TAP free energy landscape, one should run this inference independently for many different initializations. If the use case of the GRBM requires that we only train the GRBM tightly to the data space (e. g. data imputation), it makes sense to fix the initializations of the inference to points drawn from the dataset, a v,(0) = x (m) where m ∈ {1,. , M } , (37) c v,(0) = 0. (38) In order to train the GRBM more holistically, structured random initializations can help probe modes outside of the data space.
(42) In the presented gradients, we make the point that the set of data samples and the set of TAP solutions can have different cardinality. For example, one might employ a mini-batch strategy to training, where the set of data samples used in the gradient calculation might be on the order 10 2. However, depending on the application of the GRBM, one might desire to probe a very large number of TAP solutions in order to have a more accurate picture of the representations learned by the GRBM. In this case, one might start with a very large number of initializations, resulting in a very large number, K M , of unique TAP solutions. Or, contrary, while one might start with a number of initializations equal to M , the number of unique solutions might be K M , especially early in training or when the number of hidden units is small.
MNIST - The MNIST handwritten digit dataset  [63]  consists of both a training and testing set, each with 60,0000 and 10,000 samples, respectively. The data samples are real-valued 28 × 28 pixel 8-bit grayscale images which we normalize to the dynamic range of [0, 1]. The images themselves are centered crops of the digits '0' through '9' in roughly balanced proportion. We construct two separate versions of the MNIST dataset. The first, which we refer to as binary-MNIST, applies a thresholding such that pixel values > 0. 5 are set to 1 and all others to 0. The second, real-MNIST, simply refers to the normalized dataset introduced above. CBCL -The CBCL face database  [64].
VII. 2. Learning Dynamics We now investigate the behavior of the GRBM over the course of the learning procedure, looking at a few metrics of interest: the TAP-approximated log-likelihood of the training dataset, the TAP free energy, and the number of discovered TAP solutions. We note that each of these metrics is unique to the TAP-based model of the GRBM. While it was empirically shown in  [40]  that CD does indeed increase the TAP log-likelihood in the case of binary RBMs, the specific construction of CD is entirely independent from the TAP model of the GRBM. Thus, it is hard to say that a CD or TAP-trained GRBM is "better" in a general case. At present, we present comparisons between TAP GRBMs of varying complexity trained under fixed hyper-parameters settings, as indicated in Table  I. In Fig.
It can be observed that at each level of complexity, the TAP log-likelihood of the data rapidly increases as the values of W quickly adjust from random initializations to receptive fields correlated with the training data. However, across each of the tested models, by about the 20 th epoch the rate of increase of the TAP log-likelihood tapers off to a constant rate of improvement. For reference, we also show a subset of the trained receptive fields, i. e. the rows of W , for each of the tested experiments. Since the full set of receptive fields would be too large to display, we attempt to show some representative samples in Fig. 4  by looking at the extreme samples in terms of spatial spread/localization and activity over the training set. We observe that the trained GRBMs, in the case of both binary-MNIST and real-MNIST, are able to learn both the localized and stroke features commonly observed in the literature for binary RBMs trained on the MNIST dataset  [9, 65].
It is interesting to note that even in the case of real-MNIST, where we are using the novel implementation of truncated Gauss-Bernoulli visible units (see Appendix C2), we are able to observe similar learned features as in the case of binary-MNIST. We take this as an empirical indication that the proposed framework of GRBM learning is truly learning correlations present in the dataset as intended. Finally, we see feature localization increase with the number of hidden units. To date, understanding "what" an RBM learns from the unlabelled data has mostly been a purely subjective exercise in studying the receptive fields, as shown Fig. 4. However, the interpretation of the GRBM as a TAP machine can provide us a novel insight in the nature and dynamics of GRBM learning via the stationary points of the TAP free energy, which we detail in the next section.
Because of this, the numerical evaluation we carry out here would be impractical with sampling techniques. The r-BP and mean-field models of the RBM allow us to directly obtain the modes of the model by running inference to solve the direct problem. Given a diverse set of initial conditions, such as a given training dataset, running r-BP or TAP provides a deterministic mapping between the initial conditions drawn from the data, as in (  37 )-  (38) , and the "nearest" solution of the TAP free energy. If the initial point was drawn from the dataset, then this solution can be interpreted as the RBM's bestmatching internal representation for the data point. If a large number of structurally diverse data points map to a single solution, then this may be an indicator that the GRBM parameters are not sufficient to model the diverse nature of the data, and perhaps further changes to the model parameters or hyper-parameters are required.
Conversely, if the number of solutions explodes, being roughly equivalent to the number of initial data points, then this indicates a potential spin-glass phase, that the specific RBM is over-trained, perhaps memorizing the original data samples during training. Additionally, when in such a phase, the large set TAP solutions may be replete with spurious solutions which convey very little structural information about the dataset. In this case, hyper-parameters of the model may need to be tuned in order to ensure that the model possess a meaningful generalization over the data space. To observe these effects, we obtain a subset of the TAP solutions by initializing the TAP iteration with initial conditions drawn from the data set, running the iteration until convergence, and then counting the unique TAP solutions.
We present some measures on these solutions in Fig. 2. Here, we both count the number of unique TAP solutions, as well as the distribution of the TAP free energy over these solutions, across training epochs. There are a few common features across the tested datasets. First, the early phase of training shows a marked increase of the TAP free energy, which then gradually declines as training continues. Comparing the point of inflection in the TAP free energy against the normalized TAP loglikelihood shown in Fig. 3  shows that the early phase of GRBM training is dominated by the reinforcement of the empirical moments of the training data, with the GRBM model correlations playing a small role in the the gradient of (  14 ). This makes sense, as the random initialization of W ∼ N (0, σ) for σ ≈ 10 -3 implies that the hidden units are almost independent of the training data.
( 45 ) By assuming a factorized P (x) = i m xi i (1 -m i ) 1-xi , where m i might be per-site empirical averages obtained from available training data, the posterior factorizes and we can construct the Bayes-optimal pointwise estimator (OPE) as the average x i P (xi|yi) which is just P (x i = 1|y i ) for our binary problem. Thus, the OPE at each site x i is given as P (x i = 1|y i ) = 1 1 + 1-mi mi × 1-p p 2yi-1 , = sigm ln m 1 -m + (2y i -1) ln p 1 -p. ( 46 ) For a given dataset, the OPE gives us the best-case performance using only pointwise statistics from the dataset, namely, empirical estimates of the magnetizations m i. We can see from Eq. (  46 ) that the OPE either returns the observations, in the case of p = 0, or the prior magnetizations m i , in the case p = 0. 5. In this case of complete information loss, the worst case performance is bounded according to the deviation of the dataset from its mean.
We present the performance of the OPE in Fig. 8  for the binary-MNIST dataset. This makes the OPE a valuable baseline comparison and sanity check for the GRBM approximation of P (x). As the GRBM model takes into account both pointwise and pairwise relationships in the data, a properly trained GRBM should provide estimates at least as good as the OPE. The k-Nearest-Neighbor (k-NN) algorithm represents a different heuristic approach to the same problem  [68]. In this case, the noisy measurements are compared to a set of exemplars: the training dataset. Then, according to some distance metric such as MSE or correlation, one finds the k exemplars with minimal distance to the noisy observations to serve as a basis for recovering the original binary signal. One can use some arbitrary approach for fusing these exemplars together into the final estimate, but the simplest case would be a simple average.
Additionally, we see that it performs better than the OPE in the regime p > 0. 2. This can be explained since we can think of the k-NN approach as implicitly, though indirectly, taking into account higherorder correlations in the dataset by naïvely returning data exemplars; all the estimates trivially posses the same arbitrarily complex structure as the unknown signal. Using the GRBM, we can hope to capture the best points of both approaches. First, we hope to perfectly estimate the original signal in the case p = 0. Second, we hope to leverage the pairwise correlations present in the dataset, returning estimates which retain the structure of the data even as p → 0. 5. For GRBM denoising of the BSC, we no longer have a factorized posterior. Instead, we have the GRBM likelihood given in Eq.
In the case N h = 500, the TAP inference gets caught in spurious and undesirable minima as p increases. case of N h = 500, the TAP inference results in either nearly zero-modes, or in very localized ones. This would seem to indicate that landscape of the TAP free energy around the initializations is becoming more unstable as the density of solutions increases around it. Additionally, since the TAP free energy landscape was only probed using data points during training, the clustering of solutions around noisy samples remains ambiguous. Augmenting the initializations used when calculating the TAP solutions for the gradient estimate with noisy data samples could help alleviate this problem and regularize the TAP free energy landscape in the space of noisy data samples.
For example, this can be seen in the modified Hamiltonians used for real-valued data  [69, 70]. This construction allows us to consider binary, real-valued, and sparse real-valued datasets within the same framework. Additionally, one can also consider other architectures by changing the distributions imposed on the hidden unit. Here, we present experiments using only binary hidden units, but one could also use our proposed framework for Gaussian-distributed hidden units, thus mimicking a Hopfield network  [2]. Or, also, sparse Gauss-Bernoulli distributed hidden units could mimic the same functionality as that proposed by the spike-and-slab RBM  [45]. We have left these investigations to further works on this topic. Our proposed framework also offers a possibility to explore the statistical mechanics of these latent variable models at the level of TAP approximation.
The stronger this local potential, or the smaller the weights, the more favorable the model is to the r-BP inference. This observation mirrors those made in  [41] , however, here the authors make the observation that in this setting, BP based on small-weight expansion for binary variables fails to converge. In our case, without taking some form of regularization, the inference fails entirely. Thus, large magnitude couplings W il must be backed with a high degree of evidence at site i and l. This property could be utilized for the inverse learning problem, where one must learn the couplings W given a dataset, in order to constrain the learning to parameters which are amenable to the r-BP inference. One direct manner to create probability distributions from otherwise unbounded continuous functions is via truncation.
Even though the training is performed so as to maximize the TAP log-likelihood, the AIS and PL estimates also increase, indicating the measures to be consistent. studying how consistent it is with other estimators and surrogates. We present two of them here: the pseudolikelihood (PL), and the log-likelihood estimate provided by annealed importance sampling (AIS)  [19]. For this experiment, we compare log-likelihood estimates from TAP, PL, and AIS over the course of 100 epochs of training performed on a single binary-binary RBM with 500 hidden units using the binary-MNIST dataset as training data. The training is performed by maximizing the TAP loglikelihood, and each of the three approaches produces either an estimate or a surrogate to the true log-likelihood at the end of each epoch (as in Fig.
3 FIG. 11. Training performances over 3000 training epochs for a 2-hidden layer (left) and 3-hidden layer (right) deep Boltzmann machines (DBMs) on the binarized MNIST datasets. Both models were pretrained for 50 epochs with the same learning rate of 0. 001. The training performance is measured as the normalized (per unit) TAP log-likelihood for the test images (blue) and the train images (orange).