dataesr/scientific-paragraphs-categorization huggingface dataset:
We plot in Fig. 2 (c) the histogram of Pearson correlations between the label and hidden-unit inputs, ρ μ ¼ huðvÞI μ ðvÞi D -huðvÞi D hI μ ðvÞi D ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi hI μ ðvÞ 2 i D -hI μ ðvÞi 2 D q ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi huðvÞ 2 i D -huðvÞi 2 D p : ð5Þ For some datasets (e. g. , KH sequences), hidden units have low correlations to the label. Changing the label identity of the generated data requires us to act on the states of all these hidden units in a concerted manner. In other cases, such as the Ising model and MNIST, a number of units exhibit higher correlations with the labels; see right tails of distributions in Fig.
In practice, from finite datasets one can extract reliable estimates only of the top components of q ð2Þ , while the empirically observed lower components will be dominated by noise. The Marchenko-Pastur (MP) law  [34]  describing the spectrum of correlation matrices in the null model case of independent variables can be used to estimate the thresholds between eigenvalues dominated by noise and eigenvalues reflecting the presence of structure in the data. The MP spectrum predicts that all eigenvalues λ located in the range ½λ -; λ þ have to be discarded, with λ AE ¼ ð1 AE ffiffi ffi r p Þ 2 , where r is the ratio of the numbers of variables and samples. As an example, for the MNIST0=1 dataset, we estimate λ þ ≃ 1. 6 for both 0 and 1 digits. Out of the 784 eigenvalues of q ð2Þ , only 60  (61)  are larger than this bound for the 0s dataset (1s).
IV. APPLICATION TO FACE IMAGES A. Learning with standard RBMs We first illustrate our approach on the CelebA dataset of celebrity face images  [23]. Since we choose to work with binary RBMs for simplicity, we first convert the images to binary black-and-white pixels of resolution 64 × 64, following a procedure similar to Ref. [35]  and detailed in the Supplemental Material  [26]  Appendix B. Using the annotations available in the dataset, we choose the presence or absence of eyeglasses and smiling or not smiling as our labels. We compute the vector q defined by Eq. (  7 ) for each one of these two labels. Figure  5 (a) shows sample images arranged in increasing value of their projection along this vector, as well as the histograms of these projections over the dataset for each label. Next, we train a standard RBM on this dataset.
On the generative power of the fully constrained RBM Configurations sampled from the fully constrained RBMs in Fig. 8 (a) (bottom) tend to be blurred mixtures of digits (0 and 1). In this case, the data are in fact a mixture of two widely separated distributions associated with 0s and 1s. This is reminiscent of configurations of opposite magnetization in the Ising model at low temperature in Sec. V D, and the sampled blurred digits are in analogy to the "intermediate" configurations of zero magnetization that the fully constrained RBM samples in that case [Fig. 7(c ) top]. We however see that in the Ising model, the configurations sampled from the fully constrained RBM still carry relevant information in higher-order statistics, e. g. , as shown by the behavior of the heat capacity; see Fig.
We show in Fig. 8(d ) numerical experiments illustrating the effects of such manipulations. We initialize the RBM with a digit [0 in Fig. 8(d) ] extracted from the MNIST0=1 dataset, and sample new configurations through alternate Gibbs samplings. As with standard RBM, the samples vary over time, but the digit class remains unchanged. We then flip the state of the hidden unit [middle of Fig. 8(d) ]. As a consequence, the resulting visible configuration converges to the other digit class after some short transient (see top part of panel). To evaluate the quality of the generated digits, we train two RBMs only on 0s or 1s, respectively, and evaluate the log-likelihoods of the generated digits on two standard RBMs, one trained with 0 digits only, and another trained on 1s only. These two machines provide expected reference scores for 0s and 1s.
We select this family in our work as it has a sufficient number of eukaryotic and bacterial sequences available in the Pfam database  [25]. The PF00013 family of homologous sequences includes approximately 11 000 bacterial sequences and approximately 38 000 eukaryotic sequences of the KH domain. After aligning, removing insertions, and retaining only columns with less than 50% gap (deletions) content, the sequences end up having a common length of L ¼ 62 amino acids. As the taxonomic origin of every sequence can be simply queried through the Uniprot database  [48] , we define label u ¼ 0 and 1 for, respectively, bacterial and eukaryotic proteins. To reduce common ancestry bias, the sequences are weighted according to their dissimilarity to other members of the same family  [49, 50] : The weight assigned to a sequence is proportional to the inverse of the number of sequences in the family with a Hamming distance smaller than 20% of the sequence length.
The RBM correctly observes these variations. (4) Iso10 is highly conserved in bacteria, while in eukaryotes this site is not conserved, admitting, in particular, Val, Ala. These examples suggest that the RBM can sample each subfamily, conditioned on the value of h Ã. Next, we sample the RBM starting from one bacterial or one eukaryotic sequence in the dataset as the initial condition, and with h Ã set to the value matching the initial condition. After some steps, the value of h Ã is flipped, and we monitor the dynamical evolution of the generated samples. Figure  10(f)  shows the probability that generated sequences are eukaryotic or bacterial, according to a linear classifier achieving AUC > 0. 9 on held-out test data (see Supplemental Material  [26]  Fig. S4 ). Figure  10 (g) shows a magnified view of the classifier probabilities and the log-likelihood in the vicinity of the hidden-unit switch.
VIII. ROBUSTNESS AGAINST THE SCARCITY OF LABELED DATA One important advantage of our approach is that labeled data are only necessary to estimate the vector q ð1Þ (7) used in the first-order constraint (6), or the matrix q ð2Þ (10) in the case of the second-order constraint  (9). Having determined q ð1Þ or q ð2Þ , the training of the RBM benefits from additional unlabeled data, and in this regard, our model is semisupervised. This property is useful in many real applications, where labels are assigned by humans, are costly to obtain, and thus available for only a small fraction of the data. An example is the KH domain protein-sequence dataset considered in Sec. VII, where we are able to collect reliable taxonomic labels for only 10% of the sequences. To better understand the amount of labeled data needed for our approach to be effective, we conduct further numerical experiments in which the fraction of labeled data is progressively decreased.
We consider below the linear constraint and the MNIST0=1 data for the sake of simplicity. Similar results for the KH domain are reported in Supplemental Material  [26]  Fig. S10. Since q ð1Þ becomes trivially zero when there are no data in one of the label classes, we consider balanced subsampled labeled datasets with equal numbers of labeled examples in each class. Figure  11(a)  shows the average overlap between vector q ð1Þ computed on such a subsampled labeled dataset (referred to as q ð1Þ sub ), and the vector q ð1Þ computed on the full labeled dataset (denoted by q ð1Þ full ), as a function of the number B of labeled examples available, divided by the dimension of the data N. Here, the overlap is defined by  ϕ ¼ q ð1Þ full • q ð1Þ For each given number of labeled examples, we consider 100 random realizations of the subsampled labeled dataset and estimate the average of ϕ over these realizations.
It can be seen from Fig. 11 (a) that the overlap never drops below approximately 0. 6. This result can be understood by considering the separation between the two classes of data (see inset in the figure). Writing the covariance matrix conditioned on the class label C ðuÞ ij ¼ hv i v j jui -hv i juihv j jui; ð28Þ as well as the mean data vector associated with each class v ðuÞ i ¼ hv i jui; ð29Þ we can derive a simple estimate connected to the average separation between the classes, v ð0Þ -v ð1Þ and the variances inside each class TrC ð0Þ , TrC ð1Þ (see Supplemental Material  [26]  Appendix H for a derivation) that writes hϕi ≈ 1 þ 1 B TrðC ð0Þ þ C ð1Þ Þ kv ð0Þ -v ð1Þ k 2 -1=2 ; ð30Þ where B is the total number of labeled examples, and the average is taken over all labeled datasets with B=2 examples in each class.
Thus, the overlap increases with the separation between the classes (v ð0Þ -v ð1Þ ) and decreases if the classes have large variances (TrC ð0Þ , TrC ð1Þ ), as depicted in the inset   (c) h* 1 -h* q (full) q ( s u b ) (full) q (sub) q (sub) (a) (b) sub (computed on a subsampledlabeled dataset) and q ð1Þ full (computed on the full dataset) plotted as a function of the number of labeled examples in the subsampled dataset divided by the dimension (28 × 28 ¼ 784 for MNIST). An average of over 100 random realizations of the subsampled dataset is taken. The black solid curve shows the empirical result, while the dashed green curve is the theoretical estimate  (30). Inset shows a diagram of how class separation relates to the overlap in connection to  (30). (b) For the pink and cyan dots of (a), we plot an example of the obtained vectors q of  Fig.
To further underline the advantage of our method with respect to supervised learning in a situation with few labeled data, we train normal RBMs on the subsampled labeled data, specializing on 0 or 1 digits only. As expected for the small amount of training data, these models tend to overfit. This is shown in the histograms of log-likelihood assigned to training and a withheld validation dataset in Fig. 11(d ) (top for 0 digits and bottom for 1s). The gap in the average log-likelihood of training and validation data (black and green vertical lines, respectively) is quite large, in both cases, indicating overfitting. In contrast, the partially constrained RBM (the same from the cyan dot in the previous panels of the figure) uses both the few labeled data and the large quantity of unlabeled data to avoid overfitting, and we show the log-likelihood histograms for training and validation data in Fig.
First row shows numerical estimates of the log-likelihood using RBMs with binary hidden units, along with the costs of applying Eq. (  6 ) partially or on the full hidden layer. Bottom row shows analytical results obtained in a RBM with one hidden spin unit and the remaining Gaussian hidden units (Fig. 13 ). First column shows the legend: black for the unconstrained model, red for models with all hidden units constrained, and blue for models with the constraint acting on all but one hidden unit. Subsequent columns show the results for the three datasets considered: MNIST0=1, two-dimensional Ising model (L ¼ 64), and the KH protein domain. The discontinuous arrows in the first panel highlight the likelihood costs of partial label erasure (red) and disentanglement (blue). two bounds sandwiches the true likelihood value and ensures that sampling has converged.
The results are shown for the Ising model, MNIST0=1, and PF00013 datasets considered in this work in the top row of Fig. 12. We do not consider CelebA for computational convenience. We first measure the likelihood costs ΔL part erasure [see Eq. (  11 )] for making labels inaccessible to linear discriminators with the fully constrained architecture (red bars or dots). In all datasets, the labels considered are relevant to the nature of the data, and the costs (per data configuration) induced by the constraints on the weights are significant; see Table  I. The relation between label relevance and the likelihood cost is nicely portrayed in the two-dimensional Ising model dataset. At low β, the data are essentially random, and the magnetization is mostly irrelevant to determining the probability of a configuration.
In this regime, erasing label information has little likelihood cost. As the inverse temperature increases, the magnetization becomes more relevant, and it becomes necessary for the model to account for it to achieve good likelihood. In consequence, partially erasing the magnetization in this regime results in a large likelihood loss. The top row of Fig. 12  furthermore shows the values of the log-likelihoods after releasing one hidden unit (blue bars and dots). The log-likelihood loss with respect to the unconstrained RBM ΔL disent in Eq. (  12 ) is guaranteed to be non-negative. In practice, for the MNIST0=1 and Ising model datasets, and to a lesser extent for the KH domain, we estimate this cost to be small; see Table  I. These results are consistent with the ability of the released RBM to fit and generate high-quality data in the three cases, as shown in previous sections.
We again stress that our approach combines the unsupervised nature of the RBM with constraints that are derived from labeled data. Therefore, our model can be said to be semisupervised. We show how this synergy results in a model able to work in a regime with a limited amount of labeled data. This result is important as, in many cases, labeled data are much more expensive to obtain than unlabeled data: Data have to be annotated by humans (for instance, in the PF00013 dataset of the KH domain sequences, taxonomy labels are available for less than 10% of the sequences), or costly experiments have to be done to get the label (this is the case for most biological data, which often require complex biophysical or biochemical characterizations). We demonstrate the effectiveness of this approach on four datasets from diverse domains: the CelebA dataset of face images  [23] , the Ising model from statistical physics, the MNIST collection of handwritten digit images  [24] , and protein sequences of the KH domain family  [25].
CelebA  [23]  and MNIST  [24]  are popular benchmark datasets in machine learning. In MNIST, the labels are straightforwardly associated with the digit identities. On this dataset, we show that RBM can be trained to associate one or few controlling hidden units with each digit class, which can be manipulated to sample and transition between classes. In CelebA, the labels correspond to subtle attributes of face images, like facial expressions (smiling or not smiling), or adornments (presence of eyeglasses). Even for this complex dataset, RBMs can sample good-looking images and are able to concentrate these attributes over few hidden units. The two-dimensional Ising model is a very well-studied system in statistical physics, with a precisely characterized phase transition controlled by the temperature.
Concentrating information about important features of the data into one or few hidden units of the RBM could a priori be detrimental to the ability of the model to fit the data for two reasons. First, introducing constraints on the weights is expected to impact (decrease) the log-likelihood of the data generated by the RBM. We estimate the losses in log-likelihood due to partial erasure and disentanglement for several datasets. The cost of partial erasure is related to the relevance of the label, as clearly illustrated in the dependence on temperature in the Ising model data. Remarkably, we find that disentanglement is achieved with a small relative likelihood loss, evidencing the robustness of the approach. Furthermore, when the data can be approximated as a mixture of two Gaussian distributions, we show how the log-likelihood losses could be analytically calculated and establish a connection between the likelihood costs for erasure or disentanglement and the Poincaré separation theorem.
Second, the few (often, single) released hidden units encode label-associated features in a prototypelike way. In the case of linear constraints, released weights are aligned with the q ð1Þ vector, equal to the relative difference between the centers of mass of the two label classes; see Fig. 5 (a) for an illustration on CelebA. It is however widely believed that prototypelike representations are poorer than compositional ones, in which multiple features associated with many hidden units can be combinatorially combined to create high-quality and diverse data  [61]. From this point of view, forcing some hidden units to generate prototypes could appear counterproductive. It is nevertheless a very effective way to drive class switching; see, for instance, Fig. 6. In addition, all the important features defining the data distribution are learned by the vast number of other (constrained) hidden units, which, in turn, can be combined together to collectively participate in the data generation process.
We also emphasize that, while a few hidden units capture enough label-associated features to manipulate and drive the label values, this does not mean that they concentrate all the information about the label. As clearly shown in Fig. 7 (f) for Ising and Fig. 8 (b) for MNIST0=1, there remains substantial information about the label in the constrained hidden units accessible to deep decoders. Hence, label-associated features are residually encoded in a combinatorial way by the RBM. While disentangling and manipulating representations through our "partially constrained" RBM approach offers clear advantages in terms of usability and interpretability, the other architecture we consider in this work, the socalled "fully constrained" RBM may also be of interest in practical applications. Informally speaking, fully constrained RBMs are appropriate to model the features in the data orthogonal to the ones associated with the label under consideration.
We show that fully constrained RBMs remain generative in two examples (CelebA and PF00013), where samples resemble data configurations with ambiguous class identity. In the MNIST0=1 and Ising model examples, however, the fully constrained RBM generates samples markedly different from the data (zero magnetization in the Ising case, and blurry mixtures of 0s and 1s for MNIST). We attribute this to the fact that in these latter cases, the datasets corresponding to the two values of the label are widely separated. However, as we show in the Ising case, information is preserved in higher-order moments of the samples (e. g. , heat capacity). Another example is shown in Supplemental Material  [26]  Fig. S6 , where a fully constrained RBM trained on zero MNIST digits in black or white backgrounds generates zeros encoded in the correlations between neighboring pixels.
4 FIG. 2. 42 FIG. 2. Datasets considered in the paper and entanglement of representations. (a) CelebA dataset of face images [23] ; two-dimensional Ising model; MNIST0=1 database of handwritten digits [24] ; multiple sequence alignments from the Pfam PF00013 family of the KH domain. (b) Samples generated by different RBMs trained on each dataset. See Supplemental Material [26]  Appendix A 6 for the architectures of the RBMs used in each case. (c) Histogram of the absolute value of the Pearson correlations between hidden-unit inputs and the chosen label; see Eq. (5). Smiling or not smiling for CelebA, sign of the magnetization for the Ising model, whether the digit is a 0 or 1 for MNIST0=1, and whether the KH sequence is from bacterial or eukaryotic origin.
ð1Þ sub in comparison to q ð1Þ full. (c) Label manipulation using the subsampled q ð1Þ sub in the two cases. (d) Histogram of log-likelihoods of (subsampled) training and withheld dataset for a RBM trained on a subset of 0 (top) or 1 (bottom) digits, corresponding to the labeled datasets used in the cyan dot in the previous panels. The black and green vertical lines indicate the average values. (e) Histogram of loglikelihoods of training and withheld dataset of the partially constrained RBM in the cyan setting of the previous panels.
A totally different line of research in statistical mechanics is focusing on the properties of artificial neural networks  [12] [13] [14]. A simple way to think about them is as devices performing a non-linear regression task. In a huge number of cases, networks are trained such that their output to a given input pattern of a dataset, is the desired one (the label of an image for example). Therefore they can be thought to be doing regression tasks, which can be achieved with many cost functions, the simplest one being the square loss. In this case for each point in the dataset, the cost associated to a configuration of the network which does not provide the right label for that point, is defined as the square between the current output and the expected one. Therefore training the network corresponds to finding the set of parameters such that for each data point the network provides the right label.
Instead the unsatisfiable phase is glassy and the landscape is a high-dimensional rough arrangement of minima and saddles. The liquid phase in the regression setting denotes the overparametrized phase and the rigidity transition is nothing but the interpolation point in which the network is becoming too small to fit all dataset. The same picture holds for classification tasks and a parallel with the jamming transition of particles has been done in recent years  [17] [18] [19] [20]. The main purpose of this work is (i) to describe the solution of the infinite dimensional dynamics of the class of models described in  [15]  and (ii) to discuss how to solve numerically the corresponding equations. We will detail the derivation and the structure of the dynamical mean field theory (DMFT) equations  [21]  describing the dynamics in the infinite size limit.
Our results clearly confirm these findings in the context of the model considered in this manuscript. Looking at Fig. 4 , one would conjecture also that the gradient descent satisfiability transition p G D J is located in the range 1. 8 < p GD J < 2 which must be compared with the thermodynamic satisfiability transition point located at p J ≃ 1. 87 estimated numerically in  [15]. In order to estimate better p G D J we have considered the following procedure. Given that the energy in the SAT phase decreases exponentially fast, we have measured the relaxation time τ(p 0 ) as the time it takes for the dynamics to bring the system to an energy below a threshold value. We fix this value to be e th = 10 -6. In the left panel of Fig. 5  we plot τ(p 0 ) at α = 1/4. We plot data from two different datasets obtained by changing the integration timestep dt to confirm that our estimate of τ is weakly dependent on this integration parameter.
• A systematic study of the algorithmic SAT/UNSAT transition with the gradient descent dynamics can be done. In this work we have showed that the algorithmic satisfiability transition is close to the thermodynamic one and this was already found in finite size numerical simulations in  [15]. Furthermore we have shown that the relaxation time follows a power law divergence close to the transition and we have measured the corresponding critical exponent. It would be interesting to compute this exponent explicitly and to compare it with Vertex or Voronoi models to understand the effect of the dimensionality of the system on the critical properties of the transition point. • In deep learning applications, one typically solves the optimization problem via stochastic gradient descent (SGD) which is an algorithm where the gradient of the loss is ap-proximately estimated by a time-changing subset (or minibatch) of the full dataset.