dataesr/scientific-paragraphs-categorization huggingface dataset:
These drawbacks should be solved in the near future to realize robust high-throughput QMC calculations for large systems with optimized (i. e. , beyond-DFT) nodal surfaces. The S22 dataset was developed by Hobza et al. for testing interaction energies for small complex systems  31. Figure. 3 shows the benchmark result for the S22-set. The corresponding numbers are shown in Tab. S-X. We employed the cc-pVQZ basis set with the accompanied ccECP  [37] [38] [39] [40]  pseudo potentials. For the DFT calculations, we used the PySCF package (v2. 0. 1)  27, 28  with PZ-LDA  64  exchange-correlation functional. The obtained MAD is 0. 610 kcal/mol. A subset of the S22-benchmark was studied by Dubecky et. al. 75, 76. They extracted a subset of the S22 benchmark test, ammonia dimer, water dimer, methane dimer, ethene dimer, ethene-ethyne, benzene-water, benzene-methane, and benzene dimer (T-shape).
78. They used quadruple ζ valence GTO basis sets fully optimized for the ECPs developed by Ovcharenko et al. 79. Their obtained MAD (0. 68 kcal/mol) is very close to ours (0. 61 kcal/mol). The A24 dataset is a set of non-covalent systems large enough to include various types of interactions  32. The dataset was intended for testing accuracy of computational methods which are used as a benchmark in larger model systems. Figure. 4 shows the benchmark result for the A24-set. The corresponding numbers are shown in Table  S -XI. We employed the cc-pVQZ basis set with the accompanied ccECP  [37] [38] [39] [40]  pseudo potentials. For the DFT calculations, we used the PySCF package (v2. 0. 1)  27, 28  with PZ-LDA  64  exchange-correlation functional. The obtained MAD is 0. 315 kcal/mol. The full set of the A24-benchmark was studied by Dubecky et.
al. 76. They investigated the effects of the basis set, Jastrow factor, and optimization protocols. They finally obtained MAD of 0. 15 kcal/mol with the single-determinant trial WFs of Slater-Jastrow type using B3LYP orbitals and aug-TZV basis sets accompanied with the ECPs developed by Burkatzki et al. 42. Their MAD (0. 15 kcal/mol) is very closed to the value reported in this study (0. 315 kcal/mol). The SCAI dataset is developed to benchmark interactions between amino acid side chains  33. The dataset contains a representative set of 24 of the 400 (i. e. , 20 × 20) possible interacting side chain pairs. Figure. 5 shows the benchmark result for the SCAI-set. The corresponding numbers are shown in Table. S-XII. We employed the cc-pVQZ basis set with the accompanied ccECP  [37] [38] [39] [40]  pseudo potentials.
For the DFT calculations, we used the PySCF package (v2. 0. 1)  27, 28  with PZ-LDA  64  exchange-correlation functional. The obtained MAD is 0. 402 kcal/mol, which is as small as the S22 and A24 benchmark sets. To the best of our knowledge, no one has benchmarked the SCAI data set using QMC. Among the systems, 704-DH1 and 705-DHN1 show the largest deviations, -1. 33 (26) kcal/mol and -2. 15 (26) kcal/mol, respectively.
Benchmarking Equations of State in Solids We report the Equation of States (EOSs) of 12 solids computed at the VMC and LRDMC levels by TurboWorkflows. Such a benchmark has been done for several crystals to test the accuracy of QMC calculations  87, 88. Table  VIII  shows the Crystallography Open Database(COD)-IDs  89, 90  of the 12 crystals computed in this demonstration. The 12 crystals were chosen because Ref. 91 summarizes experimental lattice parameters at 0 K with the zero-point energy subtracted, which are directly comparable with our results. First of all, we carefully checked the basis-set convergence since PySCF, which generates trial WFs for the subsequent TurboRVB calculations, employs the localized basis set also for the periodic systems. In other words, the convergence is sometimes difficult to be achieved unlike the plane-wave one.
VII. CONCLUSIONS In this paper, we describe the features of the recently developed TurboGenius and demonstrate its applications. Tur-boGenius is a collection of python wrappers for the ab initio QMC code, TurboRVB. The users can combine the modules implemented in TurboGenius with their python scripts to manage QMC tasks in a python script. TurboWorkflows, which is implemented using TurboGenius, is a python package realizing QMC workflows. TurboWorkflows enables one to run sequential QMC calculations fully automatically and manage file and job transfers from/to cluster machines. As demonstrated in this paper, these Python packages are particularly helpful in performing validations of the methods and algorithms and in conducting benchmark calculations for various materials. In terms of future works, for example, generating a materials database with the ab initio QMC would be a very intriguing work, considering the recent successes of the DFT-based materials databases.
As mentioned in the introduction, the importance of data provenance and data curation in the field of materials science has increased thanks to the development of information science and technology. In this regard, an accurate QMC database can be utilized, for instance, for the construction of machine learning potentials with accuracy exceeding DFT-based ones and for training machine learning exchange-correlation functionals. A package for high-throughput electronic structure calculations is an infrastructure technology in the materials science community. Thus, it should be continuously developed and maintained as an open-source package for the long-term perspective.
ACKNOWLEDGMENTS K. N. is grateful for computational resources from the Numerical Materials Simulator at  National Institute for Materials Science (NIMS). K. N. and M. C. are grateful for computational resources of the supercomputer Fugaku provided by  RIKEN  through the  HPCI System Research Projects  (Project IDs:  hp200164 ,  hp210038 ,  hp220060 , and  hp230030 ). K. N. acknowledges financial support from the  JSPS   Overseas Research Fellowships , from  Grant-in-Aid for Early Career Sci-.
The music industry is now more complex and competitive than ever before. In recent years, the search for collaborations with other artists has become a common strategy of musicians to maintain their presence in the sector. Besides, existing music streaming services such as Spotify have exposed large data feeds that can be used to develop innovative services within the realm of music. In this context, the present work introduces PRESTO, a novel recommendation system to suggest musicians new collaborations with other artists by means of an ensemble of Graph Neural Networks. The system is fed with an heterogeneous graph representing the time evolution and the stationary aspects of a musician's career. Finally, the proposal has been evaluated with a dataset comprising more than 200,000 artists, with an average F1 score above 0.
This calls for a hybrid deep learning architecture combining different types of convolutional and recurrent layers to capture the static and timeevolving features of the artists. To do so, the proposed system is founded on an heterogeneous graph to model the relationships among artists, songs, music genres, their latent features and their temporal evolution. Next, an ensemble of different types of Graph Neural Networks (GNNs) is used to anticipate future relationships by means of a link-prediction approach. To feed the model, we have used a large dataset from Spotify and Last-FM based on collaborations among multiple artist covering a 17-year period. Then, the predicted new links can be used as recommended pairwise collaborations among artists to better define the future development of an artist's career.
Related Work The analysis of streaming music data has been the subject of different lines of research within the field of music-related recommendation systems  [9]. First of all, there are works that have focused on analysing the built-in song recommenders included in platforms such as Spotify or Apple Music from different points of view. For instance, authors in  [10]  evaluate the satisfaction level of the Spotify recommender among different users, whereas the work in  [11]  points out the gender bias of some of these recommendation systems. Another prominent course of action has been the development of ad-hoc recommendation systems to suggest new songs based on listeners' preferences or contextual factors. In that sense, the usage of deep learning techniques is remarkable due to their capacity of extracting latent factors of music items in an isolated or sequential manner  [12].
Datasets The Next, we fused both feeds by removing duplicates giving rise to a single dataset D comprising 1,354,932 tracks, 259,698 unique artists and 2,166 music genres covering a 17-year period from 2005/02/18 to to 2022-10-11. The artists included in this dataset will be represented as the set A, the genres as G and the songs as S in the following sections. Last, for each song in D, we extracted 10 features (represented as F s from now on) from the Spotify Developer Platform (SDP)  5. These features are related to the song's audio properties (loudness, speechiness and instrumentalness), context (liveness and acousticness) and mood (danceability, valence, energy, and tempo), along with its release date as the tenth feature  6. Moreover, for each single artist, we extracted 3 features (represented as F a ), namely 1) his/her number of followers in Spotify, 2) associated music genres and 3) popularity score.
As an illustrative example, if artists A and B collaborate together on a song S, then the graph will comprise a collaborate edge between A and B and a creates edge from A to S and from B to S. As indicated in detail in sec. 4, the latter type of relationship is the target edge that PRESTO will focus on predicting as a result of the recommendation process. Given the dataset D, this heterogeneous graph comprises 259,698 artists, 1. 354,935 songs and 2,166 genres, all of them as nodes. Moreover, it comprises 202,759 has relationships, 2,575,996 is, 1,509,557 creates and 337,910 collaborates ones.
Temporal split of the heterogeneous graph In order to capture the musical evolution of all the artists, we split the global dataset D (see sec 3. 1) in 30-day slices. The rationale of this time granularity is that, as Fig. 9  shows, songs in the dataset D tend to be released equally by month of the year, from February to November. Note that the number of released songs was much higher in January than in the other months, whereas it was lower in December. In order to smooth such outliers, each slice D i ⊂ D, i = [1. 12] comprised the songs whose release date fitted into the time range from the 15th day of i-th month to the 15th day of the following month. In this manner, the songs released in January/December were distributed into 2 different slices instead of one. Next, we composed an heterogeneous time graph G t i (see sec.
Furthermore, HC refers to a heterogeneous graph convolution operator that uses a particular instance of the GraphSAGE model  [29]  for each edge type. As the input graph G t i comprises 3 different node types, the final embedding E t i is just the concatenation of the h t i,artist , h t i,genre and h t i,song. This way, it encodes the latent representation for each node type (artists, songs and genres) based on its temporal evolution. By means of this heterogeneous approach, we allow the HeteroConvLSTM module to learn weights for each node and edge type. This is instrumental given the diversity sub-graphs comprising G t i as we showed in sec. 4. 1. On the other hand, the aggregated graph G agg i is also processed by a heterogeneous version of the GraphSAGE model. In brief, the embedding generation in this model relies on an incremental aggregation of information from neighbors to compose a node representation.
Model Parameters Table  1  shows the key parameters of PRESTO used in its evaluation. The last row of this table shows that the GAT layer generated embeddings with 120 features. Hence, the final collaboration matrix C i took | A | ×120 dimensions. We can also see that we used a sampling rate of negative edges of 1. 0 during the training. This means that we synthetically generated the same number of negative edges, representing non-existing collaborations, than the number of positive edges reflecting an actual collaboration among artists. Hence, we added to each ground-truth matrix C δ i+∆ the same number of non-existing collaborations (labelled as 0) as true ones so as to enrich the learning process of the model. In implementation terms, PRESTO was developed using the Python 3. 8 programming language, along with Pytorch Geometric  [31]  and Pytorch Geometric Temporal  [32]  as the primary third-party libraries.
In particular, the system utilizes a heterogeneous graph to model the relationships among artists, songs, music genres, and their temporal evolution. It is trained on 10 songs' features from two music datasets, namely Spotify and LastFM, including audio properties, context and mood related to the songs. The results show a F1 score of 0. 752 in average, being able to predict collaborations within a one-year time. Morevoer, an ablation study has been conducted to demonstrate that the PRESTO architecture outperforms other simpler variations of the same architecture. As a result, PRESTO could be considered an interesting tool to provide valuable recommendations for artists and music companies, facilitating the production of new tracks and supporting artists in their career development. There are several directions to further improve and expand PRESTO.
Acknowledgments This research was funded by the  National Autonomous University of Mexico  (UNAM) through Projects  PAPIIT IV200321  and  PAPIIT IG101221 , and the  Mexican-French scientific cooperation program SEP-CONACYT-AUIES-ECOS-NORD Francia  through project  FONCICYT 296672 in Mexico  and project M18AU01 in France. M Sc. Ceres Perezvargas y Castor, ( CVU 771792 ) gratefully acknowledges  CONACYT  for her PhD Scholarship. The authors thank  Olivia Zamora ,  René Alcalá ,  Maricarmen Salazar ,  Lucy Mora , and  Ulises Loredo of the Laboratorio Nacional de Geoquímica y Mineralogía (LANGEM) ,  Mario Rodríguez  of the  Laboratorio Universitario de Nanotecnología Ambiental (LUNA) UNAM  for their technical support,  Mario Cayetano Salazar , of the  Laboratorio de Impactos de Procesos Naturales y Antrópicos sobre el Territorio (LIPNAT) UNAM  for his sampling support,  Gustavo Gutiérrez  and  Jorge Marquez , for their support in revising the English wording and style of the manuscript.
Further, we propose a simple yet effective method for selecting hard negatives during the training phase. In extensive series of experiments, we study the impact of a number of parameters on the quality of the learned representations (e. g. the batch size). Simulation results show that the proposed solution outperforms several competing approaches on various large-scale text classification benchmarks without requiring specialized architectures, data augmentations, memory banks, or additional unsupervised data. For instance, we achieved top-1 accuracy of 61. 94% on the Amazon-F dataset, which is 3. 54% above the best result obtained when using the cross-entropy with the same model architecture.
Contrastive learning was also shown to achieve state-of-the-art performance on many large-scale benchmark datasets. Contrastive learning is a particular form of a Siamese neural network, which consists of two or more identical subnetworks, each producing a vector representation of its re- Maximizing the similarities between each pair of augmented views can result in a trivial solution where all representations are equal to other; this is referred to as collapsing problem. Several approaches have proposed to solve the collapsing problem, one of which is contrastive learning. This latter prevents the undesirable trivial solution by contrasting between positive (similar) examples and many negative (dissimilar) examples. These methods explicitly aim at training a neural network to learn embeddings by pulling together the representations of augmented views, while pushing away the representations of augmented views of different data instances (negative examples), often by using noise-contrastive estimation.
The most common strategy is to uniformly sample from the training dataset using examples either from the current batch or from a memory bank. However, it has been observed empirically that contrastive learning methods still suffer from dimensional collapse  [38]. In this paper, we propose a supervised contrastive learning framework for text multi-class classification tasks. Inspired by the recent success of joint embedding approaches for learning representations in a self-supervised setting  [11]  [12], we develop a framework that learns sentence embeddings by maximizing the agreement between the representations of a cluster of instances belonging to the same class using a novel fully-supervised contrastive loss that guides a neural network to better separate the classes. We address the problem of text-multi-class classification applications.
We consider many positives per anchor, unlike previous works on self-supervised contrastive learning which use only a single positive example and many negatives. In other words, instead of using augmented views of the same anchor as done in self-supervised contrastive learning (which is not obvious in textual data), we leverage label information to consider many positives and many negatives for each anchor. In figure  1 , we show how we select positive/negative examples for each class. The use of many positives and many negatives in our framework allows the encoder function to better maximize the intra-class compactness and the inter-class separability (learns useful and generalizable features) than the standard framework which relies on the cross-entropy loss. Figure  2  shows tSNE plots of the learned representations of the model trained on SST-2 dataset with our loss against those learnt by cross entropy.
The experimental results show that our Framework consistently outperforms the commonly-used supervised learning framework based on cross-entropy loss on many publicly available benchmark datasets. For instance, we achieve an accuracy of 61. 94% and 95. 45% using our framework on Amazon-F and Yelp-P respectively, while the scores obtained by the cross-entropy are 58. 40% and 92. 12% using the same neural architecture. The experimental results show that our proposed method benefits from large positive examples. However, we find that beyond a certain threshold, increasing the number of positive examples does not improve the quality of the representations. In contrast, the method produces high-quality representations when the number of negative instances is large. In addition, simulations show that the sampling strategy is crucial for learning more generalizeable sentence representations for downstream tasks on several benchmark datasets.
II. RELATED WORK A. CROSS ENTROPY LOSS Cross Entropy (CE) is the de facto choice for the loss function in classification tasks. This prominence is due to many reasons. First, CE has good theoretical grounding in information theory, which makes it useful for theoretical analysis of systems  [15]. Second, CE loss has been proven to rival many loss functions in large datasets  [45]. However, it suffers from major robustness issues. Indeed, CE suffers from adversarial robustness, as was shown in  [16]  which demonstrated empirically that training with a CE loss can cause the representations to spread sparsely over the representation space during training. Additionally, introducing noisy data seems to degrade performance substantially  [19]  which is due to the fact that the cross entropy loss supposes that all the training labels are true, and neglects the fuzziness of noisy labels  [17] ,  [19].
C. SUPERVISED CONTRASTIVE REPRESENTATION LEARNING Recently, many works have extended the self-supervised contrastive learning approach to the fully-supervised setting by leveraging label information for learning representations. In the computer vision field,  [29]  propose SupCon, an objective function for the task of image classification that bridges the gap between self-supervised learning and fully supervised learning. SupCon was shown to outperform SimCLR  [11] , Max-Margin  [33]  and cross-entropy on several benchmark datasets such as ImageNet. The SupCon objective function is defined by: L sup out = i∈I - 1 |P(i)| p∈P(i) log exp(v i. v p /τ ) n∈N (i) exp(v i. v n /τ ) (2) where I ≡ {1, • • • , N } with N being the batch size, P(i) is the set of indices of all data elements in the batch that belong to the same class as data element v i , N (i) ≡ I\{i},and |.
| denotes the cardinality operator. In  [55] , the authors proposed a novel objective function for fine-tuning transformer-based language models which consists of a weighted sum of the cross-entropy loss and the above-mentioned supervised contrastive loss: (1 -λ)L CE + λL sup out. This approach was shown to outperform a strong RoBERTa-large baseline on the GLUE benchmark dataset in few-shot learning settings. Many works have extended SupCon and proposed new variants. In  [56] , for instance, the authors evaluated three variants of pixel-wise label-based contrastive loss to pre-train a semantic segmentation model.
2) Inter-class and Intra-class Distances We first define notations and describe the proposed framework for classification tasks that will be essential for the analysis. Let D = {(x i , y i )} i be the available dataset, where x i represents the i th sentence and y i is its label. Let S k = {(x i , y i )|y i = k} i denote the subset of all sentences of the dataset belonging to class k. Let B k ∼ S k be a mini-batch of randomly sampled examples from S k. Let f w (. ) denotes the encoder function where the sub-index w refers to the weights of the encoders to be learnt. Let H k = f w (B k ) ∈ R N k ×d be the highest level representation of the encoder where N k is the batch size and d is the dimension of the embedding vector. The j th row of H k is the transpose of the embedding vector associated with the j th sentence of B k , which we denote as h j , i.
A. DATASET AND TRAINING DETAILS We evaluated the effectiveness of the proposed framework on sentence classification tasks by measuring accuracy on 6 benchmark datasets namely, SST-2, Yelp-P, yelp-F, Amazon-P, Amazon-F, and IMDb. Furthermore, the framework is also tested for representation learning on low-resource language setting datasets namely, MSAC, ASAC. We summarize each dataset based on their main task, domain, number of training examples, and number of classes in Table  1.
B. TRAINING DETAILS Our framework allows various choices of the network architecture without any constraints. Here, we opt for simplicity and adopt the BiLSTM neural network architecture to compare different objective functions. For ASAC and MSAC datsets, we used the following settings: we train our framework for 15 epochs using Adam  [32]  optimized with a learning rate of 0. 003. However, for these datasets, we use an encoder with 1 hidden layer only   due to the limited number of examples that we have in these datasets. We use a hidden units of 128 neurons, and a batch size of 200. We apply dropout with probability 0. 2. Similarly, the CE is trained for a batch size of up to 400, but the best results are obtained using a batch size of 64. For the remaining datsets, SuperLoss is optimized for 60 epochs using Adam with a learning rate of 0.