Reviewerly/exHarmony huggingface dataset:
BLISS: A Billion scale Index using Iterative Re-partitioning

Representation learning has transformed the problem of information retrieval into one of finding the approximate set of nearest neighbors in a high dimensional vector space. With limited hardware resources and time-critical queries, the retrieval engines face an inherent tension between latency, accuracy, scalability, compactness, and the ability to load balance in distributed settings. To improve the trade-off, we propose a new algorithm, called BaLanced Index for Scalable Search (BLISS), a highly tunable indexing algorithm with enviably small index sizes, making it easy to scale to billions of vectors. It iteratively refines partitions of items by learning the relevant buckets directly from the query-item relevance data. To ensure that the buckets are balanced, BLISS uses the power-of-K choices strategy. We show that BLISS provides superior load balancing with high probability (and under very benign assumptions). Due to its design, BLISS can be employed for both near-neighbor retrieval (ANN problem) and extreme classification (XML problem). For the case of ANN, we train and index 4 datasets with billion vectors each. We compare the recall, inference time, indexing time, and index size for BLISS with the two most popular and well-optimized libraries- Hierarchical Navigable Small World (HNSW) graph and Facebook's FAISS. BLISS requires 100x lesser RAM than HNSW, making it fit in memory on commodity machines while taking a similar inference time as HNSW for the same recall. Against FAISS-IVF, BLISS achieves similar performance with 3-4x less memory requirement. BLISS is both data and model parallel, making it ideal for distributed implementation for training and inference. For the case of XML, BLISS surpasses the best baselines' precision while being 5x faster for inference on popular multi-label datasets with half a million classes.
Personalized Chit-Chat Generation for Recommendation Using External Chat Corpora

Chit-chat has been shown effective in engaging users in human-computer interaction. We find with a user study that generating appropriate chit-chat for news articles can help expand user interest and increase the probability that a user reads a recommended news article. Based on this observation, we propose a method to generate personalized chit-chat for news recommendation. Different from existing methods for personalized text generation, our method only requires an external chat corpus obtained from an online forum, which can be disconnected from the recommendation dataset from both the user and item (news) perspectives. This is achieved by designing a weak supervision method for estimating users' personalized interest in a chit-chat post by transferring knowledge learned by a news recommendation model. Based on the method for estimating user interest, a reinforcement learning framework is proposed to generate personalized chit-chat. Extensive experiments, including the automatic offline evaluation and user studies, demonstrate the effectiveness of our method.
Robust and Informative Text Augmentation (RITA) via Constrained Worst-Case Transformations for Low-Resource Named Entity Recognition

Recent advances in deep learning have brought remarkable performance improvements in named entity recognition (NER), specifically in token-level classification problems. However, deep learning models often require a large amount of annotated data to achieve satisfactory performance, and NER annotation is significantly time-consuming and labor-intensive due to the fine-grained labels. To address this issue, we propose a textual data augmentation method that can automatically generate informative synthetic samples, which contribute to the development of a robust classifier. The proposed method generates additional training data by estimating the optimal level of worst-case transformation of training data while preserving the original annotation, and includes them into training to construct a robust decision boundary. Extensive experiments conducted on two benchmark datasets in a low-resource environment reveal that the proposed method outperforms two baseline augmentation methods including human annotation, which is typically considered to provide a decent amount of performance boost. To elucidate the processes, we also present in-depth analyses of the generated samples and estimated model parameters.
End-to-End Semi-Supervised Ordinal Regression AUC Maximization with Convolutional Kernel Networks

Convolutional kernel networks (CKN) have been proposed to solve image classification tasks, and have shown competitive performance over classical neural networks while being easy to train and robust to overfitting. In real-world ordinal regression problems, we usually have plenty of unlabeled data but a limited number of labeled ordered data. Although recent research works have shown that directly optimizing AUC can impose a better ranking on the data than optimizing traditional error rate, it is still an open question to design an efficient semi-supervised ordinal regression AUC maximization algorithm based on CKN with convergence guarantee. To address this question, in this paper, we propose a new semi-supervised ordinal regression CKN algorithm (S^2 CKNOR) with end-to-end AUC maximization. Specifically, we decompose the ordinal regression into a series of binary classification subproblems and propose an unbiased non-convex objective function to optimize AUC, such that both labeled and unlabeled data can be used to enhance the model performance. Further, we propose a nested alternating minimization algorithm to solve the non-convex objective, where each (convex) subproblem is solved by a quadruply stochastic gradient algorithm, and the non-convex one is solved by the stochastic projected gradient method. Importantly, we prove that our S^2 CKNOR algorithm can finally converge to a critical point of the non-convex objective. Extensive experimental results demonstrate that our S^2 CKNOR achieves the best AUC results on various real-world datasets.
Causal Discovery on Non-Euclidean Data

Researchers recently started developing deep learning models capable of handling non-Euclidean data. However, because of existing framework limitations on model representations and learning algorithms, few have explored causal discovery on non-Euclidean data. This paper is the first attempt to do so. We start by proposing the Non-Euclidean Causal Model (NECM) which describes the causal generative relationship of non-Euclidean data and creates a new tensor data type along with a mapping process for the non-Euclidean causal mechanism. Second, within the NECM, we propose the non-Euclidean Hybrid Learning (NEHL) method, a causal discovery algorithm relying on the concept of the ball covariance recently introduced in the statistics field. Third, we generate two types of non-Euclidean datasets: Functional Data and Symmetric Positive Definite manifold data in conformity with the NECM. Finally, experimental results on the generated data and real-world data demonstrate the effectiveness of the proposed NEHL method.
Towards a Native Quantum Paradigm for Graph Representation Learning

Graph representation learning has been extensively studied, and recent models can well incorporate both node features and graph structures. Despite these progress, the inherent scalability challenge for classical computers of processing graph data and solving the downstream tasks (many are NP-hard) is still a bottleneck for existing classical graph learning models. On the other hand, quantum computing is known a promising direction for its theoretically verified scalability as well as the increasing evidence for the access to physical quantum machine in near-term. Different from many existing classical-quantum hybrid machine learning models on graphs, in this paper we take a more aggressive initiative for developing a native quantum paradigm for (attributed) graph representation learning, which to our best knowledge, has not been fulfilled in literature yet. Specifically, our model adopts the well-established theory and technique in quantum computing e.g. quantum random walk, and adapt it to the attributed graph. Then the node attribute quantum state sequence is fed into a quantum recurrent network to obtain the final node embedding. Experimental results on three public datasets show the effectiveness of our quantum model which also outperforms a classical learning approach GraphRNA notably in terms of efficiency even on a classical computer. Though it is still restricted to the classical loss-based learning paradigm with gradient descent for model parameter training, while our computing scheme is compatible with quantum computing without involving classical computers. This is in fact largely in contrast to many hybrid quantum graph learning models which often involve many steps and modules having to be performed on classical computers.
Flexible Modeling and Multitask Learning using Differentiable Tree Ensembles

Decision tree ensembles are widely used and competitive learning models. Despite their success, popular toolkits for learning tree ensembles have limited modeling capabilities. For instance, these toolkits support a limited number of loss functions and are restricted to single task learning. We propose a flexible framework for learning tree ensembles, which goes beyond existing toolkits to support arbitrary loss functions, missing responses, and multi-task learning. Our framework builds on differentiable (a.k.a. soft) tree ensembles, which can be trained using first-order methods. However, unlike classical trees, differentiable trees are difficult to scale. We therefore propose a novel tensor-based formulation of differentiable trees that allows for efficient vectorization on GPUs. We introduce FASTEL: a new toolkit (based on Tensorflow 2) for learning differentiable tree ensembles. We perform experiments on a collection of 28 real open-source and proprietary datasets, which demonstrate that our framework can lead to 100x more compact and 23% more expressive tree ensembles than those obtained by popular toolkits.
Reward Optimizing Recommendation using Deep Learning and Fast Maximum Inner Product Search

How can we build and optimize a recommender system that must rapidly fill slates (i.e. banners) of personalized recommendations? The combination of deep learning stacks with fast maximum inner product search (MIPS) algorithms have shown it is possible to deploy flexible models in production that can rapidly deliver personalized recommendations to users. Albeit promising, this methodology is unfortunately not sufficient to build a recommender system which maximizes the reward, e.g. the probability of click. Usually instead a proxy loss is optimized and A/B testing is used to test if the system actually improved performance. This tutorial takes participants through the necessary steps to model the reward and directly optimize the reward of recommendation engines built upon fast search algorithms to produce high-performance reward-optimizing recommender systems.
SOS

Score-based generative models (SGMs) are a recent breakthrough in generating fake images. SGMs are known to surpass other generative models, e.g., generative adversarial networks (GANs) and variational autoencoders (VAEs). Being inspired by their big success, in this work, we fully customize them for generating fake tabular data. In particular, we are interested in oversampling minor classes since imbalanced classes frequently lead to sub-optimal training outcomes. To our knowledge, we are the first presenting a score-based tabular data oversampling method. Firstly, we re-design our own score network since we have to process tabular data. Secondly, we propose two options for our generation method: the former is equivalent to a style transfer for tabular data and the latter uses the standard generative policy of SGMs. Lastly, we define a fine-tuning method, which further enhances the oversampling quality. In our experiments with 6 datasets and 10 baselines, our method outperforms other oversampling methods in all cases.
Beyond Point Prediction: Capturing Zero-Inflated &amp; Heavy-Tailed Spatiotemporal Data with Deep Extreme Mixture Models

Zero-inflated, heavy-tailed spatiotemporal data is common across science and engineering, from climate science to meteorology and seismology. A central modeling objective in such settings is to forecast the intensity, frequency, and timing of extreme and non-extreme events; yet in the context of deep learning, this objective presents several key challenges. First, a deep learning framework applied to such data must unify a mixture of distributions characterizing the zero events, moderate events, and extreme events. Second, the framework must be capable of enforcing parameter constraints across each component of the mixture distribution. Finally, the framework must be flexible enough to accommodate for any changes in the threshold used to define an extreme event after training. To address these challenges, we propose Deep Extreme Mixture Model (DEMM), fusing a deep learning-based hurdle model with extreme value theory to enable point and distribution prediction of zero-inflated, heavy-tailed spatiotemporal variables. The framework enables users to dynamically set a threshold for defining extreme events at inference-time without the need for retraining. We present an extensive experimental analysis applying DEMM to precipitation forecasting, and observe significant improvements in point and distribution prediction. All code is available at https://github.com/andrewmcdonald27/DeepExtremeMixtureModel.
Model Monitoring in Practice

Artificial Intelligence (AI) is increasingly playing an integral role in determining our day-to-day experiences. Increasingly, the applications of AI are no longer limited to search and recommendation systems, such as web search and movie and product recommendations, but AI is also being used in decisions and processes that are critical for individuals, businesses, and society. With AI based solutions in high-stakes domains such as hiring, lending, criminal justice, healthcare, and education, the resulting personal and professional implications of AI are far-reaching. Consequently, it becomes critical to ensure that these models are making accurate predictions, are robust to shifts in the data, are not relying on spurious features, and are not unduly discriminating against minority groups. To this end, several approaches spanning various areas such as explainability, fairness, and robustness have been proposed in recent literature, and many papers and tutorials on these topics have been presented in recent computer science conferences. However, there is relatively less attention on the need for monitoring machine learning (ML) models once they are deployed and the associated research challenges.
A Practical Introduction to Federated Learning

As Internet users attach importance to their own privacy, and a number of laws and regulations go into effect in most countries, Internet products need to provide users with privacy protection. As one of the feasible solutions to provide such privacy protection, federated learning has rapidly gained popularity in both academia and industry in recent years. In this tutorial, we will start off with some real-world tasks to illustrate the topic of federated learning, and cover some basic concepts and important scenarios including cross-device and cross-silo settings. Along with it, we will give several demonstrations with popular federated learning frameworks. We will also show how to do the automatic hyperparameter tuning with federated learning to significantly save their efforts in practice. Then we dive into three parallel hot topics, Personalized Federated Learning, Federated Graph Learning, and Attack in Federated Learning. For each of them, we will motivate it with real-world applications, illustrate the state-of-the-art methods, and discuss their pros and cons using concrete examples. As the last part, we will point out some future research directions.
Releasing Private Data for Numerical Queries

Prior work on private data release has only studied counting queries or linear queries, where each tuple in the dataset contributes a value in [0,1] and a query returns the sum of the values. However, many data analytical tasks involve numerical values that are arbitrary real numbers. In this paper, we present a new mechanism to privatize a dataset D for a given set Q of numerical queries, achieving an error of Õ (√n • Δw(D)) for each query w ∈ Q, where Δw(D) is the maximum contribution of any tuple in D queried by w. This instance- and query-specific error bound not only is theoretically appealing, but also leads to excellent practical performance.
Hyperbolic Neural Networks: Theory, Architectures and Applications

Recent studies have revealed important properties that are unique to graph datasets such as hierarchies and global structures. This has driven research into hyperbolic space due to their ability to effectively encode the inherent hierarchy present in graph datasets. However, a major bottleneck here is the obscurity of hyperbolic geometry and a better comprehension of its gyrovector operations. In this tutorial, we aim to introduce researchers and practitioners in the data mining community to the hyperbolic equivariants of the Euclidean operations that are necessary to tackle their application to neural networks. We describe the popular hyperbolic variants of GNN architectures and explain their implementation, in contrast to the Euclidean counterparts. Also, we motivate our tutorial through critical analysis of existing applications in the areas of graph mining, knowledge graph reasoning, search, NLP, and computer vision.
Deep Search Relevance Ranking in Practice

Machine learning techniques for developing industry-scale search engines have long been a prominent part of most domains and their online products. Search relevance algorithms are key components of products across different fields, including e-commerce, streaming services, and social networks. In this tutorial, we give an introduction to such large-scale search ranking systems, specifically focusing on deep learning techniques in this area. The topics we cover are the following: (1) Overview of search ranking systems in practice, including classical and machine learning techniques; (2) Introduction to sequential and language models in the context of search ranking; and (3) Knowledge distillation approaches for this area. For each of the aforementioned sessions, we first give an introductory talk and then go over an hands-on tutorial to really hone in on the concepts. We cover fundamental concepts using demos, case studies, and hands-on examples, including the latest Deep Learning methods that have achieved state-of-the-art results in generating the most relevant search results. Moreover, we show example implementations of these methods in python, leveraging a variety of open-source machine-learning/deep-learning libraries as well as real industrial data or open-source data.
concept2code: Deep Reinforcement Learning for Conversational AI

Deep Reinforcement Learning uses best of both Reinforcement Learning and Deep Learning for solving problems which cannot be addressed by them individually. Deep Reinforcement Learning has been used widely for games, robotics etc. Limited work has been done for applying Deep Reinforcement Learning for Conversational AI. Hence, in this tutorial cover application of Deep Reinforcement Learning for Conversational AI.
Institutional Repositories in Russia and Abroad: Review of Publications

In contemporary information space, repository formation is one of the urgent tasks of libraries of scientific and educational institutions. Institutional repositories (IRs) provide unhindered access to scholarly works posted by scientists and specialists for general user, increasing the demand for the library's information resources. The article represents the review of foreign and domestic publications on the formation and development of institutional repositories. The authors consider the advantages of forming Open Access (OA) Repositories both among prospective authors and potential users: archiving and keeping materials in the IR up-to-date, ensuring visibility and possibility of using scientific research through global access, rapid dissemination of research results, increasing the citation of submitted materials, growth of prestige of the organization, the development of scientific communication, etc. The authors found out that placing publications in international archives (arXiv, BioXiv, etc.) or corporate projects (BASE, HathiTrust) has a greater impact on the visibility and potential citation of publications. The article identifies the problems of formation and use of repositories: labour intensity of content placement process, the complexity of data exchange, copyright management, the lack of incentives in providing information about their scientific work, the need for additional time and labour resources when maintaining the IR, the financial costs of maintaining the repository, and the difficulties of search in the IR. The authors note that the IR functionality depends on the software capabilities. The paper gives the examples of new generation repositories and presents their features: self-filling, automatic classification based on text mining and machine learning, data management processes between different systems, easy-to-use interfaces.
Organizational Culture as a Development Resource in the Conditions of Changing Institutional Functions of Libraries

In the context of technological and socio-cultural transformations, libraries are in search for landmarks and new formats of activity. The vector of transformation of library functions is from the preservation and transmission of information to agentive: structuring communities, managing information, communicative functions. Libraries are becoming conductors of social innovation. The article considers organizational culture as a catalyst for the development of libraries in the future. The authors pose the problem of inconsistency of organizational culture with new demands and needs, as well as justify the need for the formation of an innovative type of culture. The paper presents the results of quantitative and qualitative research in the Municipal Information Library System (MILS) of Tomsk in 2021. Questionnaires and interviews were conducted with employees of three libraries of Tomsk MILS, one of which took part in the Federal project “Cultural Environment” of the National project “Culture” on the creation of model municipal libraries. The authors study organizational culture at the formal and informal levels. The analysis showed that each of the libraries demonstrated insufficient level of compliance with the context of social transformations. Barriers of organizational culture in public libraries are conservative mission, management attitudes to support performing competencies, features of clan culture (orientation to isolation, lack of openness for cooperation on the part of staff both within the library network and with external communities, underestimation of the role of partnership), technological focus of activities, etc. Reserves for the development of organizational culture are the willingness of staff to take initiatives, focus on support and building positive relationships in the team. As an opportunity to manage organizational culture, the authors note that transformations should begin with a change of emphasis in the libraryʼs mission based on a new spatial approach.