OpenRaiser/Intern-Atlas huggingface dataset:
Scenarios and branch points to future machine intelligence
Venue: arXiv cs.MA 2023
Year: 2023

We discuss scenarios and branch points to four major possible consequences regarding future machine intelligence; 1) the singleton scenario where the first and only super-intelligence acquires a decisive strategic advantage, 2) the multipolar scenario where the singleton scenario is not technically denied but political or other factors in human society or multi-agent interactions between the intelligent agents prevent a single agent from gaining a decisive strategic advantage, 3) the ecosystem scenario where the singleton scenario is denied and many autonomous intelligent agents operate in such a way that they are interdependent and virtually unstoppable, and 4) the upper-bound scenario where cognitive capabilities that can be achieved by human-designed intelligent agents or their descendants are inherently limited to the sub-human level. We identify six major constraints that can form branch points to these scenarios; (1) constraints on autonomy, (2) constraints on the ability to improve self-structure, (3) constraints related to thermodynamic efficiency, (4) constraints on updating physical infrastructure, (5) constraints on relative advantage, and (6) constraints on locality.
Enhancing Classification with Hierarchical Scalable Query on Fusion Transformer
Venue: arXiv cs.CV 2023
Year: 2023

Real-world vision based applications require fine-grained classification for various area of interest like e-commerce, mobile applications, warehouse management, etc. where reducing the severity of mistakes and improving the classification accuracy is of utmost importance. This paper proposes a method to boost fine-grained classification through a hierarchical approach via learnable independent query embeddings. This is achieved through a classification network that uses coarse class predictions to improve the fine class accuracy in a stage-wise sequential manner. We exploit the idea of hierarchy to learn query embeddings that are scalable across all levels, thus making this a relevant approach even for extreme classification where we have a large number of classes. The query is initialized with a weighted Eigen image calculated from training samples to best represent and capture the variance of the object. We introduce transformer blocks to fuse intermediate layers at which query attention happens to enhance the spatial representation of feature maps at different scales. This multi-scale fusion helps improve the accuracy of small-size objects. We propose a two-fold approach for the unique representation of learnable queries. First, at each hierarchical level, we leverage cluster based loss that ensures maximum separation between inter-class query embeddings and helps learn a better (query) representation in higher dimensional spaces. Second, we fuse coarse level queries with finer level queries weighted by a learned scale factor. We additionally introduce a novel block called Cross Attention on Multi-level queries with Prior (CAMP) Block that helps reduce error propagation from coarse level to finer level, which is a common problem in all hierarchical classifiers. Our method is able to outperform the existing methods with an improvement of ~11% at the fine-grained classification.
Text classification dataset and analysis for Uzbek language
Venue: arXiv cs.CL 2023
Year: 2023

Text classification is an important task in Natural Language Processing (NLP), where the goal is to categorize text data into predefined classes. In this study, we analyse the dataset creation steps and evaluation techniques of multi-label news categorisation task as part of text classification. We first present a newly obtained dataset for Uzbek text classification, which was collected from 10 different news and press websites and covers 15 categories of news, press and law texts. We also present a comprehensive evaluation of different models, ranging from traditional bag-of-words models to deep learning architectures, on this newly created dataset. Our experiments show that the Recurrent Neural Network (RNN) and Convolutional Neural Network (CNN) based models outperform the rule-based models. The best performance is achieved by the BERTbek model, which is a transformer-based BERT model trained on the Uzbek corpus. Our findings provide a good baseline for further research in Uzbek text classification.
Large Language Models Are State-of-the-Art Evaluators of Translation Quality
Venue: arXiv cs.CL 2023
Year: 2023

We describe GEMBA, a GPT-based metric for assessment of translation quality, which works both with a reference translation and without. In our evaluation, we focus on zero-shot prompting, comparing four prompt variants in two modes, based on the availability of the reference. We investigate nine versions of GPT models, including ChatGPT and GPT-4. We show that our method for translation quality assessment only works with GPT~3.5 and larger models. Comparing to results from WMT22's Metrics shared task, our method achieves state-of-the-art accuracy in both modes when compared to MQM-based human labels. Our results are valid on the system level for all three WMT22 Metrics shared task language pairs, namely English into German, English into Russian, and Chinese into English. This provides a first glimpse into the usefulness of pre-trained, generative large language models for quality assessment of translations. We publicly release all our code and prompt templates used for the experiments described in this work, as well as all corresponding scoring results, to allow for external validation and reproducibility.
Rethinking Multi-Interest Learning for Candidate Matching in Recommender Systems
Venue: arXiv cs.IR 2023
Year: 2023

Existing research efforts for multi-interest candidate matching in recommender systems mainly focus on improving model architecture or incorporating additional information, neglecting the importance of training schemes. This work revisits the training framework and uncovers two major problems hindering the expressiveness of learned multi-interest representations. First, the current training objective (i.e., uniformly sampled softmax) fails to effectively train discriminative representations in a multi-interest learning scenario due to the severe increase in easy negative samples. Second, a routing collapse problem is observed where each learned interest may collapse to express information only from a single item, resulting in information loss. To address these issues, we propose the REMI framework, consisting of an Interest-aware Hard Negative mining strategy (IHN) and a Routing Regularization (RR) method. IHN emphasizes interest-aware hard negatives by proposing an ideal sampling distribution and developing a Monte-Carlo strategy for efficient approximation. RR prevents routing collapse by introducing a novel regularization term on the item-to-interest routing matrices. These two components enhance the learned multi-interest representations from both the optimization objective and the composition information. REMI is a general framework that can be readily applied to various existing multi-interest candidate matching methods. Experiments on three real-world datasets show our method can significantly improve state-of-the-art methods with easy implementation and negligible computational overhead. The source code will be released.
DEff-GAN: Diverse Attribute Transfer for Few-Shot Image Synthesis
Venue: arXiv cs.CV 2023
Year: 2023

Requirements of large amounts of data is a difficulty in training many GANs. Data efficient GANs involve fitting a generators continuous target distribution with a limited discrete set of data samples, which is a difficult task. Single image methods have focused on modeling the internal distribution of a single image and generating its samples. While single image methods can synthesize image samples with diversity, they do not model multiple images or capture the inherent relationship possible between two images. Given only a handful of images, we are interested in generating samples and exploiting the commonalities in the input images. In this work, we extend the single-image GAN method to model multiple images for sample synthesis. We modify the discriminator with an auxiliary classifier branch, which helps to generate a wide variety of samples and to classify the input labels. Our Data-Efficient GAN (DEff-GAN) generates excellent results when similarities and correspondences can be drawn between the input images or classes.
Decentralised construction of a global coordinate system in a large swarm of minimalistic robots
Venue: arXiv cs.RO 2023
Year: 2023

Collective intelligence and autonomy of robot swarms can be improved by enabling the individual robots to become aware they are the constituent units of a larger whole and what is their role. In this study, we present an algorithm to enable positional self-awareness in a swarm of minimalistic error-prone robots which can only locally broadcast messages and estimate the distance from their neighbours. Despite being unable to measure the bearing of incoming messages, the robots running our algorithm can calculate their position within a swarm deployed in a regular formation. We show through experiments with up to 200 Kilobot robots that such positional self-awareness can be employed by the robots to create a shared coordinate system and dynamically self-assign location-dependent tasks. Our solution has fewer requirements than state-of-the-art algorithms and contains collective noise-filtering mechanisms. Therefore, it has an extended range of robotic platforms on which it can run. All robots are interchangeable, run the same code, and do not need any prior knowledge. Through our algorithm, robots reach collective synchronisation, and can autonomously become self-aware of the swarm's spatial configuration and their position within it.
deHuBERT: Disentangling Noise in a Self-supervised Model for Robust Speech Recognition
Venue: arXiv cs.SD 2023
Year: 2023

Existing self-supervised pre-trained speech models have offered an effective way to leverage massive unannotated corpora to build good automatic speech recognition (ASR). However, many current models are trained on a clean corpus from a single source, which tends to do poorly when noise is present during testing. Nonetheless, it is crucial to overcome the adverse influence of noise for real-world applications. In this work, we propose a novel training framework, called deHuBERT, for noise reduction encoding inspired by H. Barlow's redundancy-reduction principle. The new framework improves the HuBERT training algorithm by introducing auxiliary losses that drive the self- and cross-correlation matrix between pairwise noise-distorted embeddings towards identity matrix. This encourages the model to produce noise-agnostic speech representations. With this method, we report improved robustness in noisy environments, including unseen noises, without impairing the performance on the clean set.
Detecting and Optimising Team Interactions in Software Development
Venue: arXiv cs.SE 2023
Year: 2023

The functional interaction structure of a team captures the preferences with which members of different roles interact. This paper presents a data-driven approach to detect the functional interaction structure for software development teams from traces team members leave on development platforms during their daily work. Our approach considers differences in the activity levels of team members and uses a block-constrained configuration model to compute interaction preferences between members of different roles. We apply our approach in a case study to extract the functional interaction structure of a product team at the German IT security company genua GmbH. We subsequently validate the accuracy of the detected interaction structure in interviews with five team members. Finally, we show how our approach enables teams to compare their functional interaction structure against synthetically created benchmark scenarios. Specifically, we evaluate the level of knowledge diffusion in the team and identify areas where the team can further improve. Our approach is computationally efficient and can be applied in real time to manage a team's interaction structure.
Experience in Engineering Complex Systems: Active Preference Learning with Multiple Outcomes and Certainty Levels
Venue: arXiv cs.LG 2023
Year: 2023

Black-box optimization refers to the optimization problem whose objective function and/or constraint sets are either unknown, inaccessible, or non-existent. In many applications, especially with the involvement of humans, the only way to access the optimization problem is through performing physical experiments with the available outcomes being the preference of one candidate with respect to one or many others. Accordingly, the algorithm so-called Active Preference Learning has been developed to exploit this specific information in constructing a surrogate function based on standard radial basis functions, and then forming an easy-to-solve acquisition function which repetitively suggests new decision vectors to search for the optimal solution. Based on this idea, our approach aims to extend the algorithm in such a way that can exploit further information effectively, which can be obtained in reality such as: 5-point Likert type scale for the outcomes of the preference query (i.e., the preference can be described in not only "this is better than that" but also "this is much better than that" level), or multiple outcomes for a single preference query with possible additive information on how certain the outcomes are. The validation of the proposed algorithm is done through some standard benchmark functions, showing a promising improvement with respect to the state-of-the-art algorithm.
Graph-based Knowledge Distillation: A survey and experimental evaluation
Venue: arXiv cs.LG 2023
Year: 2023

Graph, such as citation networks, social networks, and transportation networks, are prevalent in the real world. Graph Neural Networks (GNNs) have gained widespread attention for their robust expressiveness and exceptional performance in various graph applications. However, the efficacy of GNNs is heavily reliant on sufficient data labels and complex network models, with the former obtaining hardly and the latter computing costly. To address the labeled data scarcity and high complexity of GNNs, Knowledge Distillation (KD) has been introduced to enhance existing GNNs. This technique involves transferring the soft-label supervision of the large teacher model to the small student model while maintaining prediction performance. This survey offers a comprehensive overview of Graph-based Knowledge Distillation methods, systematically categorizing and summarizing them while discussing their limitations and future directions. This paper first introduces the background of graph and KD. It then provides a comprehensive summary of three types of Graph-based Knowledge Distillation methods, namely Graph-based Knowledge Distillation for deep neural networks (DKD), Graph-based Knowledge Distillation for GNNs (GKD), and Self-Knowledge Distillation based Graph-based Knowledge Distillation (SKD). Each type is further divided into knowledge distillation methods based on the output layer, middle layer, and constructed graph. Subsequently, various algorithms' ideas are analyzed and compared, concluding with the advantages and disadvantages of each algorithm supported by experimental results. In addition, the applications of graph-based knowledge distillation in CV, NLP, RS, and other fields are listed. Finally, the graph-based knowledge distillation is summarized and prospectively discussed. We have also released related resources at https://github.com/liujing1023/Graph-based-Knowledge-Distillation.
Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following
Venue: arXiv cs.CL 2023
Year: 2023

In this paper, we present our finding that prepending a Task-Agnostic Prefix Prompt (TAPP) to the input improves the instruction-following ability of various Large Language Models (LLMs) during inference. TAPP is different from canonical prompts for LLMs in that it is a fixed prompt prepended to the beginning of every input regardless of the target task for zero-shot generalization. We observe that both base LLMs (i.e. not fine-tuned to follow instructions) and instruction-tuned models benefit from TAPP, resulting in 34.58% and 12.26% improvement on average, respectively. This implies that the instruction-following ability of LLMs can be improved during inference time with a fixed prompt constructed with simple heuristics. We hypothesize that TAPP assists language models to better estimate the output distribution by focusing more on the instruction of the target task during inference. In other words, such ability does not seem to be sufficiently activated in not only base LLMs but also many instruction-fine-tuned LLMs. All experiments are reproducible from https://github.com/seonghyeonye/TAPP.
Massively Parallel Computation in a Heterogeneous Regime
Venue: arXiv cs.DS 2023
Year: 2023

Massively-parallel graph algorithms have received extensive attention over the past decade, with research focusing on three memory regimes: the superlinear regime, the near-linear regime, and the sublinear regime. The sublinear regime is the most desirable in practice, but conditional hardness results point towards its limitations. In this work we study a \emph{heterogeneous} model, where the memory of the machines varies in size. We focus mostly on the heterogeneous setting created by adding a single near-linear machine to the sublinear MPC regime, and show that even a single large machine suffices to circumvent most of the conditional hardness results for the sublinear regime: for graphs with $n$ vertices and $m$ edges, we give (a) an MST algorithm that runs in $O(\log\log(m/n))$ rounds; (b) an algorithm that constructs an $O(k)$-spanner of size $O(n^{1+1/k})$ in $O(1)$ rounds; and (c) a maximal-matching algorithm that runs in $O(\sqrt{\log(m/n)}\log\log(m/n))$ rounds. We also observe that the best known near-linear MPC algorithms for several other graph problems which are conjectured to be hard in the sublinear regime (minimum cut, maximal independent set, and vertex coloring) can easily be transformed to work in the heterogeneous MPC model with a single near-linear machine, while retaining their original round complexity in the near-linear regime. If the large machine is allowed to have \emph{superlinear} memory, all of the problems above can be solved in $O(1)$ rounds.
Is Japanese CCGBank empirically correct? A case study of passive and causative constructions
Venue: arXiv cs.CL 2023
Year: 2023

The Japanese CCGBank serves as training and evaluation data for developing Japanese CCG parsers. However, since it is automatically generated from the Kyoto Corpus, a dependency treebank, its linguistic validity still needs to be sufficiently verified. In this paper, we focus on the analysis of passive/causative constructions in the Japanese CCGBank and show that, together with the compositional semantics of ccg2lambda, a semantic parsing system, it yields empirically wrong predictions for the nested construction of passives and causatives.
Security in Distributed Systems by Verifiable Location-Based Identities
Venue: arXiv cs.CR 2023
Year: 2023

Proof-of-Location (PoL) is a lightweight security concept for Internet-of-Things (IoT) networks, focusing on the sensor nodes as the least performant and most vulnerable parts of IoT networks. PoL builds on the identification of network participants based on their physical location. It introduces a secondary message type to exchange location information. Via these messages, the nodes can verify the integrity of other network participants and reach a consensus to identify potential attackers and prevent malicious information from spreading. The paper presents the concretization of the concept to allow implementation on real hardware. The evaluation based on this implementation demonstrates the feasibility of PoL and enables identifying further steps to develop a deployable protocol.
Fusion of ML with numerical simulation for optimized propeller design
Venue: arXiv cs.LG 2023
Year: 2023

In computer-aided engineering design, the goal of a designer is to find an optimal design on a given requirement using the numerical simulator in loop with an optimization method. In this design optimization process, a good design optimization process is one that can reduce the time from inception to design. In this work, we take a class of design problem, that is computationally cheap to evaluate but has high dimensional design space. In such cases, traditional surrogate-based optimization does not offer any benefits. In this work, we propose an alternative way to use ML model to surrogate the design process that formulates the search problem as an inverse problem and can save time by finding the optimal design or at least a good initial seed design for optimization. By using this trained surrogate model with the traditional optimization method, we can get the best of both worlds. We call this as Surrogate Assisted Optimization (SAO)- a hybrid approach by mixing ML surrogate with the traditional optimization method. Empirical evaluations of propeller design problems show that a better efficient design can be found in fewer evaluations using SAO.
SMPT: A Testbed for Reachability Methods in Generalized Petri Nets
Venue: arXiv cs.LO 2023
Year: 2023

SMPT (for Satisfiability Modulo Petri Net) is a model checker for reachability problems in Petri nets. It started as a portfolio of methods to experiment with symbolic model checking, and was designed to be easily extended. Some distinctive features are its ability to benefit from structural reductions and to generate verdict certificates. Our tool is quite mature and performed well compared to other state-of-the-art tools in the Model Checking Contest.
Time Series Anomaly Detection in Smart Homes: A Deep Learning Approach
Venue: arXiv cs.LG 2023
Year: 2023

Fixing energy leakage caused by different anomalies can result in significant energy savings and extended appliance life. Further, it assists grid operators in scheduling their resources to meet the actual needs of end users, while helping end users reduce their energy costs. In this paper, we analyze the patterns pertaining to the power consumption of dishwashers used in two houses of the REFIT dataset. Then two autoencoder (AEs) with 1D-CNN and TCN as backbones are trained to differentiate the normal patterns from the abnormal ones. Our results indicate that TCN outperforms CNN1D in detecting anomalies in energy consumption. Finally, the data from the Fridge_Freezer and the Freezer of house No. 3 in REFIT is also used to evaluate our approach.
Interconnect Bandwidth Heterogeneity on AMD MI250x and Infinity Fabric
Venue: arXiv cs.DC 2023
Year: 2023

Demand for low-latency and high-bandwidth data transfer between GPUs has driven the development of multi-GPU nodes. Physical constraints on the manufacture and integration of such systems has yielded heterogeneous intra-node interconnects, where not all devices are connected equally. The next generation of supercomputing platforms are expected to feature AMD CPUs and GPUs. This work characterizes the extent to which interconnect heterogeneity is visible through GPU programming APIs on a system with four AMD MI250x GPUs, and provides several insights for users of such systems.