OpenRaiser/Intern-Atlas huggingface dataset:
Using Data Augmentations and VTLN to Reduce Bias in Dutch End-to-End Speech Recognition Systems
Venue: arXiv cs.CL 2023
Year: 2023

Speech technology has improved greatly for norm speakers, i.e., adult native speakers of a language without speech impediments or strong accents. However, non-norm or diverse speaker groups show a distinct performance gap with norm speakers, which we refer to as bias. In this work, we aim to reduce bias against different age groups and non-native speakers of Dutch. For an end-to-end (E2E) ASR system, we use state-of-the-art speed perturbation and spectral augmentation as data augmentation techniques and explore Vocal Tract Length Normalization (VTLN) to normalise for spectral differences due to differences in anatomy. The combination of data augmentation and VTLN reduced the average WER and bias across various diverse speaker groups by 6.9% and 3.9%, respectively. The VTLN model trained on Dutch was also effective in improving performance of Mandarin Chinese child speech, thus, showing generalisability across languages
Universal Rates for Multiclass Learning
Venue: arXiv cs.LG 2023
Year: 2023

We study universal rates for multiclass classification, establishing the optimal rates (up to log factors) for all hypothesis classes. This generalizes previous results on binary classification (Bousquet, Hanneke, Moran, van Handel, and Yehudayoff, 2021), and resolves an open question studied by Kalavasis, Velegkas, and Karbasi (2022) who handled the multiclass setting with a bounded number of class labels. In contrast, our result applies for any countable label space. Even for finite label space, our proofs provide a more precise bounds on the learning curves, as they do not depend on the number of labels. Specifically, we show that any class admits exponential rates if and only if it has no infinite Littlestone tree, and admits (near-)linear rates if and only if it has no infinite Daniely-Shalev-Shwartz-Littleston (DSL) tree, and otherwise requires arbitrarily slow rates. DSL trees are a new structure we define in this work, in which each node of the tree is given by a pseudo-cube of possible classifications of a given set of points. Pseudo-cubes are a structure, rooted in the work of Daniely and Shalev-Shwartz (2014), and recently shown by Brukhim, Carmon, Dinur, Moran, and Yehudayoff (2022) to characterize PAC learnability (i.e., uniform rates) for multiclass classification. We also resolve an open question of Kalavasis, Velegkas, and Karbasi (2022) regarding the equivalence of classes having infinite Graph-Littlestone (GL) trees versus infinite Natarajan-Littlestone (NL) trees, showing that they are indeed equivalent.
Graph Contrastive Topic Model
Venue: arXiv cs.CL 2023
Year: 2023

Existing NTMs with contrastive learning suffer from the sample bias problem owing to the word frequency-based sampling strategy, which may result in false negative samples with similar semantics to the prototypes. In this paper, we aim to explore the efficient sampling strategy and contrastive learning in NTMs to address the aforementioned issue. We propose a new sampling assumption that negative samples should contain words that are semantically irrelevant to the prototype. Based on it, we propose the graph contrastive topic model (GCTM), which conducts graph contrastive learning (GCL) using informative positive and negative samples that are generated by the graph-based sampling strategy leveraging in-depth correlation and irrelevance among documents and words. In GCTM, we first model the input document as the document word bipartite graph (DWBG), and construct positive and negative word co-occurrence graphs (WCGs), encoded by graph neural networks, to express in-depth semantic correlation and irrelevance among words. Based on the DWBG and WCGs, we design the document-word information propagation (DWIP) process to perform the edge perturbation of DWBG, based on multi-hop correlations/irrelevance among documents and words. This yields the desired negative and positive samples, which will be utilized for GCL together with the prototypes to improve learning document topic representations and latent topics. We further show that GCL can be interpreted as the structured variational graph auto-encoder which maximizes the mutual information of latent topic representations of different perspectives on DWBG. Experiments on several benchmark datasets demonstrate the effectiveness of our method for topic coherence and document representation learning compared with existing SOTA methods.
Incremental Model Transformations with Triple Graph Grammars for Multi-version Models
Venue: arXiv cs.SE 2023
Year: 2023

Like conventional software projects, projects in model-driven software engineering require adequate management of multiple versions of development artifacts, importantly allowing living with temporary inconsistencies. In previous work, multi-version models for model-driven software engineering have been introduced, which allow checking well-formedness and finding merge conflicts for multiple versions of a model at once. However, also for multi-version models, situations where different artifacts, that is, different models, are linked via automatic model transformations have to be handled. In this paper, we propose a technique for jointly handling the transformation of multiple versions of a source model into corresponding versions of a target model, which enables the use of a more compact representation that may afford improved execution time of both the transformation and further analysis operations. Our approach is based on the well-known formalism of triple graph grammars and the aforementioned encoding of model version histories called multi-version models. In addition to batch transformation of an entire model version history, the technique also covers incremental synchronization of changes in the framework of multi-version models. We show the correctness of our approach with respect to the standard semantics of triple graph grammars and conduct an empirical evaluation to investigate the performance of our technique regarding execution time and memory consumption. Our results indicate that the proposed technique affords lower memory consumption and may improve execution time for batch transformation of large version histories, but can also come with computational overhead in unfavorable cases.
Kolam Simulation using Angles at Lattice Points
Venue: arXiv cs.IT 2023
Year: 2023

Kolam is a ritual art form practised by people in South India and consists of rule-bound geometric patterns of dots and lines. Single loop Kolams are mathematical closed loop patterns drawn over a grid of dots and conforming to certain heuristics. In this work, we propose a novel encoding scheme where we map the angular movements of Kolam at lattice points into sequences containing $4$ distinct symbols. This is then used to simulate single loop Kolam procedure via turtle moves in accordance with the desired angular direction at specific points. We thus obtain sequential codes for Kolams, unique up to cyclic permutations. We specify the requirements for the algorithm and indicate the general methodology. We demonstrate a sample of Kolams using our algorithm with a software implementation in Python.
Abstractions, Scenarios, and Prompt Definitions for Process Mining with LLMs: A Case Study
Venue: arXiv cs.DB 2023
Year: 2023

Large Language Models (LLMs) are capable of answering questions in natural language for various purposes. With recent advancements (such as GPT-4), LLMs perform at a level comparable to humans for many proficient tasks. The analysis of business processes could benefit from a natural process querying language and using the domain knowledge on which LLMs have been trained. However, it is impossible to provide a complete database or event log as an input prompt due to size constraints. In this paper, we apply LLMs in the context of process mining by i) abstracting the information of standard process mining artifacts and ii) describing the prompting strategies. We implement the proposed abstraction techniques into pm4py, an open-source process mining library. We present a case study using available event logs. Starting from different abstractions and analysis questions, we formulate prompts and evaluate the quality of the answers.
MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformers
Venue: arXiv cs.CV 2023
Year: 2023

The input tokens to Vision Transformers carry little semantic meaning as they are defined as regular equal-sized patches of the input image, regardless of its content. However, processing uniform background areas of an image should not necessitate as much compute as dense, cluttered areas. To address this issue, we propose a dynamic mixed-scale tokenization scheme for ViT, MSViT. Our method introduces a conditional gating mechanism that selects the optimal token scale for every image region, such that the number of tokens is dynamically determined per input. In addition, to enhance the conditional behavior of the gate during training, we introduce a novel generalization of the batch-shaping loss. We show that our gating module is able to learn meaningful semantics despite operating locally at the coarse patch-level. The proposed gating module is lightweight, agnostic to the choice of transformer backbone, and trained within a few epochs with little training overhead. Furthermore, in contrast to token pruning, MSViT does not lose information about the input, thus can be readily applied for dense tasks. We validate MSViT on the tasks of classification and segmentation where it leads to improved accuracy-complexity trade-off.
Beyond the Library Collections: Proceedings of the 2022 Erasmus Staff Training Week at ULiège Library
Venue: arXiv cs.DL 2023
Year: 2023

No library can buy or hold everything its patrons need. At a certain point, librarians need to pool their resources and collaborate to provide access to what they don't have: Collaboration and partnership, centralized and shared collection storage, digitization projects, interlibrary loan and resource sharing, purchase on demand, PDA and EBA are notably key to success. The 2022 edition of the Erasmus Mobility Staff Training week organized at the University of Liège Library focused on services, projects and policies that libraries can deploy and promote to increase and ease access to materials that do not belong to their print or electronic holdings. More than 20 librarians, managers, and researchers in library science share their experiences and visions in this book. Keywords: Collaboration between libraries; Collection Development; Academic Libraries; Research Libraries; Interlibrary Loan; ILL; Resource Sharing; Purchase on Demand; Patron-Driven Acquisition; PDA; Evidence-Based Acquisition; EBA; Alma; Koha; RapidILL; Impala; Subito; Library systems; Digitalization; Workflows
Fraunhofer SIT at CheckThat! 2023: Tackling Classification Uncertainty Using Model Souping on the Example of Check-Worthiness Classification
Venue: arXiv cs.CL 2023
Year: 2023

This paper describes the second-placed approach developed by the Fraunhofer SIT team in the CLEF-2023 CheckThat! lab Task 1B for English. Given a text snippet from a political debate, the aim of this task is to determine whether it should be assessed for check-worthiness. Detecting check-worthy statements aims to facilitate manual fact-checking efforts by prioritizing the claims that fact-checkers should consider first. It can also be considered as primary step of a fact-checking system. Our best-performing method took advantage of an ensemble classification scheme centered on Model Souping. When applied to the English data set, our submitted model achieved an overall F1 score of 0.878 and was ranked as the second-best model in the competition.
A Versatile Hub Model For Efficient Information Propagation And Feature Selection
Venue: arXiv cs.LG 2023
Year: 2023

Hub structure, characterized by a few highly interconnected nodes surrounded by a larger number of nodes with fewer connections, is a prominent topological feature of biological brains, contributing to efficient information transfer and cognitive processing across various species. In this paper, a mathematical model of hub structure is presented. The proposed method is versatile and can be broadly applied to both computational neuroscience and Recurrent Neural Networks (RNNs) research. We employ the Echo State Network (ESN) as a means to investigate the mechanistic underpinnings of hub structures. Our findings demonstrate a substantial enhancement in performance upon incorporating the hub structure. Through comprehensive mechanistic analyses, we show that the hub structure improves model performance by facilitating efficient information processing and better feature extractions.
The Pseudonymous Certificates for Healthcare Systems
Venue: arXiv cs.CR 2023
Year: 2023

This study mainly modifies the butterfly key expansion (BKE) mechanism and applies it to the healthcare system. The system mainly includes a Root Certificate Authority (RCA), an Enrollment Certificate Authority (ECA), a Pseudonym Certificate Authority (PCA), a Registration Authority (RA), and End Entities (EEs)(i.e. user devices). Certificates can be issued by the RCA to the ECA, PCA, and RA to make them legal entities in the system. The ECA then issues device certificates (similar to identification cards for devices) to the EEs (e.g. blood pressure monitors). When patients use EEs to measure physiological information, the RA verifies that the EE is legal based on the issued multiple pseudonym certificates by the PCA. The EE then uses the pseudonym certificates to send physiological information to the RA, ensuring data integrity and non-repudiation, while also preventing identity information from being stolen. To verify the pseudonymous certificate-based healthcare system proposed in this study, the security of the system was verified using the security strengths defined by the National Institute of Standards and Technology (NIST) in the United States. Furthermore, as the BKE mechanism is primarily based on Elliptic Curve Cryptography (ECC), this study also verified the efficiency under different security strengths. Furthermore, under 256 of security strength, the mean computation time of key expansion is between 24186.584 microseconds and 57894.552 microseconds, so more than 18 public keys could be generated in one second.
FOCUS: Object-Centric World Models for Robotics Manipulation
Venue: arXiv cs.RO 2023
Year: 2023

Understanding the world in terms of objects and the possible interplays with them is an important cognition ability, especially in robotics manipulation, where many tasks require robot-object interactions. However, learning such a structured world model, which specifically captures entities and relationships, remains a challenging and underexplored problem. To address this, we propose FOCUS, a model-based agent that learns an object-centric world model. Thanks to a novel exploration bonus that stems from the object-centric representation, FOCUS can be deployed on robotics manipulation tasks to explore object interactions more easily. Evaluating our approach on manipulation tasks across different settings, we show that object-centric world models allow the agent to solve tasks more efficiently and enable consistent exploration of robot-object interactions. Using a Franka Emika robot arm, we also showcase how FOCUS could be adopted in real-world settings.
Vulnerable Source Code Detection using SonarCloud Code Analysis
Venue: arXiv cs.CY 2023
Year: 2023

In Software Development Life Cycle (SDLC), security vulnerabilities are one of the points introduced during the construction stage. Failure to detect software defects earlier after releasing the product to the market causes higher repair costs for the company. So, it decreases the company's reputation, violates user privacy, and causes an unrepairable issue for the application. The introduction of vulnerability detection enables reducing the number of false alerts to focus the limited testing efforts on potentially vulnerable files. UMKM Masa Kini (UMI) is a Point of Sales application to sell any Micro, Small, and Medium Enterprises Product (UMKM). Therefore, in the current work, we analyze the suitability of these metrics to create Machine Learning based software vulnerability detectors for UMI applications. Code is generated using a commercial tool, SonarCloud. Experimental result shows that there are 3,285 vulnerable rules detected.
Visual Question Answering (VQA) on Images with Superimposed Text
Venue: arXiv cs.CV 2023
Year: 2023

Superimposed text annotations have been under-investigated, yet are ubiquitous, useful and important, especially in medical images. Medical images also highlight the challenges posed by low resolution, noise and superimposed textual meta-information. Therefor we probed the impact of superimposing text onto medical images on VQA. Our results revealed that this textual meta-information can be added without severely degrading key measures of VQA performance. Our findings are significant because they validate the practice of superimposing text on images, even for medical images subjected to the VQA task using AI techniques. The work helps advance understanding of VQA in general and, in particular, in the domain of healthcare and medicine.
mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding
Venue: arXiv cs.CL 2023
Year: 2023

Document understanding refers to automatically extract, analyze and comprehend information from various types of digital documents, such as a web page. Existing Multi-model Large Language Models (MLLMs), including mPLUG-Owl, have demonstrated promising zero-shot capabilities in shallow OCR-free text recognition, indicating their potential for OCR-free document understanding. Nevertheless, without in-domain training, these models tend to ignore fine-grained OCR features, such as sophisticated tables or large blocks of text, which are essential for OCR-free document understanding. In this paper, we propose mPLUG-DocOwl based on mPLUG-Owl for OCR-free document understanding. Specifically, we first construct a instruction tuning dataset featuring a wide range of visual-text understanding tasks. Then, we strengthen the OCR-free document understanding ability by jointly train the model on language-only, general vision-and-language, and document instruction tuning dataset with our unified instruction tuning strategy. We also build an OCR-free document instruction understanding evaluation set LLMDoc to better compare models' capabilities on instruct compliance and document understanding. Experimental results show that our model outperforms existing multi-modal models, demonstrating its strong ability of document understanding. Besides, without specific fine-tuning, mPLUG-DocOwl generalizes well on various downstream tasks. Our code, models, training data and evaluation set are available at https://github.com/X-PLUG/mPLUG-DocOwl.
Exploring new ways: Enforcing representational dissimilarity to learn new features and reduce error consistency
Venue: arXiv cs.LG 2023
Year: 2023

Independently trained machine learning models tend to learn similar features. Given an ensemble of independently trained models, this results in correlated predictions and common failure modes. Previous attempts focusing on decorrelation of output predictions or logits yielded mixed results, particularly due to their reduction in model accuracy caused by conflicting optimization objectives. In this paper, we propose the novel idea of utilizing methods of the representational similarity field to promote dissimilarity during training instead of measuring similarity of trained models. To this end, we promote intermediate representations to be dissimilar at different depths between architectures, with the goal of learning robust ensembles with disjoint failure modes. We show that highly dissimilar intermediate representations result in less correlated output predictions and slightly lower error consistency, resulting in higher ensemble accuracy. With this, we shine first light on the connection between intermediate representations and their impact on the output predictions.
User Perspectives on Branching in Computer-Aided Design
Venue: arXiv cs.HC 2023
Year: 2023

Branching is a feature of distributed version control systems that facilitates the ``divide and conquer'' strategy present in complex and collaborative work domains. Branching has revolutionized modern software development and has the potential to similarly transform hardware product development via CAD (computer-aided design). Yet, contrasting with its status in software, branching as a feature of commercial CAD systems is in its infancy, and little research exists to investigate its use in the digital design and development of physical products. To address this knowledge gap, in this paper, we mine and analyze 719 user-generated posts from online CAD forums to qualitatively study designers' intentions for and preliminary use of branching in CAD. Our work contributes a taxonomy of CAD branching use cases, an identification of deficiencies of existing branching capabilities in CAD, and a discussion of the untapped potential of CAD branching to support a new paradigm of collaborative mechanical design. The insights gained from this study may help CAD tool developers address design shortcomings in CAD branching tools and assist CAD practitioners by raising their awareness of CAD branching to improve design efficiency and collaborative workflows in hardware development teams.
TransformerG2G: Adaptive time-stepping for learning temporal graph embeddings using transformers
Venue: arXiv cs.LG 2023
Year: 2023

Dynamic graph embedding has emerged as a very effective technique for addressing diverse temporal graph analytic tasks (i.e., link prediction, node classification, recommender systems, anomaly detection, and graph generation) in various applications. Such temporal graphs exhibit heterogeneous transient dynamics, varying time intervals, and highly evolving node features throughout their evolution. Hence, incorporating long-range dependencies from the historical graph context plays a crucial role in accurately learning their temporal dynamics. In this paper, we develop a graph embedding model with uncertainty quantification, TransformerG2G, by exploiting the advanced transformer encoder to first learn intermediate node representations from its current state ($t$) and previous context (over timestamps [$t-1, t-l$], $l$ is the length of context). Moreover, we employ two projection layers to generate lower-dimensional multivariate Gaussian distributions as each node's latent embedding at timestamp $t$. We consider diverse benchmarks with varying levels of ``novelty" as measured by the TEA (Temporal Edge Appearance) plots. Our experiments demonstrate that the proposed TransformerG2G model outperforms conventional multi-step methods and our prior work (DynG2G) in terms of both link prediction accuracy and computational efficiency, especially for high degree of novelty. Furthermore, the learned time-dependent attention weights across multiple graph snapshots reveal the development of an automatic adaptive time stepping enabled by the transformer. Importantly, by examining the attention weights, we can uncover temporal dependencies, identify influential elements, and gain insights into the complex interactions within the graph structure. For example, we identified a strong correlation between attention weights and node degree at the various stages of the graph topology evolution.