OpenRaiser/Intern-Atlas huggingface dataset:
Examining User-Friendly and Open-Sourced Large GPT Models: A Survey on Language, Multimodal, and Scientific GPT Models
Venue: arXiv cs.CL 2023
Year: 2023

Generative pre-trained transformer (GPT) models have revolutionized the field of natural language processing (NLP) with remarkable performance in various tasks and also extend their power to multimodal domains. Despite their success, large GPT models like GPT-4 face inherent limitations such as considerable size, high computational requirements, complex deployment processes, and closed development loops. These constraints restrict their widespread adoption and raise concerns regarding their responsible development and usage. The need for user-friendly, relatively small, and open-sourced alternative GPT models arises from the desire to overcome these limitations while retaining high performance. In this survey paper, we provide an examination of alternative open-sourced models of large GPTs, focusing on user-friendly and relatively small models that facilitate easier deployment and accessibility. Through this extensive survey, we aim to equip researchers, practitioners, and enthusiasts with a thorough understanding of user-friendly and relatively small open-sourced models of large GPTs, their current state, challenges, and future research directions, inspiring the development of more efficient, accessible, and versatile GPT models that cater to the broader scientific community and advance the field of general artificial intelligence. The source contents are continuously updating in https://github.com/GPT-Alternatives/gpt_alternatives.
Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIP
Venue: arXiv cs.CL 2023
Year: 2023

Mechanistic interpretability seeks to understand the neural mechanisms that enable specific behaviors in Large Language Models (LLMs) by leveraging causality-based methods. While these approaches have identified neural circuits that copy spans of text, capture factual knowledge, and more, they remain unusable for multimodal models since adapting these tools to the vision-language domain requires considerable architectural changes. In this work, we adapt a unimodal causal tracing tool to BLIP to enable the study of the neural mechanisms underlying image-conditioned text generation. We demonstrate our approach on a visual question answering dataset, highlighting the causal relevance of later layer representations for all tokens. Furthermore, we release our BLIP causal tracing tool as open source to enable further experimentation in vision-language mechanistic interpretability by the community. Our code is available at https://github.com/vedantpalit/Towards-Vision-Language-Mechanistic-Interpretability.
Modeling Player Personality Factors from In-Game Behavior and Affective Expression
Venue: arXiv cs.AI 2023
Year: 2023

Developing a thorough understanding of the target audience (and/or single individuals) is a key factor for success - which is exceptionally important and powerful for the domain of video games that can not only benefit from informed decision making during development, but ideally even tailor game content, difficulty and player experience while playing. The granular assessment of individual personality and differences across players is a particularly difficult endeavor, given the highly variant human nature, disagreement in psychological background models and because of the effortful data collection that most often builds upon long, time-consuming and deterrent questionnaires. In this work, we explore possibilities to predict a series of player personality questionnaire metrics from recorded in-game behavior and extend related work by explicitly adding affective dialog decisions to the game environment which could elevate the model's accuracy. Using random forest regression, we predicted a wide variety of personality metrics from seven established questionnaires across 62 players over 60 minute gameplay of a customized version of the role-playing game Fallout: New Vegas. While some personality variables could already be identified from reasonable underlying in-game actions and affective expressions, we did not find ways to predict others or encountered questionable correlations that could not be justified by theoretical background literature. Yet, building on the initial opportunities of this explorative study, we are striving to massively enlarge our data set to players from an ecologically valid industrial game environment and investigate the performance of more sophisticated machine learning approaches.
Unleash Model Potential: Bootstrapped Meta Self-supervised Learning
Venue: arXiv cs.LG 2023
Year: 2023

The long-term goal of machine learning is to learn general visual representations from a small amount of data without supervision, mimicking three advantages of human cognition: i) no need for labels, ii) robustness to data scarcity, and iii) learning from experience. Self-supervised learning and meta-learning are two promising techniques to achieve this goal, but they both only partially capture the advantages and fail to address all the problems. Self-supervised learning struggles to overcome the drawbacks of data scarcity, while ignoring prior knowledge that can facilitate learning and generalization. Meta-learning relies on supervised information and suffers from a bottleneck of insufficient learning. To address these issues, we propose a novel Bootstrapped Meta Self-Supervised Learning (BMSSL) framework that aims to simulate the human learning process. We first analyze the close relationship between meta-learning and self-supervised learning. Based on this insight, we reconstruct tasks to leverage the strengths of both paradigms, achieving advantages i and ii. Moreover, we employ a bi-level optimization framework that alternates between solving specific tasks with a learned ability (first level) and improving this ability (second level), attaining advantage iii. To fully harness its power, we introduce a bootstrapped target based on meta-gradient to make the model its own teacher. We validate the effectiveness of our approach with comprehensive theoretical and empirical study.
FonMTL: Towards Multitask Learning for the Fon Language
Venue: arXiv cs.CL 2023
Year: 2023

The Fon language, spoken by an average 2 million of people, is a truly low-resourced African language, with a limited online presence, and existing datasets (just to name but a few). Multitask learning is a learning paradigm that aims to improve the generalization capacity of a model by sharing knowledge across different but related tasks: this could be prevalent in very data-scarce scenarios. In this paper, we present the first explorative approach to multitask learning, for model capabilities enhancement in Natural Language Processing for the Fon language. Specifically, we explore the tasks of Named Entity Recognition (NER) and Part of Speech Tagging (POS) for Fon. We leverage two language model heads as encoders to build shared representations for the inputs, and we use linear layers blocks for classification relative to each task. Our results on the NER and POS tasks for Fon, show competitive (or better) performances compared to several multilingual pretrained language models finetuned on single tasks. Additionally, we perform a few ablation studies to leverage the efficiency of two different loss combination strategies and find out that the equal loss weighting approach works best in our case. Our code is open-sourced at https://github.com/bonaventuredossou/multitask_fon.
Symbolic & Acoustic: Multi-domain Music Emotion Modeling for Instrumental Music
Venue: arXiv cs.SD 2023
Year: 2023

Music Emotion Recognition involves the automatic identification of emotional elements within music tracks, and it has garnered significant attention due to its broad applicability in the field of Music Information Retrieval. It can also be used as the upstream task of many other human-related tasks such as emotional music generation and music recommendation. Due to existing psychology research, music emotion is determined by multiple factors such as the Timbre, Velocity, and Structure of the music. Incorporating multiple factors in MER helps achieve more interpretable and finer-grained methods. However, most prior works were uni-domain and showed weak consistency between arousal modeling performance and valence modeling performance. Based on this background, we designed a multi-domain emotion modeling method for instrumental music that combines symbolic analysis and acoustic analysis. At the same time, because of the rarity of music data and the difficulty of labeling, our multi-domain approach can make full use of limited data. Our approach was implemented and assessed using the publicly available piano dataset EMOPIA, resulting in a notable improvement over our baseline model with a 2.4% increase in overall accuracy, establishing its state-of-the-art performance.
Voice Conversion with Denoising Diffusion Probabilistic GAN Models
Venue: arXiv cs.SD 2023
Year: 2023

Voice conversion is a method that allows for the transformation of speaking style while maintaining the integrity of linguistic information. There are many researchers using deep generative models for voice conversion tasks. Generative Adversarial Networks (GANs) can quickly generate high-quality samples, but the generated samples lack diversity. The samples generated by the Denoising Diffusion Probabilistic Models (DDPMs) are better than GANs in terms of mode coverage and sample diversity. But the DDPMs have high computational costs and the inference speed is slower than GANs. In order to make GANs and DDPMs more practical we proposes DiffGAN-VC, a variant of GANs and DDPMS, to achieve non-parallel many-to-many voice conversion (VC). We use large steps to achieve denoising, and also introduce a multimodal conditional GANs to model the denoising diffusion generative adversarial network. According to both objective and subjective evaluation experiments, DiffGAN-VC has been shown to achieve high voice quality on non-parallel data sets. Compared with the CycleGAN-VC method, DiffGAN-VC achieves speaker similarity, naturalness and higher sound quality.
DiffSmooth: Certifiably Robust Learning via Diffusion Models and Local Smoothing
Venue: arXiv cs.LG 2023
Year: 2023

Diffusion models have been leveraged to perform adversarial purification and thus provide both empirical and certified robustness for a standard model. On the other hand, different robustly trained smoothed models have been studied to improve the certified robustness. Thus, it raises a natural question: Can diffusion model be used to achieve improved certified robustness on those robustly trained smoothed models? In this work, we first theoretically show that recovered instances by diffusion models are in the bounded neighborhood of the original instance with high probability; and the "one-shot" denoising diffusion probabilistic models (DDPM) can approximate the mean of the generated distribution of a continuous-time diffusion model, which approximates the original instance under mild conditions. Inspired by our analysis, we propose a certifiably robust pipeline DiffSmooth, which first performs adversarial purification via diffusion models and then maps the purified instances to a common region via a simple yet effective local smoothing strategy. We conduct extensive experiments on different datasets and show that DiffSmooth achieves SOTA-certified robustness compared with eight baselines. For instance, DiffSmooth improves the SOTA-certified accuracy from $36.0\%$ to $53.0\%$ under $\ell_2$ radius $1.5$ on ImageNet. The code is available at [https://github.com/javyduck/DiffSmooth].
Simple Modification of the Upper Confidence Bound Algorithm by Generalized Weighted Averages
Venue: arXiv cs.LG 2023
Year: 2023

The multi-armed bandit (MAB) problem is a classical problem that models sequential decision-making under uncertainty in reinforcement learning. In this study, we propose a new generalized upper confidence bound (UCB) algorithm (GWA-UCB1) by extending UCB1, which is a representative algorithm for MAB problems, using generalized weighted averages, and present an effective algorithm for various problem settings. GWA-UCB1 is a two-parameter generalization of the balance between exploration and exploitation in UCB1 and can be implemented with a simple modification of the UCB1 formula. Therefore, this algorithm can be easily applied to UCB-based reinforcement learning models. In preliminary experiments, we investigated the optimal parameters of a simple generalized UCB1 (G-UCB1), prepared for comparison and GWA-UCB1, in a stochastic MAB problem with two arms. Subsequently, we confirmed the performance of the algorithms with the investigated parameters on stochastic MAB problems when arm reward probabilities were sampled from uniform or normal distributions and on survival MAB problems assuming more realistic situations. GWA-UCB1 outperformed G-UCB1, UCB1-Tuned, and Thompson sampling in most problem settings and can be useful in many situations. The code is available at https://github.com/manome/python-mab.
ZhuJiu: A Multi-dimensional, Multi-faceted Chinese Benchmark for Large Language Models
Venue: arXiv cs.CL 2023
Year: 2023

The unprecedented performance of large language models (LLMs) requires comprehensive and accurate evaluation. We argue that for LLMs evaluation, benchmarks need to be comprehensive and systematic. To this end, we propose the ZhuJiu benchmark, which has the following strengths: (1) Multi-dimensional ability coverage: We comprehensively evaluate LLMs across 7 ability dimensions covering 51 tasks. Especially, we also propose a new benchmark that focuses on knowledge ability of LLMs. (2) Multi-faceted evaluation methods collaboration: We use 3 different yet complementary evaluation methods to comprehensively evaluate LLMs, which can ensure the authority and accuracy of the evaluation results. (3) Comprehensive Chinese benchmark: ZhuJiu is the pioneering benchmark that fully assesses LLMs in Chinese, while also providing equally robust evaluation abilities in English. (4) Avoiding potential data leakage: To avoid data leakage, we construct evaluation data specifically for 37 tasks. We evaluate 10 current mainstream LLMs and conduct an in-depth discussion and analysis of their results. The ZhuJiu benchmark and open-participation leaderboard are publicly released at http://www.zhujiu-benchmark.com/ and we also provide a demo video at https://youtu.be/qypkJ89L1Ic.
Model-based learning for location-to-channel mapping
Venue: arXiv cs.AI 2023
Year: 2023

Modern communication systems rely on accurate channel estimation to achieve efficient and reliable transmission of information. As the communication channel response is highly related to the user's location, one can use a neural network to map the user's spatial coordinates to the channel coefficients. However, these latter are rapidly varying as a function of the location, on the order of the wavelength. Classical neural architectures being biased towards learning low frequency functions (spectral bias), such mapping is therefore notably difficult to learn. In order to overcome this limitation, this paper presents a frugal, model-based network that separates the low frequency from the high frequency components of the target mapping function. This yields an hypernetwork architecture where the neural network only learns low frequency sparse coefficients in a dictionary of high frequency components. Simulation results show that the proposed neural network outperforms standard approaches on realistic synthetic data.
Self-Supervision for Tackling Unsupervised Anomaly Detection: Pitfalls and Opportunities
Venue: arXiv cs.LG 2023
Year: 2023

Self-supervised learning (SSL) is a growing torrent that has recently transformed machine learning and its many real world applications, by learning on massive amounts of unlabeled data via self-generated supervisory signals. Unsupervised anomaly detection (AD) has also capitalized on SSL, by self-generating pseudo-anomalies through various data augmentation functions or external data exposure. In this vision paper, we first underline the importance of the choice of SSL strategies on AD performance, by presenting evidences and studies from the AD literature. Equipped with the understanding that SSL incurs various hyperparameters (HPs) to carefully tune, we present recent developments on unsupervised model selection and augmentation tuning for SSL-based AD. We then highlight emerging challenges and future opportunities; on designing new pretext tasks and augmentation functions for different data modalities, creating novel model selection solutions for systematically tuning the SSL HPs, as well as on capitalizing on the potential of pretrained foundation models on AD through effective density estimation.
On the Statistical Relation of Ultra-Reliable Wireless and Location Estimation
Venue: arXiv cs.IT 2023
Year: 2023

Location information is often used as a proxy to guarantee the performance of a wireless communication link. However, localization errors can result in a significant mismatch with the guarantees, particularly detrimental to users operating the ultra-reliable low-latency communication (URLLC) regime. This paper unveils the fundamental statistical relations between location estimation uncertainty and wireless link reliability, specifically in the context of rate selection for ultra-reliable communication. We start with a simple one-dimensional narrowband Rayleigh fading scenario and build towards a two-dimensional scenario in a rich scattering environment. The wireless link reliability is characterized by the meta-probability, the probability with respect to localization error of exceeding the outage capacity, and by removing other sources of errors in the system, we show that reliability is sensitive to localization errors. The $ε$-outage coherence radius is defined and shown to provide valuable insight into the problem of location-based rate selection. However, it is generally challenging to guarantee reliability without accurate knowledge of the propagation environment. Finally, several rate-selection schemes are proposed, showcasing the problem's dynamics and revealing that properly accounting for the localization error is critical to ensure good performance in terms of reliability and achievable throughput.
Spatio-Temporal Analysis of Patient-Derived Organoid Videos Using Deep Learning for the Prediction of Drug Efficacy
Venue: arXiv cs.CV 2023
Year: 2023

Over the last ten years, Patient-Derived Organoids (PDOs) emerged as the most reliable technology to generate ex-vivo tumor avatars. PDOs retain the main characteristics of their original tumor, making them a system of choice for pre-clinical and clinical studies. In particular, PDOs are attracting interest in the field of Functional Precision Medicine (FPM), which is based upon an ex-vivo drug test in which living tumor cells (such as PDOs) from a specific patient are exposed to a panel of anti-cancer drugs. Currently, the Adenosine Triphosphate (ATP) based cell viability assay is the gold standard test to assess the sensitivity of PDOs to drugs. The readout is measured at the end of the assay from a global PDO population and therefore does not capture single PDO responses and does not provide time resolution of drug effect. To this end, in this study, we explore for the first time the use of powerful large foundation models for the automatic processing of PDO data. In particular, we propose a novel imaging-based high-throughput screening method to assess real-time drug efficacy from a time-lapse microscopy video of PDOs. The recently proposed SAM algorithm for segmentation and DINOv2 model are adapted in a comprehensive pipeline for processing PDO microscopy frames. Moreover, an attention mechanism is proposed for fusing temporal and spatial features in a multiple instance learning setting to predict ATP. We report better results than other non-time-resolved methods, indicating that the temporality of data is an important factor for the prediction of ATP. Extensive ablations shed light on optimizing the experimental setting and automating the prediction both in real-time and for forecasting.
MDS Array Codes With Small Sub-packetization Levels and Small Repair Degrees
Venue: arXiv cs.IT 2023
Year: 2023

High-rate minimum storage regenerating (MSR) codes are known to require a large sub-packetization level, which can make meta-data management difficult and hinder implementation in practical systems. A few maximum distance separable (MDS) array code constructions have been proposed to attain a much smaller sub-packetization level by sacrificing a bit of repair bandwidth. However, to the best of our knowledge, only one construction by Guruswami et al. can support the repair of a failed node without contacting all the surviving nodes. This construction is certainly of theoretical interest but not yet practical due to its requirement for very large code parameters. In this paper, we propose a generic transformation that can convert any $(\overline{n}, \overline{k})$ MSR code with a repair degree of $\overline{d}<\overline{n}-1$ into another $(n=s\overline{n},k)$ MDS array code that supports $d<n-1$ with a small sub-packetization level and $(1+ε)$-optimal repair bandwidth (i.e., $1+ε$ times the optimal value) under a specific condition. We obtain three MDS array codes with small sub-packetization levels and $(1+ε)$-optimal repair bandwidth by applying this transformation to three known MSR codes. All the new MDS array codes have a small repair degree of $d<n-1$ and work for both small and large code parameters.
Solving Fréchet Distance Problems by Algebraic Geometric Methods
Venue: arXiv cs.CG 2023
Year: 2023

We study several polygonal curve problems under the Fréchet distance via algebraic geometric methods. Let $\mathbb{X}_m^d$ and $\mathbb{X}_k^d$ be the spaces of all polygonal curves of $m$ and $k$ vertices in $\mathbb{R}^d$, respectively. We assume that $k \leq m$. Let $\mathcal{R}^d_{k,m}$ be the set of ranges in $\mathbb{X}_m^d$ for all possible metric balls of polygonal curves in $\mathbb{X}_k^d$ under the Fréchet distance. We prove a nearly optimal bound of $O(dk\log (km))$ on the VC dimension of the range space $(\mathbb{X}_m^d,\mathcal{R}_{k,m}^d)$, improving on the previous $O(d^2k^2\log(dkm))$ upper bound and approaching the current $Ω(dk\log k)$ lower bound. Our upper bound also holds for the weak Fréchet distance. We also obtain exact solutions that are hitherto unknown for curve simplification, range searching, nearest neighbor search, and distance oracle.
Turnkey Technology: A Powerful Tool for Cyber Warfare
Venue: arXiv cs.CY 2023
Year: 2023

Turnkey technology has emerged as a game-changing tool in cyber warfare, offering state and non-state actors unprecedented access to advanced cyber capabilities. The advantages of turnkey technology include rapid deployment and adaptability, lower costs and resource requirements, the democratization of cyber warfare capabilities, and enhanced offensive and defensive strategies. However, turnkey technology also introduces significant risks, such as the proliferation of cyber weapons, ethical considerations, potential collateral damage, escalation of conflicts, and legal ramifications. This paper provides a unique perspective on the implications of turnkey technology in cyber warfare, highlighting its advantages, risks, and challenges, as well as the potential strategies for mitigating these concerns. The research's novelty lies in examining real-world examples and proposing a multifaceted approach to address the challenges associated with turnkey technology in cyber warfare. This approach focuses on developing effective cybersecurity measures, establishing international norms and regulations, promoting responsible use and development of turnkey technology, and enhancing global cooperation on cyber warfare issues. By adopting this accountable and balanced approach, governments, organizations, and the international community can work together to create a more secure and stable digital environment, leveraging the benefits of turnkey technology while minimizing the associated risks and challenges.