OpenRaiser/Intern-Atlas huggingface dataset:
Large-Scale Cell Representation Learning via Divide-and-Conquer Contrastive Learning
Venue: arXiv cs.CE 2023
Year: 2023

Single-cell RNA sequencing (scRNA-seq) data is a potent tool for comprehending the "language of life" and can provide insights into various downstream biomedical tasks. Large-scale language models (LLMs) are starting to be used for cell representation learning. However, current LLM-based cell representation learning methods depend solely on the BERT architecture, causing an anisotropic embedding space that leads to inefficient semantic representation. Contrastive learning alleviates this problem by distributing the embeddings uniformly. As a larger batch size in contrastive learning results in better representation, the practical application of contrastive learning in cell representation learning is hampered by the high dimensionality of scRNA-seq data and the large parameter volume of LLMs. To address the batch size limitation, we propose a novel divide-and-conquer contrastive learning approach to decouple the batch size from the GPU memory size for cell representation learning. Based on our divide-and-conquer contrastive learning approach, we introduce Single-Cell Language Model CellLM, a large-scale cell representation learning model to handle high-dimensional scRNA-seq data with tens of thousands of genes. CellLM has over 50 million parameters trained with 2 million scRNA-seq data and makes the first attempt to learn cell language models from both normal cells and cancer cells. CellLM achieves new state-of-the-art (SOTA) results in all evaluated downstream tasks: including a 71.8 F_1-score for cell type annotation (a 3.0% absolute improvement over scBERT), an average F_1-score of 88.9 for single-cell drug sensitivity prediction in a few-shot scenario (an 8.3% absolute improvement), and a 93.4 Pearson's correlation for single-omics cell line drug sensitivity prediction (a 6.2% absolute improvement).
Meta-Learning in Spiking Neural Networks with Reward-Modulated STDP
Venue: arXiv cs.AI 2023
Year: 2023

The human brain constantly learns and rapidly adapts to new situations by integrating acquired knowledge and experiences into memory. Developing this capability in machine learning models is considered an important goal of AI research since deep neural networks perform poorly when there is limited data or when they need to adapt quickly to new unseen tasks. Meta-learning models are proposed to facilitate quick learning in low-data regimes by employing absorbed information from the past. Although some models have recently been introduced that reached high-performance levels, they are not biologically plausible. We have proposed a bio-plausible meta-learning model inspired by the hippocampus and the prefrontal cortex using spiking neural networks with a reward-based learning system. Our proposed model includes a memory designed to prevent catastrophic forgetting, a phenomenon that occurs when meta-learning models forget what they have learned as soon as the new task begins. Also, our new model can easily be applied to spike-based neuromorphic devices and enables fast learning in neuromorphic hardware. The final analysis will discuss the implications and predictions of the model for solving few-shot classification tasks. In solving these tasks, our model has demonstrated the ability to compete with the existing state-of-the-art meta-learning techniques.
A Gamified Interaction with a Humanoid Robot to explain Therapeutic Procedures in Pediatric Asthma
Venue: arXiv cs.AI 2023
Year: 2023

In chronic diseases, obtaining a correct diagnosis and providing the most appropriate treatments often is not enough to guarantee an improvement of the clinical condition of a patient. Poor adherence to medical prescriptions constitutes one of the main causes preventing achievement of therapeutic goals. This is generally true especially for certain diseases and specific target patients, such as children. An engaging and entertaining technology can be exploited in support of clinical practices to achieve better health outcomes. Our assumption is that a gamified session with a humanoid robot, compared to the usual methodologies for therapeutic education, can be more incisive in learning the correct inhalation procedure in children affected by asthma. In this perspective, we describe an interactive module implemented on the Pepper robotic platform and the setting of a study that was planned in 2020 to be held at the Pneumoallergology Pediatric clinic of CNR in Palermo. The study was canceled due to the COVID-19 pandemic. Our long-term goal is to assess, by means of a qualitative-quantitative survey plan, the impact of such an educational action, evaluating possible improvement in the adherence to the treatment.
Examining Bias in Opinion Summarisation Through the Perspective of Opinion Diversity
Venue: arXiv cs.CL 2023
Year: 2023

Opinion summarisation is a task that aims to condense the information presented in the source documents while retaining the core message and opinions. A summary that only represents the majority opinions will leave the minority opinions unrepresented in the summary. In this paper, we use the stance towards a certain target as an opinion. We study bias in opinion summarisation from the perspective of opinion diversity, which measures whether the model generated summary can cover a diverse set of opinions. In addition, we examine opinion similarity, a measure of how closely related two opinions are in terms of their stance on a given topic, and its relationship with opinion diversity. Through the lens of stances towards a topic, we examine opinion diversity and similarity using three debatable topics under COVID-19. Experimental results on these topics revealed that a higher degree of similarity of opinions did not indicate good diversity or fairly cover the various opinions originally presented in the source documents. We found that BART and ChatGPT can better capture diverse opinions presented in the source documents.
STEPS: A Benchmark for Order Reasoning in Sequential Tasks
Venue: arXiv cs.CL 2023
Year: 2023

Various human activities can be abstracted into a sequence of actions in natural text, i.e. cooking, repairing, manufacturing, etc. Such action sequences heavily depend on the executing order, while disorder in action sequences leads to failure of further task execution by robots or AI agents. Therefore, to verify the order reasoning capability of current neural models in sequential tasks, we propose a challenging benchmark , named STEPS. STEPS involves two subtask settings, focusing on determining the rationality of given next step in recipes and selecting the reasonable step from the multi-choice question, respectively. We describe the data construction and task formulations, and benchmark most of significant Large Language Models (LLMs). The experimental results demonstrate 1) The commonsense reasoning of action orders in sequential tasks are challenging to resolve via zero-shot prompting or few-shot in-context learning for LLMs; 2) Prompting method still significantly lags behind tuning-based method on STEPS.
Hardening and Speeding Up Zero-interaction Pairing and Authentication
Venue: arXiv cs.CR 2023
Year: 2023

Establishing and maintaining secure communications in the Internet of Things (IoT) is vital to protect smart devices. Zero-interaction pairing (ZIP) and zero-interaction authentication (ZIA) enable IoT devices to establish and maintain secure communications without user interaction by utilizing devices' ambient context, e.g., audio. For autonomous operation, ZIP and ZIA require the context to have enough entropy to resist attacks and complete in a timely manner. Despite the low-entropy context being the norm, like inside an unoccupied room, the research community has yet to come up with ZIP and ZIA schemes operating under such conditions. We propose HardZiPA, a novel approach that turns commodity IoT actuators into injecting devices, generating high-entropy context. Here, we combine the capability of IoT actuators to impact the environment, e.g., emitting a sound, with a pseudorandom number generator (PRNG) featured by many actuators to craft hard-to-predict context stimuli. To demonstrate the feasibility of HardZiPA, we implement it on off-the-shelf IoT actuators, i.e., smart speakers, lights, and humidifiers. We comprehensively evaluate HardZiPA, collecting over 80 hours of various context data in real-world scenarios. Our results show that HardZiPA is able to thwart advanced active attacks on ZIP and ZIA schemes, while doubling the amount of context entropy in many cases, which allows two times faster pairing and authentication.
An Interpretive Framework for Narrower Immunity Under Section 230 of the Communications Decency Act
Venue: arXiv cs.CY 2023
Year: 2023

Almost all courts to interpret Section 230 of the Communications Decency Act have construed its ambiguously worded immunity provision broadly, shielding Internet intermediaries from tort liability so long as they are not the literal authors of offensive content. Although this broad interpretation effects the basic goals of the statute, it ignores several serious textual difficulties and mistakenly extends protection too far by immunizing even direct participants in tortuous conduct. This analysis, which examines the text and history of Section 230 in light of two strains of pre-Internet vicarious liability defamation doctrine, concludes that the immunity provision of Section 230, though broad, was not intended to abrogate entirely traditional common law notions of vicarious liability. Some bases of vicarious liability remain, and their continuing validity both explains the textual puzzles courts have faced in applying Section 230 and undergirds the push by a small minority of courts to narrow the section's immunity provision.
FoSp: Focus and Separation Network for Early Smoke Segmentation
Venue: arXiv cs.CV 2023
Year: 2023

Early smoke segmentation (ESS) enables the accurate identification of smoke sources, facilitating the prompt extinguishing of fires and preventing large-scale gas leaks. But ESS poses greater challenges than conventional object and regular smoke segmentation due to its small scale and transparent appearance, which can result in high miss detection rate and low precision. To address these issues, a Focus and Separation Network (FoSp) is proposed. We first introduce a Focus module employing bidirectional cascade which guides low-resolution and high-resolution features towards mid-resolution to locate and determine the scope of smoke, reducing the miss detection rate. Next, we propose a Separation module that separates smoke images into a pure smoke foreground and a smoke-free background, enhancing the contrast between smoke and background fundamentally, improving segmentation precision. Finally, a Domain Fusion module is developed to integrate the distinctive features of the two modules which can balance recall and precision to achieve high F_beta. Futhermore, to promote the development of ESS, we introduce a high-quality real-world dataset called SmokeSeg, which contains more small and transparent smoke than the existing datasets. Experimental results show that our model achieves the best performance on three available datasets: SYN70K (mIoU: 83.00%), SMOKE5K (F_beta: 81.6%) and SmokeSeg (F_beta: 72.05%). Especially, our FoSp outperforms SegFormer by 7.71% (F_beta) for early smoke segmentation on SmokeSeg.
Vulnerable Smart Contract Function Locating Based on Multi-Relational Nested Graph Convolutional Network
Venue: arXiv cs.CR 2023
Year: 2023

The immutable and trustable characteristics of blockchain enable smart contracts to be applied in various fields. Unfortunately, smart contracts are subject to various vulnerabilities, which are frequently exploited by attackers, causing financial damage to users.In this paper, we study the problem of vulnerable smart contract function locating. We construct a novel Multi-Relational Nested contract Graph (MRNG) to better characterize the rich syntactic and semantic information in the smart contract code, including the relationships between data and instructions. An MRNG represents a smart contract, where each node represents a function in the smart contract and each edge describes the calling relationship between the functions. In addition, we create a Multi-Relational Function Graph (MRFG) for each function, which characterizes the corresponding function code. That is, each function is characterized as an MRFG, which corresponds to a node in the MRNG. Each MRFG uses different types of edges to represent the different control and data relationships between nodes within a function. We also propose a Multi-Relational Nested Graph Convolutional Network (MRN-GCN) to process the MRNG. MRN-GCN first extracts and aggregates features from each MRFG, using the edge-enhanced graph convolution network and self-attention mechanism. The extracted feature vector is then assigned to the corresponding node in the MRNG to obtain a new Featured Contract Graph (FCG) for the smart contract. Graph convolution is used to further extract features from the FCG. Finally, a feed forward network with a Sigmoid function is used to locate the vulnerable functions. Experimental results on the real-world smart contract datasets show that model MRN-GCN can effectively improve the accuracy, precision, recall and F1-score performance of vulnerable smart contract function locating.
Artificial Intelligence can facilitate selfish decisions by altering the appearance of interaction partners
Venue: arXiv cs.AI 2023
Year: 2023

The increasing prevalence of image-altering filters on social media and video conferencing technologies has raised concerns about the ethical and psychological implications of using Artificial Intelligence (AI) to manipulate our perception of others. In this study, we specifically investigate the potential impact of blur filters, a type of appearance-altering technology, on individuals' behavior towards others. Our findings consistently demonstrate a significant increase in selfish behavior directed towards individuals whose appearance is blurred, suggesting that blur filters can facilitate moral disengagement through depersonalization. These results emphasize the need for broader ethical discussions surrounding AI technologies that modify our perception of others, including issues of transparency, consent, and the awareness of being subject to appearance manipulation by others. We also emphasize the importance of anticipatory experiments in informing the development of responsible guidelines and policies prior to the widespread adoption of such technologies.
Lenient Evaluation of Japanese Speech Recognition: Modeling Naturally Occurring Spelling Inconsistency
Venue: arXiv cs.CL 2023
Year: 2023

Word error rate (WER) and character error rate (CER) are standard metrics in Speech Recognition (ASR), but one problem has always been alternative spellings: If one's system transcribes adviser whereas the ground truth has advisor, this will count as an error even though the two spellings really represent the same word. Japanese is notorious for ``lacking orthography'': most words can be spelled in multiple ways, presenting a problem for accurate ASR evaluation. In this paper we propose a new lenient evaluation metric as a more defensible CER measure for Japanese ASR. We create a lattice of plausible respellings of the reference transcription, using a combination of lexical resources, a Japanese text-processing system, and a neural machine translation model for reconstructing kanji from hiragana or katakana. In a manual evaluation, raters rated 95.4% of the proposed spelling variants as plausible. ASR results show that our method, which does not penalize the system for choosing a valid alternate spelling of a word, affords a 2.4%-3.1% absolute reduction in CER depending on the task.
Top-Down Knowledge Compilation for Counting Modulo Theories
Venue: arXiv cs.AI 2023
Year: 2023

Propositional model counting (#SAT) can be solved efficiently when the input formula is in deterministic decomposable negation normal form (d-DNNF). Translating an arbitrary formula into a representation that allows inference tasks, such as counting, to be performed efficiently, is called knowledge compilation. Top-down knowledge compilation is a state-of-the-art technique for solving #SAT problems that leverages the traces of exhaustive DPLL search to obtain d-DNNF representations. While knowledge compilation is well studied for propositional approaches, knowledge compilation for the (quantifier free) counting modulo theory setting (#SMT) has been studied to a much lesser degree. In this paper, we discuss compilation strategies for #SMT. We specifically advocate for a top-down compiler based on the traces of exhaustive DPLL(T) search.
Secure Integrated Sensing and Communication Exploiting Target Location Distribution
Venue: arXiv cs.IT 2023
Year: 2023

In this paper, we study a secure integrated sensing and communication (ISAC) system where one multi-antenna base station (BS) simultaneously serves a downlink communication user and senses the location of a target that may potentially serve as an eavesdropper via its reflected echo signals. Specifically, the location information of the target is unknown and random, while its a priori distribution is available for exploitation. First, to characterize the sensing performance, we derive the posterior Cramér-Rao bound (PCRB) which is a lower bound of the mean squared error (MSE) for target sensing exploiting prior distribution. Due to the intractability of the PCRB expression, we further derive a novel approximate upper bound of it which has a closed-form expression. Next, under an artificial noise (AN) based beamforming structure at the BS to alleviate information eavesdropping and enhance the target's reflected signal power for sensing, we formulate a transmit beamforming optimization problem to maximize the worst-case secrecy rate among all possible target (eavesdropper) locations, under a sensing accuracy threshold characterized by an upper bound on the PCRB. Despite the non-convexity of the formulated problem, we propose a two-stage approach to obtain its optimal solution by leveraging the semi-definite relaxation (SDR) technique. Numerical results validate the effectiveness of our proposed transmit beamforming design and demonstrate the non-trivial trade-off between secrecy performance and sensing performance in secure ISAC systems.
Divide and Repair: Using Options to Improve Performance of Imitation Learning Against Adversarial Demonstrations
Venue: arXiv cs.LG 2023
Year: 2023

We consider the problem of learning to perform a task from demonstrations given by teachers or experts, when some of the experts' demonstrations might be adversarial and demonstrate an incorrect way to perform the task. We propose a novel technique that can identify parts of demonstrated trajectories that have not been significantly modified by the adversary and utilize them for learning, using temporally extended policies or options. We first define a trajectory divergence measure based on the spatial and temporal features of demonstrated trajectories to detect and discard parts of the trajectories that have been significantly modified by an adversarial expert, and, could degrade the learner's performance, if used for learning, We then use an options-based algorithm that partitions trajectories and learns only from the parts of trajectories that have been determined as admissible. We provide theoretical results of our technique to show that repairing partial trajectories improves the sample efficiency of the demonstrations without degrading the learner's performance. We then evaluate the proposed algorithm for learning to play an Atari-like, computer-based game called LunarLander in the presence of different types and degrees of adversarial attacks of demonstrated trajectories. Our experimental results show that our technique can identify adversarially modified parts of the demonstrated trajectories and successfully prevent the learning performance from degrading due to adversarial demonstrations.
Empowering Business Transformation: The Positive Impact and Ethical Considerations of Generative AI in Software Product Management -- A Systematic Literature Review
Venue: arXiv cs.SE 2023
Year: 2023

Generative Artificial Intelligence (GAI) has made outstanding strides in recent years, with a good-sized impact on software product management. Drawing on pertinent articles from 2016 to 2023, this systematic literature evaluation reveals generative AI's potential applications, benefits, and constraints in this area. The study shows that technology can assist in idea generation, market research, customer insights, product requirements engineering, and product development. It can help reduce development time and costs through automatic code generation, customer feedback analysis, and more. However, the technology's accuracy, reliability, and ethical consideration persist. Ultimately, generative AI's practical application can significantly improve software product management activities, leading to more efficient use of resources, better product outcomes, and improved end-user experiences.
ModuleFormer: Modularity Emerges from Mixture-of-Experts
Venue: arXiv cs.CL 2023
Year: 2023

Large Language Models (LLMs) have achieved remarkable results. However, existing models are expensive to train and deploy, and it is also difficult to expand their knowledge beyond pre-training data without forgetting previous knowledge. This paper proposes a new neural network architecture, ModuleFormer, that leverages modularity to improve the efficiency and flexibility of large language models. ModuleFormer is based on the Sparse Mixture of Experts (SMoE). Unlike the previous SMoE-based modular language model, which requires domain-labeled data to learn domain-specific experts, ModuleFormer can induce modularity from uncurated data with its new load balancing and concentration losses. ModuleFormer is a modular architecture that includes two different types of modules: new stick-breaking attention heads and feedforward experts. Different modules are sparsely activated conditions on the input token during training and inference. In our experiment, we found that the modular architecture enables three important abilities for large pre-trained language models: 1) Efficiency, since ModuleFormer only activates a subset of its modules for each input token, thus it could achieve the same performance as dense LLMs with more than two times throughput; 2) Extendability, ModuleFormer is more immune to catastrophic forgetting than dense LLMs and can be easily extended with new modules to learn new knowledge that is not included in the training data; 3) Specialisation, finetuning ModuleFormer could specialize a subset of modules to the finetuning task and the task-unrelated modules could be easily pruned for a lightweight deployment.