OpenRaiser/Intern-Atlas huggingface dataset:
Preconditioned Visual Language Inference with Weak Supervision
Venue: arXiv cs.CL 2023
Year: 2023

Humans can infer the affordance of objects by extracting related contextual preconditions for each scenario. For example, upon seeing an image of a broken cup, we can infer that this precondition prevents the cup from being used for drinking. Reasoning with preconditions of commonsense is studied in NLP where the model explicitly gets the contextual precondition. However, it is unclear if SOTA visual language models (VLMs) can extract such preconditions and infer the affordance of objects with them. In this work, we introduce the task of preconditioned visual language inference and rationalization (PVLIR). We propose a learning resource based on three strategies to retrieve weak supervision signals for the task and develop a human-verified test set for evaluation. Our results reveal the shortcomings of SOTA VLM models in the task and draw a road map to address the challenges ahead in improving them.
Data Science in an Agent-Based Simulation World
Venue: arXiv cs.CY 2023
Year: 2023

In data science education, the importance of learning to solve real-world problems has been argued. However, there are two issues with this approach: (1) it is very costly to prepare multiple real-world problems (using real data) according to the learning objectives, and (2) the learner must suddenly tackle complex real-world problems immediately after learning from a textbook using ideal data. To solve these issues, this paper proposes data science teaching material that uses agent-based simulation (ABS). The proposed teaching material consists of an ABS model and an ABS story. To solve issue 1, the scenario of the problem can be changed according to the learning objectives by setting the appropriate parameters of the ABS model. To solve issue 2, the difficulty level of the tasks can be adjusted by changing the description in the ABS story. We show that, by using this teaching material, the learner can simulate the typical tasks performed by a data scientist in a step-by-step manner (causal inference, data understanding, hypothesis building, data collection, data wrangling, data analysis, and hypothesis testing). The teaching material described in this paper focuses on causal inference as the learning objectives and infectious diseases as the model theme for ABS, but ABS is used as a model to reproduce many types of social phenomena, and its range of expression is extremely wide. Therefore, we expect that the proposed teaching material will inspire the construction of teaching material for various objectives in data science education.
Edit Distance based RL for RNNT decoding
Venue: arXiv cs.SD 2023
Year: 2023

RNN-T is currently considered the industry standard in ASR due to its exceptional WERs in various benchmark tests and its ability to support seamless streaming and longform transcription. However, its biggest drawback lies in the significant discrepancy between its training and inference objectives. During training, RNN-T maximizes all alignment probabilities by teacher forcing, while during inference, it uses beam search which may not necessarily find the maximum probable alignment. Additionally, RNN-T's inability to experience mistakes during teacher forcing training makes it more problematic when a mistake occurs in inference. To address this issue, this paper proposes a Reinforcement Learning method that minimizes the gap between training and inference time. Our Edit Distance based RL (EDRL) approach computes rewards based on the edit distance, and trains the network at every action level. The proposed approach yielded SoTA WERs on LibriSpeech for the 600M Conformer RNN-T model.
AI and the creative realm: A short review of current and future applications
Venue: arXiv cs.AI 2023
Year: 2023

This study explores the concept of creativity and artificial intelligence (AI) and their recent integration. While AI has traditionally been perceived as incapable of generating new ideas or creating art, the development of more sophisticated AI models and the proliferation of human-computer interaction tools have opened up new possibilities for AI in artistic creation. This study investigates the various applications of AI in a creative context, differentiating between the type of art, language, and algorithms used. It also considers the philosophical implications of AI and creativity, questioning whether consciousness can be researched in machines and AI's potential interests and decision-making capabilities. Overall, we aim to stimulate a reflection on AI's use and ethical implications in creative contexts.
Word Embeddings for Banking Industry
Venue: arXiv cs.CL 2023
Year: 2023

Applications of Natural Language Processing (NLP) are plentiful, from sentiment analysis to text classification. Practitioners rely on static word embeddings (e.g. Word2Vec or GloVe) or static word representation from contextual models (e.g. BERT or ELMo) to perform many of these NLP tasks. These widely available word embeddings are built from large amount of text, so they are likely to have captured most of the vocabulary in different context. However, how well would they capture domain-specific semantics and word relatedness? This paper explores this idea by creating a bank-specific word embeddings and evaluates them against other sources of word embeddings such as GloVe and BERT. Not surprising that embeddings built from bank-specific corpora does a better job of capturing the bank-specific semantics and word relatedness. This finding suggests that bank-specific word embeddings could be a good stand-alone source or a complement to other widely available embeddings when performing NLP tasks specific to the banking industry.
Adversarial Attack Based on Prediction-Correction
Venue: arXiv cs.CR 2023
Year: 2023

Deep neural networks (DNNs) are vulnerable to adversarial examples obtained by adding small perturbations to original examples. The added perturbations in existing attacks are mainly determined by the gradient of the loss function with respect to the inputs. In this paper, the close relationship between gradient-based attacks and the numerical methods for solving ordinary differential equation (ODE) is studied for the first time. Inspired by the numerical solution of ODE, a new prediction-correction (PC) based adversarial attack is proposed. In our proposed PC-based attack, some existing attack can be selected to produce a predicted example first, and then the predicted example and the current example are combined together to determine the added perturbations. The proposed method possesses good extensibility and can be applied to all available gradient-based attacks easily. Extensive experiments demonstrate that compared with the state-of-the-art gradient-based adversarial attacks, our proposed PC-based attacks have higher attack success rates, and exhibit better transferability.
Prototyping the use of Large Language Models (LLMs) for adult learning content creation at scale
Venue: arXiv cs.CY 2023
Year: 2023

As Large Language Models (LLMs) and other forms of Generative AI permeate various aspects of our lives, their application for learning and education has provided opportunities and challenges. This paper presents an investigation into the use of LLMs in asynchronous course creation, particularly within the context of adult learning, training and upskilling. We developed a course prototype leveraging an LLM, implementing a robust human-in-the-loop process to ensure the accuracy and clarity of the generated content. Our research questions focus on the feasibility of LLMs to produce high-quality adult learning content with reduced human involvement. Initial findings indicate that taking this approach can indeed facilitate faster content creation without compromising on accuracy or clarity, marking a promising advancement in the field of Generative AI for education. Despite some limitations, the study underscores the potential of LLMs to transform the landscape of learning and education, necessitating further research and nuanced discussions about their strategic and ethical use in learning design.
Heart Diseases Prediction Using Block-chain and Machine Learning
Venue: arXiv cs.LG 2023
Year: 2023

Most people around the globe are dying due to heart disease. The main reason behind the rapid increase in the death rate due to heart disease is that there is no infrastructure developed for the healthcare department that can provide a secure way of data storage and transmission. Due to redundancy in the patient data, it is difficult for cardiac Professionals to predict the disease early on. This rapid increase in the death rate due to heart disease can be controlled by monitoring and eliminating some of the key attributes in the early stages such as blood pressure, cholesterol level, body weight, and addiction to smoking. Patient data can be monitored by cardiac Professionals (Cp) by using the advanced framework in the healthcare departments. Blockchain is the world's most reliable provider. The use of advanced systems in the healthcare departments providing new ways of dealing with diseases has been developed as well. In this article Machine Learning (ML) algorithm known as a sine-cosine weighted k-nearest neighbor (SCA-WKNN) is used for predicting the Hearth disease with the maximum accuracy among the existing approaches. Blockchain technology has been used in the research to secure the data throughout the session and can give more accurate results using this technology. The performance of the system can be improved by using this algorithm and the dataset proposed has been improved by using different resources as well.
Unifying (Machine) Vision via Counterfactual World Modeling
Venue: arXiv cs.CV 2023
Year: 2023

Leading approaches in machine vision employ different architectures for different tasks, trained on costly task-specific labeled datasets. This complexity has held back progress in areas, such as robotics, where robust task-general perception remains a bottleneck. In contrast, "foundation models" of natural language have shown how large pre-trained neural networks can provide zero-shot solutions to a broad spectrum of apparently distinct tasks. Here we introduce Counterfactual World Modeling (CWM), a framework for constructing a visual foundation model: a unified, unsupervised network that can be prompted to perform a wide variety of visual computations. CWM has two key components, which resolve the core issues that have hindered application of the foundation model concept to vision. The first is structured masking, a generalization of masked prediction methods that encourages a prediction model to capture the low-dimensional structure in visual data. The model thereby factors the key physical components of a scene and exposes an interface to them via small sets of visual tokens. This in turn enables CWM's second main idea -- counterfactual prompting -- the observation that many apparently distinct visual representations can be computed, in a zero-shot manner, by comparing the prediction model's output on real inputs versus slightly modified ("counterfactual") inputs. We show that CWM generates high-quality readouts on real-world images and videos for a diversity of tasks, including estimation of keypoints, optical flow, occlusions, object segments, and relative depth. Taken together, our results show that CWM is a promising path to unifying the manifold strands of machine vision in a conceptually simple foundation.
Efficient Multi-Task and Transfer Reinforcement Learning with Parameter-Compositional Framework
Venue: arXiv cs.RO 2023
Year: 2023

In this work, we investigate the potential of improving multi-task training and also leveraging it for transferring in the reinforcement learning setting. We identify several challenges towards this goal and propose a transferring approach with a parameter-compositional formulation. We investigate ways to improve the training of multi-task reinforcement learning which serves as the foundation for transferring. Then we conduct a number of transferring experiments on various manipulation tasks. Experimental results demonstrate that the proposed approach can have improved performance in the multi-task training stage, and further show effective transferring in terms of both sample efficiency and performance.
Multi-View Multi-Task Representation Learning for Mispronunciation Detection
Venue: arXiv cs.SD 2023
Year: 2023

The disparity in phonology between learner's native (L1) and target (L2) language poses a significant challenge for mispronunciation detection and diagnosis (MDD) systems. This challenge is further intensified by lack of annotated L2 data. This paper proposes a novel MDD architecture that exploits multiple `views' of the same input data assisted by auxiliary tasks to learn more distinctive phonetic representation in a low-resource setting. Using the mono- and multilingual encoders, the model learn multiple views of the input, and capture the sound properties across diverse languages and accents. These encoded representations are further enriched by learning articulatory features in a multi-task setup. Our reported results using the L2-ARCTIC data outperformed the SOTA models, with a phoneme error rate reduction of 11.13% and 8.60% and absolute F1 score increase of 5.89%, and 2.49% compared to the single-view mono- and multilingual systems, with a limited L2 dataset.
Type-based Qubit Allocation for a First-Order Quantum Programming Language
Venue: arXiv cs.PL 2023
Year: 2023

Qubit allocation is a process to assign physical qubits to logical qubits in a quantum program. Since some quantum computers have connectivity constraints on applications of two-qubit operations, it is mainly concerned with finding an assignment and inserting instructions to satisfy the connectivity constraints. Many methods have been proposed for the qubit allocation problem for low-level quantum programs. This paper presents a type-based framework of qubit allocation for a quantum programming language with first-order functions. In our framework, the connectivity constraints are expressed by a simple graph of qubits called a coupling graph. We formalize (1) the source language, whose type system verifies that the number of qubits required for a given program to run does not exceed the number of nodes of the coupling graph, (2) the target language, whose qualified type system verifies that a well-typed program satisfies the connectivity constraints, and (3) an algorithm to translate a source program into a target program. We prove that both languages are type-safe and that the translation algorithm is type preserving.
Systemic Risk and Vulnerability Analysis of Multi-cloud Environments
Venue: arXiv cs.CR 2023
Year: 2023

With the increasing use of multi-cloud environments, security professionals face challenges in configuration, management, and integration due to uneven security capabilities and features among providers. As a result, a fragmented approach toward security has been observed, leading to new attack vectors and potential vulnerabilities. Other research has focused on single-cloud platforms or specific applications of multi-cloud environments. Therefore, there is a need for a holistic security and vulnerability assessment and defense strategy that applies to multi-cloud platforms. We perform a risk and vulnerability analysis to identify attack vectors from software, hardware, and the network, as well as interoperability security issues in multi-cloud environments. Applying the STRIDE and DREAD threat modeling methods, we present an analysis of the ecosystem across six attack vectors: cloud architecture, APIs, authentication, automation, management differences, and cybersecurity legislation. We quantitatively determine and rank the threats in multi-cloud environments and suggest mitigation strategies.
Optimal Control of Connected Automated Vehicles with Event-Triggered Control Barrier Functions: a Test Bed for Safe Optimal Merging
Venue: arXiv cs.RO 2023
Year: 2023

We address the problem of controlling Connected and Automated Vehicles (CAVs) in conflict areas of a traffic network subject to hard safety constraints. It has been shown that such problems can be solved through a combination of tractable optimal control problems and Control Barrier Functions (CBFs) that guarantee the satisfaction of all constraints. These solutions can be reduced to a sequence of Quadratic Programs (QPs) which are efficiently solved on line over discrete time steps. However, guaranteeing the feasibility of the CBF-based QP method within each discretized time interval requires the careful selection of time steps which need to be sufficiently small. This creates computational requirements and communication rates between agents which may hinder the controller's application to real CAVs. In this paper, we overcome this limitation by adopting an event-triggered approach for CAVs in a conflict area such that the next QP is triggered by properly defined events with a safety guarantee. We present a laboratory-scale test bed we have developed to emulate merging roadways using mobile robots as CAVs which can be used to demonstrate how the event-triggered scheme is computationally efficient and can handle measurement uncertainties and noise compared to time-driven control while guaranteeing safety.
Consumer's Behavior Analysis of Electric Vehicle using Cloud Computing in the State of New York
Venue: arXiv cs.CY 2023
Year: 2023

Sales of Electric Vehicles (EVs) in the United States have grown fast in the past decade. We analyze the Electric Vehicle Drive Clean Rebate data from the New York State Energy Research and Development Authority (NYSERDA) to understand consumer behavior in EV purchasing and their potential environmental impact. Based on completed rebate applications since 2017, this dataset features the make and model of the EV that consumers purchased, the geographic location of EV consumers, transaction type to obtain the EV, projected environmental impact, and tax incentive issued. This analysis consists of a mapped and calculated statistical data analysis over an established period. Using the SAP Analytics Cloud (SAC), we first import and clean the data to generate statistical snapshots for some primary attributes. Next, different EV options were evaluated based on environmental carbon footprints and rebate amounts. Finally, visualization, geo, and time-series analysis presented further insights and recommendations. This analysis helps the reader to understand consumers' EV buying behavior, such as the change of most popular maker and model over time, acceptance of EVs in different regions in New York State, and funds required to support clean air initiatives. Conclusions from the current study will facilitate the use of renewable energy, reduce reliance on fossil fuels, and accelerate economic growth sustainably, in addition to analyzing the trend of rebate funding size over the years and predicting future funding.
Hierarchical Quadratic Random Forest Classifier
Venue: arXiv cs.LG 2023
Year: 2023

In this paper, we proposed a hierarchical quadratic random forest classifier for classifying multiresolution samples extracted from multichannel data. This forest incorporated a penalized multivariate linear discriminant in each of its decision nodes and processed squared features to realize quadratic decision boundaries in the original feature space. The penalized discriminant was based on a multiclass sparse discriminant analysis and the penalization was based on a group Lasso regularizer which was an intermediate between the Lasso and the ridge regularizer. The classification probabilities estimated by this forest and the features learned by its decision nodes could be used standalone or foster graph-based classifiers.
The disruption index is biased by citation inflation
Venue: arXiv cs.DL 2023
Year: 2023

A recent analysis of scientific publication and patent citation networks by Park et al. (Nature, 2023) suggests that publications and patents are becoming less disruptive over time. Here we show that the reported decrease in disruptiveness is an artifact of systematic shifts in the structure of citation networks unrelated to innovation system capacity. Instead, the decline is attributable to 'citation inflation', an unavoidable characteristic of real citation networks that manifests as a systematic time-dependent bias and renders cross-temporal analysis challenging. One driver of citation inflation is the ever-increasing lengths of reference lists over time, which in turn increases the density of links in citation networks, and causes the disruption index to converge to 0. A second driver is attributable to shifts in the construction of reference lists, which is increasingly impacted by self-citations that increase in the rate of triadic closure in citation networks, and thus confounds efforts to measure disruption, which is itself a measure of triadic closure. Combined, these two systematic shifts render the disruption index temporally biased, and unsuitable for cross-temporal analysis. The impact of this systematic bias further stymies efforts to correlate disruption to other measures that are also time-dependent, such as team size and citation counts. In order to demonstrate this fundamental measurement problem, we present three complementary lines of critique (deductive, empirical and computational modeling), and also make available an ensemble of synthetic citation networks that can be used to test alternative citation-based indices for systematic bias.
Over-the-Air Federated Learning In Broadband Communication
Venue: arXiv cs.LG 2023
Year: 2023

Federated learning (FL) is a privacy-preserving distributed machine learning paradigm that operates at the wireless edge. It enables clients to collaborate on model training while keeping their data private from adversaries and the central server. However, current FL approaches have limitations. Some rely on secure multiparty computation, which can be vulnerable to inference attacks. Others employ differential privacy, but this may lead to decreased test accuracy when dealing with a large number of parties contributing small amounts of data. To address these issues, this paper proposes a novel approach that integrates federated learning seamlessly into the inner workings of MIMO (Multiple-Input Multiple-Output) systems.
BEDRF: Bidirectional Edge Diffraction Response Function for Interactive Sound Propagation
Venue: arXiv cs.SD 2023
Year: 2023

We introduce bidirectional edge diffraction response function (BEDRF), a new approach to model wave diffraction around edges with path tracing. The diffraction part of the wave is expressed as an integration on path space, and the wave-edge interaction is expressed using only the localized information around points on the edge similar to a bidirectional scattering distribution function (BSDF) for visual rendering. For an infinite single wedge, our model generates the same result as the analytic solution. Our approach can be easily integrated into interactive geometric sound propagation algorithms that use path tracing to compute specular and diffuse reflections. Our resulting propagation algorithm can approximate complex wave propagation phenomena involving high-order diffraction, and is able to handle dynamic, deformable objects and moving sources and listeners. We highlight the performance of our approach in different scenarios to generate smooth auralization.