OpenRaiser/Intern-Atlas huggingface dataset:
Transferring Pre-trained Multimodal Representations with Cross-modal Similarity Matching
Venue: arXiv cs.LG 2023
Year: 2023

Despite surprising performance on zero-shot transfer, pre-training a large-scale multimodal model is often prohibitive as it requires a huge amount of data and computing resources. In this paper, we propose a method (BeamCLIP) that can effectively transfer the representations of a large pre-trained multimodal model (CLIP-ViT) into a small target model (e.g., ResNet-18). For unsupervised transfer, we introduce cross-modal similarity matching (CSM) that enables a student model to learn the representations of a teacher model by matching the relative similarity distribution across text prompt embeddings. To better encode the text prompts, we design context-based prompt augmentation (CPA) that can alleviate the lexical ambiguity of input text prompts. Our experiments show that unsupervised representation transfer of a pre-trained vision-language model enables a small ResNet-18 to achieve a better ImageNet-1K top-1 linear probe accuracy (66.2%) than vision-only self-supervised learning (SSL) methods (e.g., SimCLR: 51.8%, SwAV: 63.7%), while closing the gap with supervised learning (69.8%).
Patterns of Social Vulnerability -- An Interactive Dashboard to Explore Risks to Public Health on the US County Level
Venue: arXiv cs.HC 2023
Year: 2023

Social vulnerability is the susceptibility of a community to be adversely impacted by natural hazards and public health emergencies, such as drought, earthquakes, flooding, virus outbreaks, and the like. Climate change is at the root of many recent natural hazards while the COVID-19 pandemic is still an active threat. Social vulnerability also refers to resilience, or the ability to recover from such adverse events. To gauge the many aspects of social vulnerability the US Center of Disease Control (CDC) has subdivided social vulnerabilities into distinct themes, such as socioeconomic status, household composition, and others. Knowing a community's social vulnerabilities can help policymakers and responders to recognize risks to community health, prepare for possible hazards, or recover from disasters. In this paper we study social vulnerabilities on the US county level and present research that suggests that there are certain combinations, or patterns, of social vulnerability indicators into which US counties can be grouped. We then present an interactive dashboard that allows analysts to explore these patterns in various ways. We demonstrate our methodology using COVID-19 death rate as the hazard and show that the patterns we identified have high predictive capabilities of the pandemic's local impact.
Human Following Based on Visual Perception in the Context of Warehouse Logistics
Venue: arXiv cs.RO 2023
Year: 2023

Under the background of 5G, Internet, artificial intelligence technol,ogy and robot technology, warehousing, and logistics robot technology has developed rapidly, and products have been widely used. A practical application is to help warehouse personnel pick up or deliver heavy goods at dispersed locations based on dynamic routes. However, programs that can only accept instructions or pre-set by the system do not have more flexibility, but existing human auto-following techniques either cannot accurately identify specific targets or require a combination of lasers and cameras that are cumbersome and do not accomplish obstacle avoidance well. This paper proposed an algorithm that combines DeepSort and a width-based tracking module to track targets and use artificial potential field local path planning to avoid obstacles. The evaluation is performed in a self-designed flat bounded test field and simulated in ROS. Our method achieves the SOTA results on following and successfully reaching the end-point without hitting obstacles.
Neural network models
Venue: arXiv cs.NE 2023
Year: 2023

This work presents the current collection of mathematical models related to neural networks and proposes a new family of such with extended structure and dynamics in order to attain a selection of cognitive capabilities. It starts by providing a basic background to the morphology and physiology of the biological and the foundations and advances of the artificial neural networks. The first part then continues with a survey of all current mathematical models and some of their derived properties. In the second part, a new family of models is formulated, compared with the rest, and developed analytically and numerically. Finally, important additional aspects and any limitations to deal with in the future are discussed.
Fair Multi-Exit Framework for Facial Attribute Classification
Venue: arXiv cs.CV 2023
Year: 2023

Fairness has become increasingly pivotal in facial recognition. Without bias mitigation, deploying unfair AI would harm the interest of the underprivileged population. In this paper, we observe that though the higher accuracy that features from the deeper layer of a neural networks generally offer, fairness conditions deteriorate as we extract features from deeper layers. This phenomenon motivates us to extend the concept of multi-exit framework. Unlike existing works mainly focusing on accuracy, our multi-exit framework is fairness-oriented, where the internal classifiers are trained to be more accurate and fairer. During inference, any instance with high confidence from an internal classifier is allowed to exit early. Moreover, our framework can be applied to most existing fairness-aware frameworks. Experiment results show that the proposed framework can largely improve the fairness condition over the state-of-the-art in CelebA and UTK Face datasets.
Cross Far- and Near-field Wireless Communications in Terahertz Ultra-large Antenna Array Systems
Venue: arXiv cs.IT 2023
Year: 2023

Terahertz (THz) band owning the abundant multi-ten-GHz bandwidth is capable to support Terabit-per-second wireless communications, which is a pillar technology for 6G and beyond systems. With sub-millimeter-long antennas, ultra-massive (UM) MIMO and intelligent surface (IS) systems with thousands of array elements are exploited to effectively combat the distance limitation and blockage problems, which compose a promising THz ultra-large antenna array (ULAA) system. As a combined effect of wavelength and array aperture, the resulting coverage of THz systems ranges from near-field to far-field, leading to a new paradigm of cross-field communications. Although channel models, communications theories, and networking strategies have been studied for far-field and near-field separately, the unified design of cross-field communications that achieve high spectral efficiency and low complexity is still missing. In this article, the challenges and features of THz ULAA cross-field communications are investigated. Furthermore, cross-field solutions in three perspectives are presented, including a hybrid spherical- and planar-wave channel model, cross-field channel estimation, and widely-spaced multi-subarray hybrid beamforming, where a subarray as a basic unit in THz ULAA systems is exploited. The approximation error of channel modeling accuracy, spectral efficiency, and estimation error of these designs are numerically evaluated. Finally, as a roadmap of THz ULAA cross-field communications, multiple open problems and potential research directions are elaborated.
XDQN: Inherently Interpretable DQN through Mimicking
Venue: arXiv cs.LG 2023
Year: 2023

Although deep reinforcement learning (DRL) methods have been successfully applied in challenging tasks, their application in real-world operational settings is challenged by methods' limited ability to provide explanations. Among the paradigms for explainability in DRL is the interpretable box design paradigm, where interpretable models substitute inner constituent models of the DRL method, thus making the DRL method "inherently" interpretable. In this paper we explore this paradigm and we propose XDQN, an explainable variation of DQN, which uses an interpretable policy model trained through mimicking. XDQN is challenged in a complex, real-world operational multi-agent problem, where agents are independent learners solving congestion problems. Specifically, XDQN is evaluated in three MARL scenarios, pertaining to the demand-capacity balancing problem of air traffic management. XDQN achieves performance similar to that of DQN, while its abilities to provide global models' interpretations and interpretations of local decisions are demonstrated.
A Survey on Transformers in Reinforcement Learning
Venue: arXiv cs.LG 2023
Year: 2023

Transformer has been considered the dominating neural architecture in NLP and CV, mostly under supervised settings. Recently, a similar surge of using Transformers has appeared in the domain of reinforcement learning (RL), but it is faced with unique design choices and challenges brought by the nature of RL. However, the evolution of Transformers in RL has not yet been well unraveled. In this paper, we seek to systematically review motivations and progress on using Transformers in RL, provide a taxonomy on existing works, discuss each sub-field, and summarize future prospects.
AI Maintenance: A Robustness Perspective
Venue: arXiv cs.LG 2023
Year: 2023

With the advancements in machine learning (ML) methods and compute resources, artificial intelligence (AI) empowered systems are becoming a prevailing technology. However, current AI technology such as deep learning is not flawless. The significantly increased model complexity and data scale incur intensified challenges when lacking trustworthiness and transparency, which could create new risks and negative impacts. In this paper, we carve out AI maintenance from the robustness perspective. We start by introducing some highlighted robustness challenges in the AI lifecycle and motivating AI maintenance by making analogies to car maintenance. We then propose an AI model inspection framework to detect and mitigate robustness risks. We also draw inspiration from vehicle autonomy to define the levels of AI robustness automation. Our proposal for AI maintenance facilitates robustness assessment, status tracking, risk scanning, model hardening, and regulation throughout the AI lifecycle, which is an essential milestone toward building sustainable and trustworthy AI ecosystems.
Deepfake CAPTCHA: A Method for Preventing Fake Calls
Venue: arXiv cs.CR 2023
Year: 2023

Deep learning technology has made it possible to generate realistic content of specific individuals. These `deepfakes' can now be generated in real-time which enables attackers to impersonate people over audio and video calls. Moreover, some methods only need a few images or seconds of audio to steal an identity. Existing defenses perform passive analysis to detect fake content. However, with the rapid progress of deepfake quality, this may be a losing game. In this paper, we propose D-CAPTCHA: an active defense against real-time deepfakes. The approach is to force the adversary into the spotlight by challenging the deepfake model to generate content which exceeds its capabilities. By doing so, passive detection becomes easier since the content will be distorted. In contrast to existing CAPTCHAs, we challenge the AI's ability to create content as opposed to its ability to classify content. In this work we focus on real-time audio deepfakes and present preliminary results on video. In our evaluation we found that D-CAPTCHA outperforms state-of-the-art audio deepfake detectors with an accuracy of 91-100% depending on the challenge (compared to 71% without challenges). We also performed a study on 41 volunteers to understand how threatening current real-time deepfake attacks are. We found that the majority of the volunteers could not tell the difference between real and fake audio.
Massive MIMO and NOMA Bits-per-Antenna Efficiency under Power Allocation Policies
Venue: arXiv cs.IT 2023
Year: 2023

A comparative resource allocation analysis in terms of received bits-per-antenna spectral efficiency (SE) and energy efficiency (EE) in downlink (DL) single-cell massive multiple-input multiple-output (mMIMO) and non-orthogonal multiple access (NOMA) systems considering a BS equipped with many ($M$) antennas, while $K$ devices {operate} with a single-antenna, and {the loading of devices} $ρ= \frac{K}{M}$ ranging in $0<ρ\leq 2$ is carried out under three different \ac{PA} strategies: the inverse of the channel power allocation (PICPA), a modified water-filling ($Δ$-WF) allocation method, and the equal power allocation (EPA) reference method. Since the two devices per cluster are overlapped in the power domain in the NOMA system, the channel matrix requires transformation to perform the zero-forcing (ZF) precoding adopted in mMIMO. Hence, NOMA operating under many antennas can favor a group of devices with higher array gain, overcoming the mMIMO and operating conveniently in the higher loading range $0.6<ρ<2.0$. In such a scenario, a more realistic and helpful metric consists of {evaluating} the area under SE and EE curves, by measuring the bit-per-antenna and bit-per-antenna-per-watt efficiency, respectively. Our numerical results confirm a superiority of NOMA w.r.t. mMIMO of an order of 3x for the SE-area and 2x for the EE-area metric.
A Multi-Site Accelerator-Rich Processing Fabric for Scalable Brain-Computer Interfacing
Venue: arXiv cs.DC 2023
Year: 2023

Hull is an accelerator-rich distributed implantable Brain-Computer Interface (BCI) that reads biological neurons at data rates that are 2-3 orders of magnitude higher than the prior state of art, while supporting many neuroscientific applications. Prior approaches have restricted brain interfacing to tens of megabits per second in order to meet two constraints necessary for effective operation and safe long-term implantation -- power dissipation under tens of milliwatts and response latencies in the tens of milliseconds. Hull also adheres to these constraints, but is able to interface with the brain at much higher data rates, thereby enabling, for the first time, BCI-driven research on and clinical treatment of brain-wide behaviors and diseases that require reading and stimulating many brain locations. Central to Hull's power efficiency is its realization as a distributed system of BCI nodes with accelerator-rich compute. Hull balances modular system layering with aggressive cross-layer hardware-software co-design to integrate compute, networking, and storage. The result is a lesson in designing networked distributed systems with hardware accelerators from the ground up.
Automatic Generation of German Drama Texts Using Fine Tuned GPT-2 Models
Venue: arXiv cs.CL 2023
Year: 2023

This study is devoted to the automatic generation of German drama texts. We suggest an approach consisting of two key steps: fine-tuning a GPT-2 model (the outline model) to generate outlines of scenes based on keywords and fine-tuning a second model (the generation model) to generate scenes from the scene outline. The input for the neural model comprises two datasets: the German Drama Corpus (GerDraCor) and German Text Archive (Deutsches Textarchiv or DTA). In order to estimate the effectiveness of the proposed method, our models are compared with baseline GPT-2 models. Our models perform well according to automatic quantitative evaluation, but, conversely, manual qualitative analysis reveals a poor quality of generated texts. This may be due to the quality of the dataset or training inputs.
Applying Automated Machine Translation to Educational Video Courses
Venue: arXiv cs.CL 2023
Year: 2023

We studied the capability of automated machine translation in the online video education space by automatically translating Khan Academy videos with state-of-the-art translation models and applying text-to-speech synthesis and audio/video synchronization to build engaging videos in target languages. We also analyzed and established two reliable translation confidence estimators based on round-trip translations in order to efficiently manage translation quality and reduce human translation effort. Finally, we developed a deployable system to deliver translated videos to end users and collect user corrections for iterative improvement.
Finding Lookalike Customers for E-Commerce Marketing
Venue: arXiv cs.LG 2023
Year: 2023

Customer-centric marketing campaigns generate a large portion of e-commerce website traffic for Walmart. As the scale of customer data grows larger, expanding the marketing audience to reach more customers is becoming more critical for e-commerce companies to drive business growth and bring more value to customers. In this paper, we present a scalable and efficient system to expand targeted audience of marketing campaigns, which can handle hundreds of millions of customers. We use a deep learning based embedding model to represent customers and an approximate nearest neighbor search method to quickly find lookalike customers of interest. The model can deal with various business interests by constructing interpretable and meaningful customer similarity metrics. We conduct extensive experiments to demonstrate the great performance of our system and customer embedding model.
Cursive Caption Text Detection in Videos
Venue: arXiv cs.CV 2023
Year: 2023

Textual content appearing in videos represents an interesting index for semantic retrieval of videos (from archives), generation of alerts (live streams) as well as high level applications like opinion mining and content summarization. One of the key components of such systems is the detection of textual content in video frames and the same makes the subject of our present study. This paper presents a robust technique for detection of textual content appearing in video frames. More specifically we target text in cursive script taking Urdu text as a case study. Detection of textual regions in video frames is carried out by fine-tuning object detectors based on deep convolutional neural networks for the specific case of text detection. Since it is common to have videos with caption text in multiple-scripts, cursive text is distinguished from Latin text using a script-identification module. Finally, detection and script identification are combined in a single end-to-end trainable system. Experiments on a comprehensive dataset of around 11,000 video frames report an F-measure of 0.91.
Line-Torus Intersection for Ray Tracing: Alternative Formulations
Venue: arXiv cs.GR 2023
Year: 2023

Intersection algorithms are very important in computation of geometrical problems. Algorithms for a line intersection with linear or quadratic surfaces are quite efficient. However, algorithms for a line intersection with other surfaces are more complex and time consuming. In this case the object is usually closed into a simple bounding volume to speed up the cases when the given line cannot intersect the given object. In this paper new formulations of the line-torus intersection problem are given and new specification of the bounding volume for a torus is given as well. The presented approach is based on an idea of a line intersection with an envelope of rotating sphere that forms a torus. Due to this approach new bounding volume can be formulated which is more effective as it enables to detect cases when the line passes the "hole" of a torus, too.
Safehaul: Risk-Averse Learning for Reliable mmWave Self-Backhauling in 6G Networks
Venue: arXiv cs.NI 2023
Year: 2023

Wireless backhauling at millimeter-wave frequencies (mmWave) in static scenarios is a well-established practice in cellular networks. However, highly directional and adaptive beamforming in today's mmWave systems have opened new possibilities for self-backhauling. Tapping into this potential, 3GPP has standardized Integrated Access and Backhaul (IAB) allowing the same base station serve both access and backhaul traffic. Although much more cost-effective and flexible, resource allocation and path selection in IAB mmWave networks is a formidable task. To date, prior works have addressed this challenge through a plethora of classic optimization and learning methods, generally optimizing a Key Performance Indicator (KPI) such as throughput, latency, and fairness, and little attention has been paid to the reliability of the KPI. We propose Safehaul, a risk-averse learning-based solution for IAB mmWave networks. In addition to optimizing average performance, Safehaul ensures reliability by minimizing the losses in the tail of the performance distribution. We develop a novel simulator and show via extensive simulations that Safehaul not only reduces the latency by up to 43.2% compared to the benchmarks but also exhibits significantly more reliable performance (e.g., 71.4% less variance in achieved latency).
Introducing Model Inversion Attacks on Automatic Speaker Recognition
Venue: arXiv cs.SD 2023
Year: 2023

Model inversion (MI) attacks allow to reconstruct average per-class representations of a machine learning (ML) model's training data. It has been shown that in scenarios where each class corresponds to a different individual, such as face classifiers, this represents a severe privacy risk. In this work, we explore a new application for MI: the extraction of speakers' voices from a speaker recognition system. We present an approach to (1) reconstruct audio samples from a trained ML model and (2) extract intermediate voice feature representations which provide valuable insights into the speakers' biometrics. Therefore, we propose an extension of MI attacks which we call sliding model inversion. Our sliding MI extends standard MI by iteratively inverting overlapping chunks of the audio samples and thereby leveraging the sequential properties of audio data for enhanced inversion performance. We show that one can use the inverted audio data to generate spoofed audio samples to impersonate a speaker, and execute voice-protected commands for highly secured systems on their behalf. To the best of our knowledge, our work is the first one extending MI attacks to audio data, and our results highlight the security risks resulting from the extraction of the biometric data in that setup.
Online Fake Review Detection Using Supervised Machine Learning And BERT Model
Venue: arXiv cs.CL 2023
Year: 2023

Online shopping stores have grown steadily over the past few years. Due to the massive growth of these businesses, the detection of fake reviews has attracted attention. Fake reviews are seriously trying to mislead customers and thereby undermine the honesty and authenticity of online shopping environments. So far, various fake review classifiers have been proposed that take into account the actual content of the review. To improve the accuracies of existing fake review classification or detection approaches, we propose to use BERT (Bidirectional Encoder Representation from Transformers) model to extract word embeddings from texts (i.e. reviews). Word embeddings are obtained in various basic methods such as SVM (Support vector machine), Random Forests, Naive Bayes, and others. The confusion matrix method was also taken into account to evaluate and graphically represent the results. The results indicate that the SVM classifiers outperform the others in terms of accuracy and f1-score with an accuracy of 87.81%, which is 7.6% higher than the classifier used in the previous study [5].