OpenRaiser/Intern-Atlas huggingface dataset:
Focus on the Challenges: Analysis of a User-friendly Data Search Approach with CLIP in the Automotive Domain
Venue: arXiv cs.RO 2023
Year: 2023

Handling large amounts of data has become a key for developing automated driving systems. Especially for developing highly automated driving functions, working with images has become increasingly challenging due to the sheer size of the required data. Such data has to satisfy different requirements to be usable in machine learning-based approaches. Thus, engineers need to fully understand their large image data sets for the development and test of machine learning algorithms. However, current approaches lack automatability, are not generic and are limited in their expressiveness. Hence, this paper aims to analyze a state-of-the-art text and image embedding neural network and guides through the application in the automotive domain. This approach enables the search for similar images and the search based on a human understandable text-based description. Our experiments show the automatability and generalizability of our proposed method for handling large data sets in the automotive domain.
Learning Representative Trajectories of Dynamical Systems via Domain-Adaptive Imitation
Venue: arXiv cs.LG 2023
Year: 2023

Domain-adaptive trajectory imitation is a skill that some predators learn for survival, by mapping dynamic information from one domain (their speed and steering direction) to a different domain (current position of the moving prey). An intelligent agent with this skill could be exploited for a diversity of tasks, including the recognition of abnormal motion in traffic once it has learned to imitate representative trajectories. Towards this direction, we propose DATI, a deep reinforcement learning agent designed for domain-adaptive trajectory imitation using a cycle-consistent generative adversarial method. Our experiments on a variety of synthetic families of reference trajectories show that DATI outperforms baseline methods for imitation learning and optimal control in this setting, keeping the same per-task hyperparameters. Its generalization to a real-world scenario is shown through the discovery of abnormal motion patterns in maritime traffic, opening the door for the use of deep reinforcement learning methods for spatially-unconstrained trajectory data mining.
Salience-based stakeholder selection to maintain stakeholder coverage in solving the next release problem
Venue: arXiv cs.SE 2023
Year: 2023

Stakeholders quantification plays a basic role in selecting the appropriate requirements because their judgement is a major criteria since not all of them have the same importance. Original proposals quantified stakeholders assigning them a weight. Nonetheless, actual projects manage a numerous stakeholders community hindering the inclusion of all their weights. This work proposes grouping strategies as means to reduce the number of stakeholders to manage in requirements selection keeping a proper coverage (i.e. howthe selection fulfils stakeholder demands). Our approach is based on stakeholders' salience, defined in terms of power, legitimacy and urgency attributes. Diverse strategies are applied selecting important stakeholders groups in a specific project. We use k-means}, k-medoids and hierarchical clustering, after deciding the number of cluster (4 and 3) based on validation indices. Either for all the stakeholders and each important group several requirements selection optimization problems have been solved. Tests find no significant differences for coverage when important stakeholders are filtered using clustering, regardless of the technique and number of groups, with a reduction between 66.32% to 87.75% in the number of stakeholders being considered. Applying clustering methods on data obtained from a real-world project is useful to identify the group of important stakeholders. The number of groups suggested matches the stakeholders theory and the coverage values in the requirements selection is kept.
A Meta-heuristic Approach to Estimate and Explain Classifier Uncertainty
Venue: arXiv cs.LG 2023
Year: 2023

Trust is a crucial factor affecting the adoption of machine learning (ML) models. Qualitative studies have revealed that end-users, particularly in the medical domain, need models that can express their uncertainty in decision-making allowing users to know when to ignore the model's recommendations. However, existing approaches for quantifying decision-making uncertainty are not model-agnostic, or they rely on complex statistical derivations that are not easily understood by laypersons or end-users, making them less useful for explaining the model's decision-making process. This work proposes a set of class-independent meta-heuristics that can characterize the complexity of an instance in terms of factors are mutually relevant to both human and ML decision-making. The measures are integrated into a meta-learning framework that estimates the risk of misclassification. The proposed framework outperformed predicted probabilities in identifying instances at risk of being misclassified. The proposed measures and framework hold promise for improving model development for more complex instances, as well as providing a new means of model abstention and explanation.
Feature-compatible Progressive Learning for Video Copy Detection
Venue: arXiv cs.CV 2023
Year: 2023

Video Copy Detection (VCD) has been developed to identify instances of unauthorized or duplicated video content. This paper presents our second place solutions to the Meta AI Video Similarity Challenge (VSC22), CVPR 2023. In order to compete in this challenge, we propose Feature-Compatible Progressive Learning (FCPL) for VCD. FCPL trains various models that produce mutually-compatible features, meaning that the features derived from multiple distinct models can be directly compared with one another. We find this mutual compatibility enables feature ensemble. By implementing progressive learning and utilizing labeled ground truth pairs, we effectively gradually enhance performance. Experimental results demonstrate the superiority of the proposed FCPL over other competitors. Our code is available at https://github.com/WangWenhao0716/VSC-DescriptorTrack-Submission and https://github.com/WangWenhao0716/VSC-MatchingTrack-Submission.
Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning
Venue: arXiv cs.CL 2023
Year: 2023

The end-to-end speech translation (E2E-ST) model has gradually become a mainstream paradigm due to its low latency and less error propagation. However, it is non-trivial to train such a model well due to the task complexity and data scarcity. The speech-and-text modality differences result in the E2E-ST model performance usually inferior to the corresponding machine translation (MT) model. Based on the above observation, existing methods often use sharingmechanisms to carry out implicit knowledge transfer by imposing various constraints. However, the final model often performs worse on the MT task than the MT model trained alone, which means that the knowledge transfer ability of this method is also limited. To deal with these problems, we propose the FCCL (Fine- and Coarse- Granularity Contrastive Learning) approach for E2E-ST, which makes explicit knowledge transfer through cross-modal multi-grained contrastive learning. A key ingredient of our approach is applying contrastive learning at both sentence- and frame-level to give the comprehensive guide for extracting speech representations containing rich semantic information.In addition, we adopt a simple whitening method to alleviate the representation degeneration in the MT model, which adversely affects contrast learning. Experiments on the MuST-C benchmark show that our proposed approach significantly outperforms the state-of-the-art E2E-ST baselines on all eight language pairs. Further analysis indicates that FCCL can free up its capacity from learning grammatical structure information and force more layers to learn semantic information.
Towards a Benchmark for Scientific Understanding in Humans and Machines
Venue: arXiv cs.AI 2023
Year: 2023

Scientific understanding is a fundamental goal of science, allowing us to explain the world. There is currently no good way to measure the scientific understanding of agents, whether these be humans or Artificial Intelligence systems. Without a clear benchmark, it is challenging to evaluate and compare different levels of and approaches to scientific understanding. In this Roadmap, we propose a framework to create a benchmark for scientific understanding, utilizing tools from philosophy of science. We adopt a behavioral notion according to which genuine understanding should be recognized as an ability to perform certain tasks. We extend this notion by considering a set of questions that can gauge different levels of scientific understanding, covering information retrieval, the capability to arrange information to produce an explanation, and the ability to infer how things would be different under different circumstances. The Scientific Understanding Benchmark (SUB), which is formed by a set of these tests, allows for the evaluation and comparison of different approaches. Benchmarking plays a crucial role in establishing trust, ensuring quality control, and providing a basis for performance evaluation. By aligning machine and human scientific understanding we can improve their utility, ultimately advancing scientific understanding and helping to discover new insights within machines.
Attention Scheme Inspired Softmax Regression
Venue: arXiv cs.LG 2023
Year: 2023

Large language models (LLMs) have made transformed changes for human society. One of the key computation in LLMs is the softmax unit. This operation is important in LLMs because it allows the model to generate a distribution over possible next words or phrases, given a sequence of input words. This distribution is then used to select the most likely next word or phrase, based on the probabilities assigned by the model. The softmax unit plays a crucial role in training LLMs, as it allows the model to learn from the data by adjusting the weights and biases of the neural network. In the area of convex optimization such as using central path method to solve linear programming. The softmax function has been used a crucial tool for controlling the progress and stability of potential function [Cohen, Lee and Song STOC 2019, Brand SODA 2020]. In this work, inspired the softmax unit, we define a softmax regression problem. Formally speaking, given a matrix $A \in \mathbb{R}^{n \times d}$ and a vector $b \in \mathbb{R}^n$, the goal is to use greedy type algorithm to solve \begin{align*} \min_{x} \| \langle \exp(Ax), {\bf 1}_n \rangle^{-1} \exp(Ax) - b \|_2^2. \end{align*} In certain sense, our provable convergence result provides theoretical support for why we can use greedy algorithm to train softmax function in practice.
Fully Autonomous Programming with Large Language Models
Venue: arXiv cs.SE 2023
Year: 2023

Current approaches to program synthesis with Large Language Models (LLMs) exhibit a "near miss syndrome": they tend to generate programs that semantically resemble the correct answer (as measured by text similarity metrics or human evaluation), but achieve a low or even zero accuracy as measured by unit tests due to small imperfections, such as the wrong input or output format. This calls for an approach known as Synthesize, Execute, Debug (SED), whereby a draft of the solution is generated first, followed by a program repair phase addressing the failed tests. To effectively apply this approach to instruction-driven LLMs, one needs to determine which prompts perform best as instructions for LLMs, as well as strike a balance between repairing unsuccessful programs and replacing them with newly generated ones. We explore these trade-offs empirically, comparing replace-focused, repair-focused, and hybrid debug strategies, as well as different template-based and model-based prompt-generation techniques. We use OpenAI Codex as the LLM and Program Synthesis Benchmark 2 as a database of problem descriptions and tests for evaluation. The resulting framework outperforms both conventional usage of Codex without the repair phase and traditional genetic programming approaches.
GPT-NER: Named Entity Recognition via Large Language Models
Venue: arXiv cs.CL 2023
Year: 2023

Despite the fact that large-scale Language Models (LLM) have achieved SOTA performances on a variety of NLP tasks, its performance on NER is still significantly below supervised baselines. This is due to the gap between the two tasks the NER and LLMs: the former is a sequence labeling task in nature while the latter is a text-generation model. In this paper, we propose GPT-NER to resolve this issue. GPT-NER bridges the gap by transforming the sequence labeling task to a generation task that can be easily adapted by LLMs e.g., the task of finding location entities in the input text "Columbus is a city" is transformed to generate the text sequence "@@Columbus## is a city", where special tokens @@## marks the entity to extract. To efficiently address the "hallucination" issue of LLMs, where LLMs have a strong inclination to over-confidently label NULL inputs as entities, we propose a self-verification strategy by prompting LLMs to ask itself whether the extracted entities belong to a labeled entity tag. We conduct experiments on five widely adopted NER datasets, and GPT-NER achieves comparable performances to fully supervised baselines, which is the first time as far as we are concerned. More importantly, we find that GPT-NER exhibits a greater ability in the low-resource and few-shot setups, when the amount of training data is extremely scarce, GPT-NER performs significantly better than supervised models. This demonstrates the capabilities of GPT-NER in real-world NER applications where the number of labeled examples is limited.
A primer on getting neologisms from foreign languages to under-resourced languages
Venue: arXiv cs.CL 2023
Year: 2023

Mainly due to lack of support, most under-resourced languages have a reduced lexicon in most realms and domains of increasing importance, then their speakers need to significantly augment it. Although neologisms should arise from the languages themselves, external sources are widely accepted. However, we dispute the "common sense" of using the imposed official languages, which are highly probably a legacy of colonialism, as the only source, and we propose to introduce neologisms from any language as long as these neologisms "sound like" native words of the target languages.
Study of Robust Adaptive Beamforming with Covariance Matrix Reconstruction Based on Power Spectral Estimation and Uncertainty Region
Venue: arXiv cs.NI 2023
Year: 2023

In this work, a simple and effective robust adaptive beamforming technique is proposed for uniform linear arrays, which is based on the power spectral estimation and uncertainty region (PSEUR) of the interference plus noise (IPN) components. In particular, two algorithms are presented to find the angular sector of interference in every snapshot based on the adopted spatial uncertainty region of the interference direction. Moreover, a power spectrum is introduced based on the estimation of the power of interference and noise components, which allows the development of a robust approach to IPN covariance matrix reconstruction. The proposed method has two main advantages. First, an angular region that contains the interference direction is updated based on the statistics of the array data. Secondly, the proposed IPN-PSEUR method avoids estimating the power spectrum of the whole range of possible directions of the interference sector. Simulation results show that the performance of the proposed IPN-PSEUR beamformer is almost always close to the optimal value across a wide range of signal-to-noise ratios.
New Closed-Form ASER Expressions for Dual-Hop Mixed THz-RF Cooperative Relay Networks
Venue: arXiv cs.IT 2023
Year: 2023

In this paper, we consider a dual-hop mixed THz-RF system model for backhaul-fronthaul applications where the link between source and destination is established only through the relay node in which decode-and-forward relaying protocol is used. The THz link suffers from the joint impact of antenna misalignment and stochastic characteristics of wireless channels, including the effect of environmental conditions such as pressure, humidity, and temperature. The envelope of THz link in the first hop follows a generalized $α-μ$ distribution, and for the RF end, the Nakagami-$m$ distribution is considered. In this context, we obtain new closed-form expressions of the cumulative density function and the moment-generating function of the end-to-end signal-to-noise ratio. Further, we derive the average symbol error rate expressions for coherent rectangular quadrature amplitude modulation (RQAM) and coherent hexagonal QAM (HQAM), as well as the non-coherent modulation scheme. The asymptotic behavior is also discussed to examine the system's diversity. Furthermore, the impact of several parameters, such as fading coefficients of individual links and antenna misalignment, as well as the distance between nodes, are also highlighted in the system's performance. Moreover, Monte Carlo simulations are used to validate the presented analytical framework. Finally, the presented numerical insights aid in the extraction of practical design principles.
Segment Anything Model for Medical Image Analysis: an Experimental Study
Venue: arXiv cs.CV 2023
Year: 2023

Training segmentation models for medical images continues to be challenging due to the limited availability of data annotations. Segment Anything Model (SAM) is a foundation model that is intended to segment user-defined objects of interest in an interactive manner. While the performance on natural images is impressive, medical image domains pose their own set of challenges. Here, we perform an extensive evaluation of SAM's ability to segment medical images on a collection of 19 medical imaging datasets from various modalities and anatomies. We report the following findings: (1) SAM's performance based on single prompts highly varies depending on the dataset and the task, from IoU=0.1135 for spine MRI to IoU=0.8650 for hip X-ray. (2) Segmentation performance appears to be better for well-circumscribed objects with prompts with less ambiguity and poorer in various other scenarios such as the segmentation of brain tumors. (3) SAM performs notably better with box prompts than with point prompts. (4) SAM outperforms similar methods RITM, SimpleClick, and FocalClick in almost all single-point prompt settings. (5) When multiple-point prompts are provided iteratively, SAM's performance generally improves only slightly while other methods' performance improves to the level that surpasses SAM's point-based performance. We also provide several illustrations for SAM's performance on all tested datasets, iterative segmentation, and SAM's behavior given prompt ambiguity. We conclude that SAM shows impressive zero-shot segmentation performance for certain medical imaging datasets, but moderate to poor performance for others. SAM has the potential to make a significant impact in automated medical image segmentation in medical imaging, but appropriate care needs to be applied when using it.
Exploring the visualisation of hierarchical cybersecurity data within the Metaverse
Venue: arXiv cs.HC 2023
Year: 2023

A prototype Metaverse experience was created in which users could explore hierarchical cybersecurity data. A small group of participants were surveyed on their attitudes to the Metaverse. They then completed a short series of tasks in the environment. Questions were asked to assess if they were suffering from Cybersickness. After completing further tasks, their attitudes were surveyed regarding future uses of the metaverse in the organisation. A second cohort of participants attended an online seminar. They completed a survey about their attitudes to the Metaverse. They then watched a short video of the Metaverse experience. Afterwards, they answered questions related to their attitudes towards future uses of the metaverse in the organisation. The results of these questionnaires were assessed to see whether participants were receptive to the idea of working with data inside the Metaverse in the future.
VREd: A Virtual Reality-Based Classroom for Online Education Using Unity3D WebGL
Venue: arXiv cs.CY 2023
Year: 2023

Virtual reality is the way of the future. The use of virtual reality is expanding over time across all sectors, from the entertainment industry to the military and space. VREd is a similar concept where a virtual reality-based classroom is used for online education where the user will have better interaction and more control. Unity3D and WebGL software have been used for implementation. Students or learners accustomed to contemporary technologies may find the traditional educational system unappealing because of its flaws. Incorporating the latest technologies can increase the curiosity and learning abilities of students. The system architecture of VREd is similar to that of an actual classroom, allowing both students and teachers to access all of the course materials and interact with one another using only an internet connection. The environment and the background are also customizable. Therefore, all the users can comfortably use the system and feel at home. We can create an effective educational system that raises educational quality by utilizing virtual reality.
Detecting Worker Attention Lapses in Human-Robot Interaction: An Eye Tracking and Multimodal Sensing Study
Venue: arXiv cs.RO 2023
Year: 2023

The advent of industrial robotics and autonomous systems endow human-robot collaboration in a massive scale. However, current industrial robots are restrained in co-working with human in close proximity due to inability of interpreting human agents' attention. Human attention study is non-trivial since it involves multiple aspects of the mind: perception, memory, problem solving, and consciousness. Human attention lapses are particularly problematic and potentially catastrophic in industrial workplace, from assembling electronics to operating machines. Attention is indeed complex and cannot be easily measured with single-modality sensors. Eye state, head pose, posture, and manifold environment stimulus could all play a part in attention lapses. To this end, we propose a pipeline to annotate multimodal dataset of human attention tracking, including eye tracking, fixation detection, third-person surveillance camera, and sound. We produce a pilot dataset containing two fully annotated phone assembly sequences in a realistic manufacturing environment. We evaluate existing fatigue and drowsiness prediction methods for attention lapse detection. Experimental results show that human attention lapses in production scenarios are more subtle and imperceptible than well-studied fatigue and drowsiness.