OpenRaiser/Intern-Atlas huggingface dataset:
Unveiling and Mitigating Bias in Audio Visual Segmentation
Venue: arXiv cs.CV 2024
Year: 2024

Community researchers have developed a range of advanced audio-visual segmentation models aimed at improving the quality of sounding objects' masks. While masks created by these models may initially appear plausible, they occasionally exhibit anomalies with incorrect grounding logic. We attribute this to real-world inherent preferences and distributions as a simpler signal for learning than the complex audio-visual grounding, which leads to the disregard of important modality information. Generally, the anomalous phenomena are often complex and cannot be directly observed systematically. In this study, we made a pioneering effort with the proper synthetic data to categorize and analyze phenomena as two types "audio priming bias" and "visual prior" according to the source of anomalies. For audio priming bias, to enhance audio sensitivity to different intensities and semantics, a perception module specifically for audio perceives the latent semantic information and incorporates information into a limited set of queries, namely active queries. Moreover, the interaction mechanism related to such active queries in the transformer decoder is customized to adapt to the need for interaction regulating among audio semantics. For visual prior, multiple contrastive training strategies are explored to optimize the model by incorporating a biased branch, without even changing the structure of the model. During experiments, observation demonstrates the presence and the impact that has been produced by the biases of the existing model. Finally, through experimental evaluation of AVS benchmarks, we demonstrate the effectiveness of our methods in handling both types of biases, achieving competitive performance across all three subsets.
Distortion Recovery: A Two-Stage Method for Guitar Effect Removal
Venue: arXiv cs.SD 2024
Year: 2024

Removing audio effects from electric guitar recordings makes it easier for post-production and sound editing. An audio distortion recovery model not only improves the clarity of the guitar sounds but also opens up new opportunities for creative adjustments in mixing and mastering. While progress have been made in creating such models, previous efforts have largely focused on synthetic distortions that may be too simplistic to accurately capture the complexities seen in real-world recordings. In this paper, we tackle the task by using a dataset of guitar recordings rendered with commercial-grade audio effect VST plugins. Moreover, we introduce a novel two-stage methodology for audio distortion recovery. The idea is to firstly process the audio signal in the Mel-spectrogram domain in the first stage, and then use a neural vocoder to generate the pristine original guitar sound from the processed Mel-spectrogram in the second stage. We report a set of experiments demonstrating the effectiveness of our approach over existing methods, through both subjective and objective evaluation metrics.
Deformable Convolution Based Road Scene Semantic Segmentation of Fisheye Images in Autonomous Driving
Venue: arXiv cs.CV 2024
Year: 2024

This study investigates the effectiveness of modern Deformable Convolutional Neural Networks (DCNNs) for semantic segmentation tasks, particularly in autonomous driving scenarios with fisheye images. These images, providing a wide field of view, pose unique challenges for extracting spatial and geometric information due to dynamic changes in object attributes. Our experiments focus on segmenting the WoodScape fisheye image dataset into ten distinct classes, assessing the Deformable Networks' ability to capture intricate spatial relationships and improve segmentation accuracy. Additionally, we explore different loss functions to address class imbalance issues and compare the performance of conventional CNN architectures with Deformable Convolution-based CNNs, including Vanilla U-Net and Residual U-Net architectures. The significant improvement in mIoU score resulting from integrating Deformable CNNs demonstrates their effectiveness in handling the geometric distortions present in fisheye imagery, exceeding the performance of traditional CNN architectures. This underscores the significant role of Deformable convolution in enhancing semantic segmentation performance for fisheye imagery.
Goedel logics: Prenex fragments
Venue: arXiv cs.LO 2024
Year: 2024

In this paper, we provide a complete classification for the first-order Goedel logics concerning the property that the formulas admit logically equivalent prenex normal forms. We show that the only first-order Goedel logics that admit such prenex forms are those with finite truth value sets since they allow all quantifier-shift rules and the logic $G_\uparrow$ with only one accumulation point at 1 in the infinite truth value set. In all the other cases, there are generally no logically equivalent prenex normal forms. We will also see that $G_\uparrow$ is the intersection of all finite first-order Goedel logics. The second part of this paper investigates the existence of effective equivalence between the validity of a formula and the validity of some prenex normal form. The existence of such a normal form is obvious for finite valued Goedel logic and $G_\uparrow$. Goedel logics with an uncountable truth value set admit the prenex normal forms if and only if every surrounding of 0 is uncountable or 0 is an isolated point. Otherwise, uncountable Goedel logics are not recursively enumerable, however, the prenex fragment is always recursively enumerable. Therefore, there is no effective translation between the valid formula and the valid prenex normal form. However, the existence of effectively constructible validity equivalent prenex forms for the countable case is still up for debate.
Can Large Language Models Automatically Jailbreak GPT-4V?
Venue: arXiv cs.CL 2024
Year: 2024

GPT-4V has attracted considerable attention due to its extraordinary capacity for integrating and processing multimodal information. At the same time, its ability of face recognition raises new safety concerns of privacy leakage. Despite researchers' efforts in safety alignment through RLHF or preprocessing filters, vulnerabilities might still be exploited. In our study, we introduce AutoJailbreak, an innovative automatic jailbreak technique inspired by prompt optimization. We leverage Large Language Models (LLMs) for red-teaming to refine the jailbreak prompt and employ weak-to-strong in-context learning prompts to boost efficiency. Furthermore, we present an effective search method that incorporates early stopping to minimize optimization time and token expenditure. Our experiments demonstrate that AutoJailbreak significantly surpasses conventional methods, achieving an Attack Success Rate (ASR) exceeding 95.3\%. This research sheds light on strengthening GPT-4V security, underscoring the potential for LLMs to be exploited in compromising GPT-4V integrity.
A Survey of Text Style Transfer: Applications and Ethical Implications
Venue: arXiv cs.CL 2024
Year: 2024

Text style transfer (TST) is an important task in controllable text generation, which aims to control selected attributes of language use, such as politeness, formality, or sentiment, without altering the style-independent content of the text. The field has received considerable research attention in recent years and has already been covered in several reviews, but the focus has mostly been on the development of new algorithms and learning from different types of data (supervised, unsupervised, out-of-domain, etc.) and not so much on the application side. However, TST-related technologies are gradually reaching a production- and deployment-ready level, and therefore, the inclusion of the application perspective in TST research becomes crucial. Similarly, the often overlooked ethical considerations of TST technology have become a pressing issue. This paper presents a comprehensive review of TST applications that have been researched over the years, using both traditional linguistic approaches and more recent deep learning methods. We discuss current challenges, future research directions, and ethical implications of TST applications in text generation. By providing a holistic overview of the landscape of TST applications, we hope to stimulate further research and contribute to a better understanding of the potential as well as ethical considerations associated with TST.
PLM-Net: Perception Latency Mitigation Network for Vision-Based Lateral Control of Autonomous Vehicles
Venue: arXiv cs.RO 2024
Year: 2024

This study introduces the Perception Latency Mitigation Network (PLM-Net), a novel deep learning approach for addressing perception latency in vision-based Autonomous Vehicle (AV) lateral control systems. Perception latency is the delay between capturing the environment through vision sensors (e.g., cameras) and applying an action (e.g., steering). This issue is understudied in both classical and neural-network-based control methods. Reducing this latency with powerful GPUs and FPGAs is possible but impractical for automotive platforms. PLM-Net comprises the Base Model (BM) and the Timed Action Prediction Model (TAPM). BM represents the original Lane Keeping Assist (LKA) system, while TAPM predicts future actions for different latency values. By integrating these models, PLM-Net mitigates perception latency. The final output is determined through linear interpolation of BM and TAPM outputs based on real-time latency. This design addresses both constant and varying latency, improving driving trajectories and steering control. Experimental results validate the efficacy of PLM-Net across various latency conditions. Source code: https://github.com/AwsKhalil/oscar/tree/devel-plm-net.
What Matters in Range View 3D Object Detection
Venue: arXiv cs.CV 2024
Year: 2024

Lidar-based perception pipelines rely on 3D object detection models to interpret complex scenes. While multiple representations for lidar exist, the range-view is enticing since it losslessly encodes the entire lidar sensor output. In this work, we achieve state-of-the-art amongst range-view 3D object detection models without using multiple techniques proposed in past range-view literature. We explore range-view 3D object detection across two modern datasets with substantially different properties: Argoverse 2 and Waymo Open. Our investigation reveals key insights: (1) input feature dimensionality significantly influences the overall performance, (2) surprisingly, employing a classification loss grounded in 3D spatial proximity works as well or better compared to more elaborate IoU-based losses, and (3) addressing non-uniform lidar density via a straightforward range subsampling technique outperforms existing multi-resolution, range-conditioned networks. Our experiments reveal that techniques proposed in recent range-view literature are not needed to achieve state-of-the-art performance. Combining the above findings, we establish a new state-of-the-art model for range-view 3D object detection -- improving AP by 2.2% on the Waymo Open dataset while maintaining a runtime of 10 Hz. We establish the first range-view model on the Argoverse 2 dataset and outperform strong voxel-based baselines. All models are multi-class and open-source. Code is available at https://github.com/benjaminrwilson/range-view-3d-detection.
Convexification of the Quantum Network Utility Maximisation Problem
Venue: arXiv cs.NI 2024
Year: 2024

Network Utility Maximisation (NUM) addresses the problem of allocating resources fairly within a network and explores the ways to achieve optimal allocation in real-world networks. Although extensively studied in classical networks, NUM is an emerging area of research in the context of quantum networks. In this work, we consider the quantum network utility maximisation (QNUM) problem in a static setting, where a user's utility takes into account the assigned quantum quality (fidelity) via a generic entanglement measure as well as the corresponding rate of entanglement generation. Under certain assumptions, we demonstrate that the QNUM problem can be formulated as an optimisation problem with the rate allocation vector as the only decision variable. Using a change of variable technique known in the field of geometric programming, we then establish sufficient conditions under which this formulation can be reduced to a convex problem, a class of optimisation problems that can be solved efficiently and with certainty even in high dimensions. We further show that this technique preserves convexity, enabling us to formulate convex QNUM problems in networks where some routes have certain entanglement measures that do not readily admit convex formulation, while others do. This allows us to compute the optimal resource allocation in networks where heterogeneous applications run over different routes.
SE3ET: SE(3)-Equivariant Transformer for Low-Overlap Point Cloud Registration
Venue: arXiv cs.RO 2024
Year: 2024

Partial point cloud registration is a challenging problem in robotics, especially when the robot undergoes a large transformation, causing a significant initial pose error and a low overlap between measurements. This work proposes exploiting equivariant learning from 3D point clouds to improve registration robustness. We propose SE3ET, an SE(3)-equivariant registration framework that employs equivariant point convolution and equivariant transformer designs to learn expressive and robust geometric features. We tested the proposed registration method on indoor and outdoor benchmarks where the point clouds are under arbitrary transformations and low overlapping ratios. We also provide generalization tests and run-time performance.
Networks of Networks: Complexity Class Principles Applied to Compound AI Systems Design
Venue: arXiv cs.AI 2024
Year: 2024

As practitioners seek to surpass the current reliability and quality frontier of monolithic models, Compound AI Systems consisting of many language model inference calls are increasingly employed. In this work, we construct systems, which we call Networks of Networks (NoNs) organized around the distinction between generating a proposed answer and verifying its correctness, a fundamental concept in complexity theory that we show empirically extends to Language Models (LMs). We introduce a verifier-based judge NoN with K generators, an instantiation of "best-of-K" or "judge-based" compound AI systems. Through experiments on synthetic tasks such as prime factorization, and core benchmarks such as the MMLU, we demonstrate notable performance gains. For instance, in factoring products of two 3-digit primes, a simple NoN improves accuracy from 3.7\% to 36.6\%. On MMLU, a verifier-based judge construction with only 3 generators boosts accuracy over individual GPT-4-Turbo calls by 2.8\%. Our analysis reveals that these gains are most pronounced in domains where verification is notably easier than generation--a characterization which we believe subsumes many reasoning and procedural knowledge tasks, but doesn't often hold for factual and declarative knowledge-based settings. For mathematical and formal logic reasoning-based subjects of MMLU, we observe a 5-8\% or higher gain, whilst no gain on others such as geography and religion. We provide key takeaways for ML practitioners, including the importance of considering verification complexity, the impact of witness format on verifiability, and a simple test to determine the potential benefit of this NoN approach for a given problem distribution. This work aims to inform future research and practice in the design of compound AI systems.
Adapting Image-based RL Policies via Predicted Rewards
Venue: arXiv cs.RO 2024
Year: 2024

Image-based reinforcement learning (RL) faces significant challenges in generalization when the visual environment undergoes substantial changes between training and deployment. Under such circumstances, learned policies may not perform well leading to degraded results. Previous approaches to this problem have largely focused on broadening the training observation distribution, employing techniques like data augmentation and domain randomization. However, given the sequential nature of the RL decision-making problem, it is often the case that residual errors are propagated by the learned policy model and accumulate throughout the trajectory, resulting in highly degraded performance. In this paper, we leverage the observation that predicted rewards under domain shift, even though imperfect, can still be a useful signal to guide fine-tuning. We exploit this property to fine-tune a policy using reward prediction in the target domain. We have found that, even under significant domain shift, the predicted reward can still provide meaningful signal and fine-tuning substantially improves the original policy. Our approach, termed Predicted Reward Fine-tuning (PRFT), improves performance across diverse tasks in both simulated benchmarks and real-world experiments. More information is available at project web page: https://sites.google.com/view/prft.
Blockchain security for ransomware detection
Venue: arXiv cs.CR 2024
Year: 2024

Blockchain networks are critical for safeguarding digital transactions and assets, but they are increasingly targeted by ransomware attacks exploiting zero-day vulnerabilities. Traditional detection techniques struggle due to the complexity of these exploits and the lack of comprehensive datasets. The UGRansome dataset addresses this gap by offering detailed features for analysing ransomware and zero-day attacks, including timestamps, attack types, protocols, network flows, and financial impacts in bitcoins (BTC). This study uses the Lazy Predict library to automate machine learning (ML) on the UGRansome dataset. The study aims to enhance blockchain security through ransomware detection based on zero-day exploit recognition using the UGRansome dataset. Lazy Predict streamlines different ML model comparisons and identifies effective algorithms for threat detection. Key features such as timestamps, protocols, and financial data are used to predict anomalies as zero-day threats and to classify known signatures as ransomware. Results demonstrate that ML can significantly improve cybersecurity in blockchain environments. The DecisionTreeClassifier and ExtraTreeClassifier, with their high performance and low training times, are ideal candidates for deployment in real-time threat detection systems.
Cluster Model for parsimonious selection of variables and enhancing Students Employability Prediction
Venue: arXiv cs.CY 2024
Year: 2024

Educational Data Mining (EDM) is a promising field, where data mining is widely used for predicting students performance. One of the most prevalent and recent challenge that higher education faces today is making students skillfully employable. Institutions possess large volume of data; still they are unable to reveal knowledge and guide their students. Data in education is generally very large, multidimensional and unbalanced in nature. Process of extracting knowledge from such data has its own set of problems and is a very complicated task. In this paper, Engineering and MCA (Masters in Computer Applications) students data is collected from various universities and institutes pan India. The dataset is large, unbalanced and multidimensional in nature. A cluster based model is presented in this paper, which, when applied at preprocessing stage helps in parsimonious selection of variables and improves the performance of predictive algorithms. Hence, facilitate in better prediction of Students Employability.
Exploring Fusion Techniques in Multimodal AI-Based Recruitment: Insights from FairCVdb
Venue: arXiv cs.CY 2024
Year: 2024

Despite the large body of work on fairness-aware learning for individual modalities like tabular data, images, and text, less work has been done on multimodal data, which fuses various modalities for a comprehensive analysis. In this work, we investigate the fairness and bias implications of multimodal fusion techniques in the context of multimodal AI-based recruitment systems using the FairCVdb dataset. Our results show that early-fusion closely matches the ground truth for both demographics, achieving the lowest MAEs by integrating each modality's unique characteristics. In contrast, late-fusion leads to highly generalized mean scores and higher MAEs. Our findings emphasise the significant potential of early-fusion for accurate and fair applications, even in the presence of demographic biases, compared to late-fusion. Future research could explore alternative fusion strategies and incorporate modality-related fairness constraints to improve fairness. For code and additional insights, visit: https://github.com/Swati17293/Multimodal-AI-Based-Recruitment-FairCVdb
A Framework for AI assisted Musical Devices
Venue: arXiv cs.HC 2024
Year: 2024

In this paper we present a novel framework for the study and design of AI assisted musical devices (AIMEs). Initially, we present a taxonomy of these devices and illustrate it with a set of scenarios and personas. Later, we propose a generic architecture for the implementation of AIMEs and present some examples from the scenarios. We show that the proposed framework and architecture are a valid tool for the study of intelligent musical devices.
DeepCell: A Ubiquitous Accurate Provider-side Cellular-based Localization
Venue: arXiv cs.CY 2024
Year: 2024

Although outdoor localization is already available to the general public and businesses through the wide spread use of the GPS, it is not supported by low-end phones, requires a direct line of sight to satellites and can drain phone battery quickly. The current fingerprinting solutions can provide high-accuracy localization but are based on the client side. This limits their ubiquitous deployment and accuracy. In this paper, we introduce DeepCell: a provider-side fingerprinting localization system that can provide high accuracy localization for any cell phone. To build its fingerprint, DeepCell leverages the unlabeled cellular measurements recorded by the cellular provider while opportunistically synchronizing with selected client devices to get location labels. The fingerprint is then used to train a deep neural network model that is harnessed for localization. To achieve this goal, DeepCell need to address a number of challenges including using unlabeled data from the provider side, handling noise and sparsity, scaling the data to large areas, and finally providing enough data that is required for training deep models without overhead. Evaluation of DeepCell in a typical realistic environment shows that it can achieve a consistent median accuracy of 29m. This accuracy outperforms the state-of-the-art client-based cellular-based systems by more than 75.4%. In addition, the same accuracy is extended to low-end phones.
Wonderful Matrices: More Efficient and Effective Architecture for Language Modeling Tasks
Venue: arXiv cs.LG 2024
Year: 2024

We prove the availability of inner product form position encoding in the state space dual algorithm and study the effectiveness of different position embeddings in the hybrid quadratic causal self-attention and state space dual algorithms. We propose inner function attention with dynamic mask, which can improve the expressiveness of the attention algorithm and avoid the sequence noise significantly affecting the accuracy of the attention score. We also design cross domain mixture of experts, which can improve the granularity of the sparse activation feedforward network while maintaining the efficiency of parameter utilization and retrieval. The combination of these methods constitutes our foundation model architecture: Wonderful Matrices. We conduct experiments on the language modeling task and find that Wonderful Matrices are more efficient and effective in handling complex language tasks.