OpenRaiser/Intern-Atlas huggingface dataset:
Performance Level Evaluation Model based on ELM
Venue: arXiv cs.HC 2024
Year: 2024

Human factor evaluation is crucial in designing civil aircraft cockpits. This process relies on the physiological and cognitive characteristics of the flight crew to ensure that the cockpit design aligns with their capabilities and enhances flight safety. Modern physiological data acquisition and analysis technology, developed to replace traditional subjective human evaluation, has become an effective method for verifying and evaluating cockpit human factors design. Given the high-dimensional and complex nature of pilot physiological signals, these uncertainties significantly impact pilot performance. This paper proposes a pilot performance evaluation model based on an Extreme Learning Machine (ELM) to predict flight performance through pilots' physiological signals and further explores the quantitative relationship between human factors and civil aviation safety.
When Does Visual Prompting Outperform Linear Probing for Vision-Language Models? A Likelihood Perspective
Venue: arXiv cs.CV 2024
Year: 2024

Adapting pre-trained models to new tasks can exhibit varying effectiveness across datasets. Visual prompting, a state-of-the-art parameter-efficient transfer learning method, can significantly improve the performance of out-of-distribution tasks. On the other hand, linear probing, a standard transfer learning method, can sometimes become the best approach. We propose a log-likelihood ratio (LLR) approach to analyze the comparative benefits of visual prompting and linear probing. By employing the LLR score alongside resource-efficient visual prompts approximations, our cost-effective measure attains up to a 100-fold reduction in run time compared to full training, while achieving prediction accuracies up to 91%. The source code is available at https://github.com/IBM/VP-LLR.
AstroMAE: Redshift Prediction Using a Masked Autoencoder with a Novel Fine-Tuning Architecture
Venue: arXiv cs.CV 2024
Year: 2024

Redshift prediction is a fundamental task in astronomy, essential for understanding the expansion of the universe and determining the distances of astronomical objects. Accurate redshift prediction plays a crucial role in advancing our knowledge of the cosmos. Machine learning (ML) methods, renowned for their precision and speed, offer promising solutions for this complex task. However, traditional ML algorithms heavily depend on labeled data and task-specific feature extraction. To overcome these limitations, we introduce AstroMAE, an innovative approach that pretrains a vision transformer encoder using a masked autoencoder method on Sloan Digital Sky Survey (SDSS) images. This technique enables the encoder to capture the global patterns within the data without relying on labels. To the best of our knowledge, AstroMAE represents the first application of a masked autoencoder to astronomical data. By ignoring labels during the pretraining phase, the encoder gathers a general understanding of the data. The pretrained encoder is subsequently fine-tuned within a specialized architecture tailored for redshift prediction. We evaluate our model against various vision transformer architectures and CNN-based models, demonstrating the superior performance of AstroMAEs pretrained model and fine-tuning architecture.
The Role of Large Language Models in Musicology: Are We Ready to Trust the Machines?
Venue: arXiv cs.SD 2024
Year: 2024

In this work, we explore the use and reliability of Large Language Models (LLMs) in musicology. From a discussion with experts and students, we assess the current acceptance and concerns regarding this, nowadays ubiquitous, technology. We aim to go one step further, proposing a semi-automatic method to create an initial benchmark using retrieval-augmented generation models and multiple-choice question generation, validated by human experts. Our evaluation on 400 human-validated questions shows that current vanilla LLMs are less reliable than retrieval augmented generation from music dictionaries. This paper suggests that the potential of LLMs in musicology requires musicology driven research that can specialized LLMs by including accurate and reliable domain knowledge.
Real-Time Indoor Object Detection based on hybrid CNN-Transformer Approach
Venue: arXiv cs.CV 2024
Year: 2024

Real-time object detection in indoor settings is a challenging area of computer vision, faced with unique obstacles such as variable lighting and complex backgrounds. This field holds significant potential to revolutionize applications like augmented and mixed realities by enabling more seamless interactions between digital content and the physical world. However, the scarcity of research specifically fitted to the intricacies of indoor environments has highlighted a clear gap in the literature. To address this, our study delves into the evaluation of existing datasets and computational models, leading to the creation of a refined dataset. This new dataset is derived from OpenImages v7, focusing exclusively on 32 indoor categories selected for their relevance to real-world applications. Alongside this, we present an adaptation of a CNN detection model, incorporating an attention mechanism to enhance the model's ability to discern and prioritize critical features within cluttered indoor scenes. Our findings demonstrate that this approach is not just competitive with existing state-of-the-art models in accuracy and speed but also opens new avenues for research and application in the field of real-time indoor object detection.
Latent Distillation for Continual Object Detection at the Edge
Venue: arXiv cs.CV 2024
Year: 2024

While numerous methods achieving remarkable performance exist in the Object Detection literature, addressing data distribution shifts remains challenging. Continual Learning (CL) offers solutions to this issue, enabling models to adapt to new data while maintaining performance on previous data. This is particularly pertinent for edge devices, common in dynamic environments like automotive and robotics. In this work, we address the memory and computation constraints of edge devices in the Continual Learning for Object Detection (CLOD) scenario. Specifically, (i) we investigate the suitability of an open-source, lightweight, and fast detector, namely NanoDet, for CLOD on edge devices, improving upon larger architectures used in the literature. Moreover, (ii) we propose a novel CL method, called Latent Distillation~(LD), that reduces the number of operations and the memory required by state-of-the-art CL approaches without significantly compromising detection performance. Our approach is validated using the well-known VOC and COCO benchmarks, reducing the distillation parameter overhead by 74\% and the Floating Points Operations~(FLOPs) by 56\% per model update compared to other distillation methods.
Detecting and Measuring Security Implications of Entangled Domain Verification in CDN
Venue: arXiv cs.CR 2024
Year: 2024

Content Delivery Networks (CDNs) offer a protection layer for enhancing the security of websites. However, a significant security flaw named Absence of Domain Verification (DVA) has become emerging recently. Although this threat is recognized, the current practices and security flaws of domain verification strategies in CDNs have not been thoroughly investigated. In this paper, we present DVAHunter, an automated system for detecting DVA vulnerabilities that can lead to domain abuse in CDNs. Our evaluation of 45 major CDN providers reveals the prevalence of DVA: most (39/45) providers do not perform any verification, and even those that do remain exploitable. Additionally, we used DVAHunter to conduct a large-scale measurement of 89M subdomains from Tranco's Top 1M sites hosted on the 45 CDNs under evaluation. Our focus was on two primary DVA exploitation scenarios: covert communication and domain hijacking. We identified over 332K subdomains vulnerable to domain abuse. This tool provides deeper insights into DVA exploitation and allows us to propose viable mitigation practices for CDN providers. To date, we have received vulnerability confirmations from 12 providers; 6 (e.g., Edgio, Kuocai) have implemented fixes, and 1 (ChinaNetCenter) are actively working on solutions based on our recommendations.
3D-LEX v1.0: 3D Lexicons for American Sign Language and Sign Language of the Netherlands
Venue: arXiv cs.CV 2024
Year: 2024

In this work, we present an efficient approach for capturing sign language in 3D, introduce the 3D-LEX v1.0 dataset, and detail a method for semi-automatic annotation of phonetic properties. Our procedure integrates three motion capture techniques encompassing high-resolution 3D poses, 3D handshapes, and depth-aware facial features, and attains an average sampling rate of one sign every 10 seconds. This includes the time for presenting a sign example, performing and recording the sign, and archiving the capture. The 3D-LEX dataset includes 1,000 signs from American Sign Language and an additional 1,000 signs from the Sign Language of the Netherlands. We showcase the dataset utility by presenting a simple method for generating handshape annotations directly from 3D-LEX. We produce handshape labels for 1,000 signs from American Sign Language and evaluate the labels in a sign recognition task. The labels enhance gloss recognition accuracy by 5% over using no handshape annotations, and by 1% over expert annotations. Our motion capture data supports in-depth analysis of sign features and facilitates the generation of 2D projections from any viewpoint. The 3D-LEX collection has been aligned with existing sign language benchmarks and linguistic resources, to support studies in 3D-aware sign language processing.
Focus Agent: LLM-Powered Virtual Focus Group
Venue: arXiv cs.HC 2024
Year: 2024

In the domain of Human-Computer Interaction, focus groups represent a widely utilised yet resource-intensive methodology, often demanding the expertise of skilled moderators and meticulous preparatory efforts. This study introduces the ``Focus Agent,'' a Large Language Model (LLM) powered framework that simulates both the focus group (for data collection) and acts as a moderator in a focus group setting with human participants. To assess the data quality derived from the Focus Agent, we ran five focus group sessions with a total of 23 human participants as well as deploying the Focus Agent to simulate these discussions with AI participants. Quantitative analysis indicates that Focus Agent can generate opinions similar to those of human participants. Furthermore, the research exposes some improvements associated with LLMs acting as moderators in focus group discussions that include human participants.
GradINN: Gradient Informed Neural Network
Venue: arXiv cs.LG 2024
Year: 2024

We propose Gradient Informed Neural Networks (GradINNs), a methodology inspired by Physics Informed Neural Networks (PINNs) that can be used to efficiently approximate a wide range of physical systems for which the underlying governing equations are completely unknown or cannot be defined, a condition that is often met in complex engineering problems. GradINNs leverage prior beliefs about a system's gradient to constrain the predicted function's gradient across all input dimensions. This is achieved using two neural networks: one modeling the target function and an auxiliary network expressing prior beliefs, e.g., smoothness. A customized loss function enables training the first network while enforcing gradient constraints derived from the auxiliary network. We demonstrate the advantages of GradINNs, particularly in low-data regimes, on diverse problems spanning non time-dependent systems (Friedman function, Stokes Flow) and time-dependent systems (Lotka-Volterra, Burger's equation). Experimental results showcase strong performance compared to standard neural networks and PINN-like approaches across all tested scenarios.
Evaluating the precision of the HTC VIVE Ultimate Tracker with robotic and human movements under varied environmental conditions
Venue: arXiv cs.RO 2024
Year: 2024

The HTC VIVE Ultimate Tracker, utilizing inside-out tracking with internal stereo cameras providing 6 DoF tracking without external cameras, offers a cost-efficient and straightforward setup for motion tracking. Initially designed for the gaming and VR industry, we explored its application beyond VR, providing source code for data capturing in both C++ and Python without requiring a VR headset. This study is the first to evaluate the tracker's precision across various experimental scenarios. To assess the robustness of the tracking precision, we employed a robotic arm as a precise and repeatable source of motion. Using the OptiTrack system as a reference, we conducted tests under varying experimental conditions: lighting, movement velocity, environmental changes caused by displacing objects in the scene, and human movement in front of the trackers, as well as varying the displacement size relative to the calibration center. On average, the HTC VIVE Ultimate Tracker achieved a precision of 4.98 mm +/- 4 mm across various conditions. The most critical factors affecting accuracy were lighting conditions, movement velocity, and range of motion relative to the calibration center. For practical evaluation, we captured human movements with 5 trackers in realistic motion capture scenarios. Our findings indicate sufficient precision for capturing human movements, validated through two tasks: a low-dynamic pick-and-place task and high-dynamic fencing movements performed by an elite athlete. Even though its precision is lower than that of conventional fixed-camera-based motion capture systems and its performance is influenced by several factors, the HTC VIVE Ultimate Tracker demonstrates adequate accuracy for a variety of motion tracking applications. Its ability to capture human or object movements outside of VR or MOCAP environments makes it particularly versatile.
Deep learning for objective estimation of Parkinsonian tremor severity
Venue: arXiv cs.CV 2024
Year: 2024

Accurate assessment of Parkinsonian tremor is vital for monitoring disease progression and evaluating treatment efficacy. We introduce a pixel-based deep learning model designed to analyse postural tremor in Parkinson's disease (PD) from video data, overcoming the limitations of traditional pose estimation techniques. Trained on 2,742 assessments from five specialised movement disorder centres across two continents, the model demonstrated robust concordance with clinical evaluations. It effectively predicted treatment effects for levodopa and deep brain stimulation (DBS), detected lateral asymmetry of symptoms, and differentiated between different tremor severities. Feature space analysis revealed a non-linear, structured distribution of tremor severity, with low-severity scores occupying a larger portion of the feature space. The model also effectively identified outlier videos, suggesting its potential for adaptive learning and quality control in clinical settings. Our approach offers a scalable and objective method for tremor scoring, with potential integration into other MDS-UPDRS motor assessments, including bradykinesia and gait. The system's adaptability and performance underscore its promise for high-frequency, longitudinal monitoring of PD symptoms, complementing clinical expertise and enhancing decision-making in patient management. Future work will extend this pixel-based methodology to other cardinal symptoms of PD, aiming to develop a comprehensive, multi-symptom model for automated Parkinson's disease severity assessment.
Foundations of Large Language Model Compression -- Part 1: Weight Quantization
Venue: arXiv cs.LG 2024
Year: 2024

In recent years, compression of large language models (LLMs) has emerged as an important problem to enable language model deployment on resource-constrained devices, reduce computational costs, and mitigate the environmental footprint of large-scale AI infrastructure. In this paper, we lay down the foundation for LLM quantization from a convex optimization perspective and propose a quantization technique that builds on this foundation for optimum quantization outcomes. Our quantization framework, CVXQ, scales to models containing hundreds of billions of weight parameters and provides users with the flexibility to compress models to any specified model size, post-training. A reference implementation of CVXQ can be obtained from github.com/seannz/cvxq.
AllWeatherNet:Unified Image Enhancement for Autonomous Driving under Adverse Weather and Lowlight-conditions
Venue: arXiv cs.CV 2024
Year: 2024

Adverse conditions like snow, rain, nighttime, and fog, pose challenges for autonomous driving perception systems. Existing methods have limited effectiveness in improving essential computer vision tasks, such as semantic segmentation, and often focus on only one specific condition, such as removing rain or translating nighttime images into daytime ones. To address these limitations, we propose a method to improve the visual quality and clarity degraded by such adverse conditions. Our method, AllWeather-Net, utilizes a novel hierarchical architecture to enhance images across all adverse conditions. This architecture incorporates information at three semantic levels: scene, object, and texture, by discriminating patches at each level. Furthermore, we introduce a Scaled Illumination-aware Attention Mechanism (SIAM) that guides the learning towards road elements critical for autonomous driving perception. SIAM exhibits robustness, remaining unaffected by changes in weather conditions or environmental scenes. AllWeather-Net effectively transforms images into normal weather and daytime scenes, demonstrating superior image enhancement results and subsequently enhancing the performance of semantic segmentation, with up to a 5.3% improvement in mIoU in the trained domain. We also show our model's generalization ability by applying it to unseen domains without re-training, achieving up to 3.9% mIoU improvement. Code can be accessed at: https://github.com/Jumponthemoon/AllWeatherNet.
ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis
Venue: arXiv cs.CV 2024
Year: 2024

Despite recent advancements in neural 3D reconstruction, the dependence on dense multi-view captures restricts their broader applicability. In this work, we propose \textbf{ViewCrafter}, a novel method for synthesizing high-fidelity novel views of generic scenes from single or sparse images with the prior of video diffusion model. Our method takes advantage of the powerful generation capabilities of video diffusion model and the coarse 3D clues offered by point-based representation to generate high-quality video frames with precise camera pose control. To further enlarge the generation range of novel views, we tailored an iterative view synthesis strategy together with a camera trajectory planning algorithm to progressively extend the 3D clues and the areas covered by the novel views. With ViewCrafter, we can facilitate various applications, such as immersive experiences with real-time rendering by efficiently optimizing a 3D-GS representation using the reconstructed 3D points and the generated novel views, and scene-level text-to-3D generation for more imaginative content creation. Extensive experiments on diverse datasets demonstrate the strong generalization capability and superior performance of our method in synthesizing high-fidelity and consistent novel views.
Political DEBATE: Efficient Zero-shot and Few-shot Classifiers for Political Text
Venue: arXiv cs.CL 2024
Year: 2024

Social scientists quickly adopted large language models due to their ability to annotate documents without supervised training, an ability known as zero-shot learning. However, due to their compute demands, cost, and often proprietary nature, these models are often at odds with replication and open science standards. This paper introduces the Political DEBATE (DeBERTa Algorithm for Textual Entailment) language models for zero-shot and few-shot classification of political documents. These models are not only as good, or better than, state-of-the art large language models at zero and few-shot classification, but are orders of magnitude more efficient and completely open source. By training the models on a simple random sample of 10-25 documents, they can outperform supervised classifiers trained on hundreds or thousands of documents and state-of-the-art generative models with complex, engineered prompts. Additionally, we release the PolNLI dataset used to train these models -- a corpus of over 200,000 political documents with highly accurate labels across over 800 classification tasks.
UFO, universe, reptilians and creatures communities on Brazilian Telegram: when the sky is not the limit and conspiracy theories seek answers beyond humanity
Venue: arXiv cs.CY 2024
Year: 2024

Interest in extraterrestrial phenomena and conspiracy theories involving UFOs and reptilians has been growing on Brazilian Telegram, especially in times of global uncertainty, such as during the COVID-19 pandemic. Therefore, this study aims to address the research question: how are Brazilian conspiracy theory communities on UFO, universe, reptilians and creatures topics characterized and articulated on Telegram? It is worth noting that this study is part of a series of seven studies whose main objective is to understand and characterize Brazilian conspiracy theory communities on Telegram. This series of seven studies is openly and originally available on arXiv at Cornell University, applying a mirrored method across the seven studies, changing only the thematic object of analysis and providing investigation replicability, including with proprietary and authored codes, adding to the culture of free and open-source software. Regarding the main findings of this study, the following were observed: UFO communities act as gateways for theories about reptilians, connecting narratives of global control with extraterrestrial beings; Discussions about UFOs and the universe grew significantly during the Pandemic, reflecting a renewed interest in extraterrestrial phenomena; Reptilians remain a significant subculture within conspiracy theories, with a notable growth during the Pandemic; The thematic overlap between UFOs, reptilians and esotericism reveals a cohesive ecosystem of disinformation, making factual correction a challenge; UFO communities function as amplifiers of other conspiracy theories, connecting different themes and strengthening the disinformation network.