OpenRaiser/Intern-Atlas huggingface dataset:
Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentation
Venue: arXiv cs.CV 2024
Year: 2024

Despite the significant progress in deep learning for dense visual recognition problems, such as semantic segmentation, traditional methods are constrained by fixed class sets. Meanwhile, vision-language foundation models, such as CLIP, have showcased remarkable effectiveness in numerous zero-shot image-level tasks, owing to their robust generalizability. Recently, a body of work has investigated utilizing these models in open-vocabulary semantic segmentation (OVSS). However, existing approaches often rely on impractical supervised pre-training or access to additional pre-trained networks. In this work, we propose a strong baseline for training-free OVSS, termed Neighbour-Aware CLIP (NACLIP), representing a straightforward adaptation of CLIP tailored for this scenario. Our method enforces localization of patches in the self-attention of CLIP's vision transformer which, despite being crucial for dense prediction tasks, has been overlooked in the OVSS literature. By incorporating design choices favouring segmentation, our approach significantly improves performance without requiring additional data, auxiliary pre-trained networks, or extensive hyperparameter tuning, making it highly practical for real-world applications. Experiments are performed on 8 popular semantic segmentation benchmarks, yielding state-of-the-art performance on most scenarios. Our code is publicly available at https://github.com/sinahmr/NACLIP.
Adapting CNNs for Fisheye Cameras without Retraining
Venue: arXiv cs.CV 2024
Year: 2024

The majority of image processing approaches assume images are in or can be rectified to a perspective projection. However, in many applications it is beneficial to use non conventional cameras, such as fisheye cameras, that have a larger field of view (FOV). The issue arises that these large-FOV images can't be rectified to a perspective projection without significant cropping of the original image. To address this issue we propose Rectified Convolutions (RectConv); a new approach for adapting pre-trained convolutional networks to operate with new non-perspective images, without any retraining. Replacing the convolutional layers of the network with RectConv layers allows the network to see both rectified patches and the entire FOV. We demonstrate RectConv adapting multiple pre-trained networks to perform segmentation and detection on fisheye imagery from two publicly available datasets. Our approach requires no additional data or training, and operates directly on the native image as captured from the camera. We believe this work is a step toward adapting the vast resources available for perspective images to operate across a broad range of camera geometries.
Interest Maximization in Social Networks
Venue: arXiv cs.SI 2024
Year: 2024

Nowadays, organizations use viral marketing strategies to promote their products through social networks. It is expensive to directly send the product promotional information to all the users in the network. In this context, Kempe et al. \cite{kempe2003maximizing} introduced the Influence Maximization (IM) problem, which identifies $k$ most influential nodes (spreader nodes), such that the maximum number of people in the network adopts the promotional message. Many variants of the IM problem have been studied in the literature, namely, Perfect Evangelising Set (PES), Perfect Awareness Problem (PAP), etc. In this work, we propose a maximization version of PAP called the \IM{} problem. Different people have different levels of interest in a particular product. This is modeled by assigning an interest value to each node in the network. Then, the problem is to select $k$ initial spreaders such that the sum of the interest values of the people (nodes) who become aware of the message is maximized. We study the \IM{} problem under two popular diffusion models: the Linear Threshold Model (LTM) and the Independent Cascade Model (ICM). We show that the \IM{} problem is NP-Hard under LTM. We give linear programming formulation for the problem under LTM. We propose four heuristic algorithms for the \IM{} problem: \LBE{} (\LB{}), Maximum Degree First Heuristic (\MD{}), \PBE{} (\PB{}), and Maximum Profit Based Greedy Heuristic (\MP{}). Extensive experimentation has been carried out on many real-world benchmark data sets for both diffusion models. The results show that among the proposed heuristics, \MP{} performs better in maximizing the interest value.
RLEMMO: Evolutionary Multimodal Optimization Assisted By Deep Reinforcement Learning
Venue: arXiv cs.NE 2024
Year: 2024

Solving multimodal optimization problems (MMOP) requires finding all optimal solutions, which is challenging in limited function evaluations. Although existing works strike the balance of exploration and exploitation through hand-crafted adaptive strategies, they require certain expert knowledge, hence inflexible to deal with MMOP with different properties. In this paper, we propose RLEMMO, a Meta-Black-Box Optimization framework, which maintains a population of solutions and incorporates a reinforcement learning agent for flexibly adjusting individual-level searching strategies to match the up-to-date optimization status, hence boosting the search performance on MMOP. Concretely, we encode landscape properties and evolution path information into each individual and then leverage attention networks to advance population information sharing. With a novel reward mechanism that encourages both quality and diversity, RLEMMO can be effectively trained using a policy gradient algorithm. The experimental results on the CEC2013 MMOP benchmark underscore the competitive optimization performance of RLEMMO against several strong baselines.
Practical Region-level Attack against Segment Anything Models
Venue: arXiv cs.CV 2024
Year: 2024

Segment Anything Models (SAM) have made significant advancements in image segmentation, allowing users to segment target portions of an image with a single click (i.e., user prompt). Given its broad applications, the robustness of SAM against adversarial attacks is a critical concern. While recent works have explored adversarial attacks against a pre-defined prompt/click, their threat model is not yet realistic: (1) they often assume the user-click position is known to the attacker (point-based attack), and (2) they often operate under a white-box setting with limited transferability. In this paper, we propose a more practical region-level attack where attackers do not need to know the precise user prompt. The attack remains effective as the user clicks on any point on the target object in the image, hiding the object from SAM. Also, by adapting a spectrum transformation method, we make the attack more transferable under a black-box setting. Both control experiments and testing against real-world SAM services confirm its effectiveness.
Investigating Neural Machine Translation for Low-Resource Languages: Using Bavarian as a Case Study
Venue: arXiv cs.CL 2024
Year: 2024

Machine Translation has made impressive progress in recent years offering close to human-level performance on many languages, but studies have primarily focused on high-resource languages with broad online presence and resources. With the help of growing Large Language Models, more and more low-resource languages achieve better results through the presence of other languages. However, studies have shown that not all low-resource languages can benefit from multilingual systems, especially those with insufficient training and evaluation data. In this paper, we revisit state-of-the-art Neural Machine Translation techniques to develop automatic translation systems between German and Bavarian. We investigate conditions of low-resource languages such as data scarcity and parameter sensitivity and focus on refined solutions that combat low-resource difficulties and creative solutions such as harnessing language similarity. Our experiment entails applying Back-translation and Transfer Learning to automatically generate more training data and achieve higher translation performance. We demonstrate noisiness in the data and present our approach to carry out text preprocessing extensively. Evaluation was conducted using combined metrics: BLEU, chrF and TER. Statistical significance results with Bonferroni correction show surprisingly high baseline systems, and that Back-translation leads to significant improvement. Furthermore, we present a qualitative analysis of translation errors and system limitations.
Pretraining and Updates of Domain-Specific LLM: A Case Study in the Japanese Business Domain
Venue: arXiv cs.CL 2024
Year: 2024

The development of Large Language Models (LLMs) in various languages has been advancing, but the combination of non-English languages with domain-specific contexts remains underexplored. This paper presents our findings from training and evaluating a Japanese business domain-specific LLM designed to better understand business-related documents, such as the news on current affairs, technical reports, and patents. Additionally, LLMs in this domain require regular updates to incorporate the most recent knowledge. Therefore, we also report our findings from the first experiments and evaluations involving updates to this LLM using the latest article data, which is an important problem setting that has not been addressed in previous research. From our experiments on a newly created benchmark dataset for question answering in the target domain, we found that (1) our pretrained model improves QA accuracy without losing general knowledge, and (2) a proper mixture of the latest and older texts in the training data for the update is necessary. Our pretrained model and business domain benchmark are publicly available to support further studies.
Guided Masked Self-Distillation Modeling for Distributed Multimedia Sensor Event Analysis
Venue: arXiv cs.MM 2024
Year: 2024

Observations with distributed sensors are essential in analyzing a series of human and machine activities (referred to as 'events' in this paper) in complex and extensive real-world environments. This is because the information obtained from a single sensor is often missing or fragmented in such an environment; observations from multiple locations and modalities should be integrated to analyze events comprehensively. However, a learning method has yet to be established to extract joint representations that effectively combine such distributed observations. Therefore, we propose Guided Masked sELf-Distillation modeling (Guided-MELD) for inter-sensor relationship modeling. The basic idea of Guided-MELD is to learn to supplement the information from the masked sensor with information from other sensors needed to detect the event. Guided-MELD is expected to enable the system to effectively distill the fragmented or redundant target event information obtained by the sensors without being overly dependent on any specific sensors. To validate the effectiveness of the proposed method in novel tasks of distributed multimedia sensor event analysis, we recorded two new datasets that fit the problem setting: MM-Store and MM-Office. These datasets consist of human activities in a convenience store and an office, recorded using distributed cameras and microphones. Experimental results on these datasets show that the proposed Guided-MELD improves event tagging and detection performance and outperforms conventional inter-sensor relationship modeling methods. Furthermore, the proposed method performed robustly even when sensors were reduced.
AdaContour: Adaptive Contour Descriptor with Hierarchical Representation
Venue: arXiv cs.CV 2024
Year: 2024

Existing angle-based contour descriptors suffer from lossy representation for non-starconvex shapes. By and large, this is the result of the shape being registered with a single global inner center and a set of radii corresponding to a polar coordinate parameterization. In this paper, we propose AdaContour, an adaptive contour descriptor that uses multiple local representations to desirably characterize complex shapes. After hierarchically encoding object shapes in a training set and constructing a contour matrix of all subdivided regions, we compute a robust low-rank robust subspace and approximate each local contour by linearly combining the shared basis vectors to represent an object. Experiments show that AdaContour is able to represent shapes more accurately and robustly than other descriptors while retaining effectiveness. We validate AdaContour by integrating it into off-the-shelf detectors to enable instance segmentation which demonstrates faithful performance. The code is available at https://github.com/tding1/AdaContour.
Performance Analysis of Decentralized Physical Infrastructure Networks and Centralized Clouds
Venue: arXiv cs.CR 2024
Year: 2024

The advent of Decentralized Physical Infrastructure Networks (DePIN) represents a shift in the digital infrastructure of today's Internet. While Centralized Service Providers (CSP) monopolize cloud computing, DePINs aim to enhance data sovereignty and confidentiality and increase resilience against a single point of failure. Due to the novelty of the emerging field of DePIN, this work focuses on the potential of DePINs to disrupt traditional centralized architectures by taking advantage of the Internet of Things (IoT) devices and crypto-economic design in combination with blockchains. This combination yields Acurast, a more distributed, resilient, and user-centric physical infrastructure deployment. Through comparative analysis with centralized systems, particularly in serverless computing contexts, this work seeks to lay the first steps in scientifically evaluating DePINs and quantitatively comparing them in terms of efficiency and effectiveness in real-world applications. The findings suggest DePINs' potential to (i) reduce trust assumptions and physically decentralized infrastructure, (ii) increase efficiency and performance simultaneously while improving the computation's (iii) confidentiality and verifiability.
Multi-Step Traffic Prediction for Multi-Period Planning in Optical Networks
Venue: arXiv cs.NI 2024
Year: 2024

A multi-period planning framework is proposed that exploits multi-step ahead traffic predictions to address service overprovisioning and improve adaptability to traffic changes, while ensuring the necessary quality-of-service (QoS) levels. An encoder-decoder deep learning model is initially leveraged for multi-step ahead prediction by analyzing real-traffic traces. This information is then exploited by multi-period planning heuristics to efficiently utilize available network resources while minimizing undesired service disruptions (caused due to lightpath re-allocations), with these heuristics outperforming a single-step ahead prediction approach.
Communication-Efficient Model Aggregation with Layer Divergence Feedback in Federated Learning
Venue: arXiv cs.DC 2024
Year: 2024

Federated Learning (FL) facilitates collaborative machine learning by training models on local datasets, and subsequently aggregating these local models at a central server. However, the frequent exchange of model parameters between clients and the central server can result in significant communication overhead during the FL training process. To solve this problem, this paper proposes a novel FL framework, the Model Aggregation with Layer Divergence Feedback mechanism (FedLDF). Specifically, we calculate model divergence between the local model and the global model from the previous round. Then through model layer divergence feedback, the distinct layers of each client are uploaded and the amount of data transferred is reduced effectively. Moreover, the convergence bound reveals that the access ratio of clients has a positive correlation with model performance. Simulation results show that our algorithm uploads local models with reduced communication overhead while upholding a superior global model performance.
On the Impact of Reactive Region on the Near-Field Channel Gain
Venue: arXiv cs.IT 2024
Year: 2024

The near-field channel gain is analyzed by considering both radiating and reactive components of the electromagnetic field. Novel expressions are derived for the channel gains of spatially-discrete (SPD) and continuous-aperture (CAP) arrays, which are more accurate than conventional results that neglect the reactive region. To gain further insights, asymptotic analyses are carried out in the large aperture size, based on which the impact of the reactive region is discussed. It is proved that for both SPD and CAP arrays, the impact of the reactive region on near-field channel gain is negligible, even as the array aperture size approaches infinity.
Automatic Speech Recognition Advancements for Indigenous Languages of the Americas
Venue: arXiv cs.CL 2024
Year: 2024

Indigenous languages are a fundamental legacy in the development of human communication, embodying the unique identity and culture of local communities in America. The Second AmericasNLP (Americas Natural Language Processing) Competition Track 1 of NeurIPS (Neural Information Processing Systems) 2022 proposed the task of training automatic speech recognition (ASR) systems for five Indigenous languages: Quechua, Guarani, Bribri, Kotiria, and Wa'ikhana. In this paper, we describe the fine-tuning of a state-of-the-art ASR model for each target language, using approximately 36.65 h of transcribed speech data from diverse sources enriched with data augmentation methods. We systematically investigate, using a Bayesian search, the impact of the different hyperparameters on the Wav2vec2.0 XLS-R (Cross-Lingual Speech Representations) variants of 300 M and 1 B parameters. Our findings indicate that data and detailed hyperparameter tuning significantly affect ASR accuracy, but language complexity determines the final result. The Quechua model achieved the lowest character error rate (CER) (12.14), while the Kotiria model, despite having the most extensive dataset during the fine-tuning phase, showed the highest CER (36.59). Conversely, with the smallest dataset, the Guarani model achieved a CER of 15.59, while Bribri and Wa'ikhana obtained, respectively, CERs of 34.70 and 35.23. Additionally, Sobol' sensitivity analysis highlighted the crucial roles of freeze fine-tuning updates and dropout rates. We release our best models for each language, marking the first open ASR models for Wa'ikhana and Kotiria. This work opens avenues for future research to advance ASR techniques in preserving minority Indigenous languages
Joint Physical-Digital Facial Attack Detection Via Simulating Spoofing Clues
Venue: arXiv cs.CV 2024
Year: 2024

Face recognition systems are frequently subjected to a variety of physical and digital attacks of different types. Previous methods have achieved satisfactory performance in scenarios that address physical attacks and digital attacks, respectively. However, few methods are considered to integrate a model that simultaneously addresses both physical and digital attacks, implying the necessity to develop and maintain multiple models. To jointly detect physical and digital attacks within a single model, we propose an innovative approach that can adapt to any network architecture. Our approach mainly contains two types of data augmentation, which we call Simulated Physical Spoofing Clues augmentation (SPSC) and Simulated Digital Spoofing Clues augmentation (SDSC). SPSC and SDSC augment live samples into simulated attack samples by simulating spoofing clues of physical and digital attacks, respectively, which significantly improve the capability of the model to detect "unseen" attack types. Extensive experiments show that SPSC and SDSC can achieve state-of-the-art generalization in Protocols 2.1 and 2.2 of the UniAttackData dataset, respectively. Our method won first place in "Unified Physical-Digital Face Attack Detection" of the 5th Face Anti-spoofing Challenge@CVPR2024. Our final submission obtains 3.75% APCER, 0.93% BPCER, and 2.34% ACER, respectively. Our code is available at https://github.com/Xianhua-He/cvpr2024-face-anti-spoofing-challenge.
Supervisory Control Theory with Event Forcing
Venue: arXiv cs.FL 2024
Year: 2024

In the Ramadge-Wonham supervisory control theory the only interaction mechanism between supervisor and plant is that the supervisor may enable/disable events from the plant and the plant makes a final decision about which of the enabled events is actually taking place. In this paper, the interaction between supervisor and plant is enriched by allowing the supervisor to force specific events (called forcible events) that are allowed to preempt uncontrollable events. A notion of forcible-controllability is defined that captures the interplay between controllability of a supervisor w.r.t. the uncontrollable events provided by a plant in the setting with event forcing. Existence of a maximally permissive, forcibly-controllable, nonblocking supervisor is shown and an algorithm is provided that computes such a supervisor. The approach is illustrated by two small case studies.
Revisiting Feature Prediction for Learning Visual Representations from Video
Venue: arXiv cs.CV 2024
Year: 2024

This paper explores feature prediction as a stand-alone objective for unsupervised learning from video and introduces V-JEPA, a collection of vision models trained solely using a feature prediction objective, without the use of pretrained image encoders, text, negative examples, reconstruction, or other sources of supervision. The models are trained on 2 million videos collected from public datasets and are evaluated on downstream image and video tasks. Our results show that learning by predicting video features leads to versatile visual representations that perform well on both motion and appearance-based tasks, without adaption of the model's parameters; e.g., using a frozen backbone. Our largest model, a ViT-H/16 trained only on videos, obtains 81.9% on Kinetics-400, 72.2% on Something-Something-v2, and 77.9% on ImageNet1K.
SpectralMamba: Efficient Mamba for Hyperspectral Image Classification
Venue: arXiv cs.CV 2024
Year: 2024

Recurrent neural networks and Transformers have recently dominated most applications in hyperspectral (HS) imaging, owing to their capability to capture long-range dependencies from spectrum sequences. However, despite the success of these sequential architectures, the non-ignorable inefficiency caused by either difficulty in parallelization or computationally prohibitive attention still hinders their practicality, especially for large-scale observation in remote sensing scenarios. To address this issue, we herein propose SpectralMamba -- a novel state space model incorporated efficient deep learning framework for HS image classification. SpectralMamba features the simplified but adequate modeling of HS data dynamics at two levels. First, in spatial-spectral space, a dynamical mask is learned by efficient convolutions to simultaneously encode spatial regularity and spectral peculiarity, thus attenuating the spectral variability and confusion in discriminative representation learning. Second, the merged spectrum can then be efficiently operated in the hidden state space with all parameters learned input-dependent, yielding selectively focused responses without reliance on redundant attention or imparallelizable recurrence. To explore the room for further computational downsizing, a piece-wise scanning mechanism is employed in-between, transferring approximately continuous spectrum into sequences with squeezed length while maintaining short- and long-term contextual profiles among hundreds of bands. Through extensive experiments on four benchmark HS datasets acquired by satellite-, aircraft-, and UAV-borne imagers, SpectralMamba surprisingly creates promising win-wins from both performance and efficiency perspectives.