OpenRaiser/Intern-Atlas huggingface dataset:
Towards Explaining Satellite Based Poverty Predictions with Convolutional Neural Networks
Venue: arXiv cs.CV 2023
Year: 2023

Deep convolutional neural networks (CNNs) have been shown to predict poverty and development indicators from satellite images with surprising accuracy. This paper presents a first attempt at analyzing the CNNs responses in detail and explaining the basis for the predictions. The CNN model, while trained on relatively low resolution day- and night-time satellite images, is able to outperform human subjects who look at high-resolution images in ranking the Wealth Index categories. Multiple explainability experiments performed on the model indicate the importance of the sizes of the objects, pixel colors in the image, and provide a visualization of the importance of different structures in input images. A visualization is also provided of type images that maximize the network prediction of Wealth Index, which provides clues on what the CNN prediction is based on.
Enhancing Image Captioning with Neural Models
Venue: arXiv cs.CV 2023
Year: 2023

This research explores the realm of neural image captioning using deep learning models. The study investigates the performance of different neural architecture configurations, focusing on the inject architecture, and proposes a novel quality metric for evaluating caption generation. Through extensive experimentation and analysis, this work sheds light on the challenges and opportunities in image captioning, providing insights into model behavior and overfitting. The results reveal that while the merge models exhibit a larger vocabulary and higher ROUGE scores, the inject architecture generates relevant and concise image captions. The study also highlights the importance of refining training data and optimizing hyperparameters for improved model performance. This research contributes to the growing body of knowledge in neural image captioning and encourages further exploration in the field, emphasizing the democratization of artificial intelligence.
Consensus group decision making under model uncertainty with a view towards environmental policy making
Venue: arXiv cs.MA 2023
Year: 2023

In this paper we propose a consensus group decision making scheme under model uncertainty consisting of an iterative two-stage procedure and based on the concept of Fréchet barycenter. Each step consists of two stages: the agents first update their position in the opinion metric space by a local barycenter characterized by the agents' immediate interactions and then a moderator makes a proposal in terms of a global barycenter, checking for consensus at each step. In cases of large heterogeneous groups the procedure can be complemented by an auxiliary initial homogenization step, consisting of a clustering procedure in opinion space, leading to large homogeneous groups for which the aforementioned procedure will be applied. The scheme is illustrated in examples motivated from environmental economics.
Slotted Aloha for Optical Wireless Communications in Internet of Underwater Things
Venue: arXiv cs.NI 2023
Year: 2023

In this work, we design and analyse a Slotted ALOHA (SA) solution for Optical Wireless Communication (OWC)-based Internet of Underwater Things (IoUT). In the proposed system, user devices exchange data with an access point (AP) which exploits the capture effect. The space spanned by the IoUT nodes is three-dimensional, i.e., users are located in half-sphere centered at the AP placed at the bottom of a floating object at the water surface level. The analytical expressions for the system throughput and reliability expressed in terms of the outage probability are derived. Based on the simulated signal-to-noise-and-interference-ratio statistics and derived analytical expressions, we present numerical results that investigate the trade-off between the system performance and the IoUT system parameters, such as the number of users, activation probability and type of water medium. The presented conclusions provide valuable insights into the design of an SA-based solution for IoUT communications.
Efficient and Secure Energy Trading with Electric Vehicles and Distributed Ledger Technology
Venue: arXiv cs.NI 2023
Year: 2023

Efficient energy management of Distributed Renewable Energy Resources (DRER) enables a more sustainable and efficient energy ecosystem. Therefore, we propose a holistic Energy Management System (EMS), utilising the computational and energy storage capabilities of nearby Electric Vehicles (EVs), providing a low-latency and efficient management platform for DRER. Through leveraging the inherent, immutable features of Distributed Ledger Technology (DLT) and smart contracts, we create a secure management environment, facilitating interactions between multiple EVs and energy resources. Using a privacy-preserving load forecasting method powered by Vehicular Fog Computing (VFC), we integrate the computational resources of the EVs. Using DLT and our forecasting framework, we accommodate efficient management algorithms in a secure and low-latency manner enabling greater utilisation of the energy storage resources. Finally, we assess our proposed EMS in terms of monetary and energy utility metrics, establishing the increased benefits of multiple interacting EVs and load forecasting. Through the proposed system, we have established the potential of our framework to create a more sustainable and efficient energy ecosystem whilst providing measurable benefits to participating agents.
MaxMem: Colocation and Performance for Big Data Applications on Tiered Main Memory Servers
Venue: arXiv cs.OS 2023
Year: 2023

We present MaxMem, a tiered main memory management system that aims to maximize Big Data application colocation and performance. MaxMem uses an application-agnostic and lightweight memory occupancy control mechanism based on fast memory miss ratios to provide application QoS under increasing colocation. By relying on memory access sampling and binning to quickly identify per-process memory heat gradients, MaxMem maximizes performance for many applications sharing tiered main memory simultaneously. MaxMem is designed as a user-space memory manager to be easily modifiable and extensible, without complex kernel code development. On a system with tiered main memory consisting of DRAM and Intel Optane persistent memory modules, our evaluation confirms that MaxMem provides 11% and 38% better throughput and up to 80% and an order of magnitude lower 99th percentile latency than HeMem and Linux AutoNUMA, respectively, with a Big Data key-value store in dynamic colocation scenarios.
CellMixer: Annotation-free Semantic Cell Segmentation of Heterogeneous Cell Populations
Venue: arXiv cs.CV 2023
Year: 2023

In recent years, several unsupervised cell segmentation methods have been presented, trying to omit the requirement of laborious pixel-level annotations for the training of a cell segmentation model. Most if not all of these methods handle the instance segmentation task by focusing on the detection of different cell instances ignoring their type. While such models prove adequate for certain tasks, like cell counting, other applications require the identification of each cell's type. In this paper, we present CellMixer, an innovative annotation-free approach for the semantic segmentation of heterogeneous cell populations. Our augmentation-based method enables the training of a segmentation model from image-level labels of homogeneous cell populations. Our results show that CellMixer can achieve competitive segmentation performance across multiple cell types and imaging modalities, demonstrating the method's scalability and potential for broader applications in medical imaging, cellular biology, and diagnostics.
VisionaryVR: An Optical Simulation Tool for Evaluating and Optimizing Vision Correction Solutions in Virtual Reality
Venue: arXiv cs.CV 2023
Year: 2023

Developing and evaluating vision science methods require robust and efficient tools for assessing their performance in various real-world scenarios. This study presents a novel virtual reality (VR) simulation tool that simulates real-world optical methods while giving high experimental control to the experiment. The tool incorporates an experiment controller, to smoothly and easily handle multiple conditions, a generic eye-tracking controller, that works with most common VR eye-trackers, a configurable defocus simulator, and a generic VR questionnaire loader to assess participants' behavior in virtual reality. This VR-based simulation tool bridges the gap between theoretical and applied research on new optical methods, corrections, and therapies. It enables vision scientists to increase their research tools with a robust, realistic, and fast research environment.
${L}^{\infty}$-norm computation for linear time-invariant systems depending on parameters
Venue: arXiv cs.SC 2023
Year: 2023

This paper focuses on representing the $L^{\infty}$-norm of finite-dimensional linear time-invariant systems with parameter-dependent coefficients. Previous studies tackled the problem in a non-parametric scenario by simplifying it to finding the maximum $y$-projection of real solutions $(x, y)$ of a system of the form $Σ=\{P=0, \, \partial P/\partial x=0\}$, where $P \in \Z[x, y]$. To solve this problem, standard computer algebra methods were employed and analyzed \cite{bouzidi2021computation}. In this paper, we extend our approach to address the parametric case. We aim to represent the "maximal" $y$-projection of real solutions of $Σ$ as a function of the given parameters. %a set of parameters $α$. To accomplish this, we utilize cylindrical algebraic decomposition. This method allows us to determine the desired value as a function of the parameters within specific regions of parameter space.
Explaining Knock-on Effects of Bias Mitigation
Venue: arXiv cs.LG 2023
Year: 2023

In machine learning systems, bias mitigation approaches aim to make outcomes fairer across privileged and unprivileged groups. Bias mitigation methods work in different ways and have known "waterfall" effects, e.g., mitigating bias at one place may manifest bias elsewhere. In this paper, we aim to characterise impacted cohorts when mitigation interventions are applied. To do so, we treat intervention effects as a classification task and learn an explainable meta-classifier to identify cohorts that have altered outcomes. We examine a range of bias mitigation strategies that work at various stages of the model life cycle. We empirically demonstrate that our meta-classifier is able to uncover impacted cohorts. Further, we show that all tested mitigation strategies negatively impact a non-trivial fraction of cases, i.e., people who receive unfavourable outcomes solely on account of mitigation efforts. This is despite improvement in fairness metrics. We use these results as a basis to argue for more careful audits of static mitigation interventions that go beyond aggregate metrics.
InceptionCaps: A Performant Glaucoma Classification Model for Data-scarce Environment
Venue: arXiv cs.CV 2023
Year: 2023

Glaucoma is an irreversible ocular disease and is the second leading cause of visual disability worldwide. Slow vision loss and the asymptomatic nature of the disease make its diagnosis challenging. Early detection is crucial for preventing irreversible blindness. Ophthalmologists primarily use retinal fundus images as a non-invasive screening method. Convolutional neural networks (CNN) have demonstrated high accuracy in the classification of medical images. Nevertheless, CNN's translation-invariant nature and inability to handle the part-whole relationship between objects make its direct application unsuitable for glaucomatous fundus image classification, as it requires a large number of labelled images for training. This work reviews existing state of the art models and proposes InceptionCaps, a novel capsule network (CapsNet) based deep learning model having pre-trained InceptionV3 as its convolution base, for automatic glaucoma classification. InceptionCaps achieved an accuracy of 0.956, specificity of 0.96, and AUC of 0.9556, which surpasses several state-of-the-art deep learning model performances on the RIM-ONE v2 dataset. The obtained result demonstrates the robustness of the proposed deep learning model.
Latent Space Explorer: Visual Analytics for Multimodal Latent Space Exploration
Venue: arXiv cs.LG 2023
Year: 2023

Machine learning models built on training data with multiple modalities can reveal new insights that are not accessible through unimodal datasets. For example, cardiac magnetic resonance images (MRIs) and electrocardiograms (ECGs) are both known to capture useful information about subjects' cardiovascular health status. A multimodal machine learning model trained from large datasets can potentially predict the onset of heart-related diseases and provide novel medical insights about the cardiovascular system. Despite the potential benefits, it is difficult for medical experts to explore multimodal representation models without visual aids and to test the predictive performance of the models on various subpopulations. To address the challenges, we developed a visual analytics system called Latent Space Explorer. Latent Space Explorer provides interactive visualizations that enable users to explore the multimodal representation of subjects, define subgroups of interest, interactively decode data with different modalities with the selected subjects, and inspect the accuracy of the embedding in downstream prediction tasks. A user study was conducted with medical experts and their feedback provided useful insights into how Latent Space Explorer can help their analysis and possible new direction for further development in the medical domain.
Grounding Everything: Emerging Localization Properties in Vision-Language Transformers
Venue: arXiv cs.CV 2023
Year: 2023

Vision-language foundation models have shown remarkable performance in various zero-shot settings such as image retrieval, classification, or captioning. But so far, those models seem to fall behind when it comes to zero-shot localization of referential expressions and objects in images. As a result, they need to be fine-tuned for this task. In this paper, we show that pretrained vision-language (VL) models allow for zero-shot open-vocabulary object localization without any fine-tuning. To leverage those capabilities, we propose a Grounding Everything Module (GEM) that generalizes the idea of value-value attention introduced by CLIPSurgery to a self-self attention path. We show that the concept of self-self attention corresponds to clustering, thus enforcing groups of tokens arising from the same object to be similar while preserving the alignment with the language space. To further guide the group formation, we propose a set of regularizations that allows the model to finally generalize across datasets and backbones. We evaluate the proposed GEM framework on various benchmark tasks and datasets for semantic segmentation. It shows that GEM not only outperforms other training-free open-vocabulary localization methods, but also achieves state-of-the-art results on the recently proposed OpenImagesV7 large-scale segmentation benchmark.
SimPLoID: Harnessing probabilistic logic programming for infectious disease epidemiology
Venue: arXiv cs.PL 2023
Year: 2023

High quality epidemiological modelling is essential in order to combat the spread of infectious diseases. In this contribution, we present SimPLoID, an epidemiological modelling framework based on the probabilistic logic programming language ProbLog. SimPLoiI combines concepts from compartmental modelling, such as the classic Susceptible-Infected-Recovered (SIR) model, with network-based modelling. As a proof of concept, SimPLoID showcases the potential of declarative probabilistic logic programming for a natural, flexible and compact expression of infectious disease dynamics. In particular, its modularity makes it easily extendable in the face of changing requirements. This application area benefits especially from the precisely specified semantics of the ProbLog language and from the well-maintained engines, which support a variety of query types from exact inference to Monte Carlo simulation. We also provide a domain-specific language designed for researchers not trained in programming, which is compiled to ProbLog clauses within an interactive Python application.
Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling
Venue: arXiv cs.CV 2023
Year: 2023

Humans possess a remarkable ability to integrate auditory and visual information, enabling a deeper understanding of the surrounding environment. This early fusion of audio and visual cues, demonstrated through cognitive psychology and neuroscience research, offers promising potential for developing multimodal perception models. However, training early fusion architectures poses significant challenges, as the increased model expressivity requires robust learning frameworks to harness their enhanced capabilities. In this paper, we address this challenge by leveraging the masked reconstruction framework, previously successful in unimodal settings, to train audio-visual encoders with early fusion. Additionally, we propose an attention-based fusion module that captures interactions between local audio and visual representations, enhancing the model's ability to capture fine-grained interactions. While effective, this procedure can become computationally intractable, as the number of local representations increases. Thus, to address the computational complexity, we propose an alternative procedure that factorizes the local representations before representing audio-visual interactions. Extensive evaluations on a variety of datasets demonstrate the superiority of our approach in audio-event classification, visual sound localization, sound separation, and audio-visual segmentation. These contributions enable the efficient training of deeply integrated audio-visual models and significantly advance the usefulness of early fusion architectures.
Adding Domain Knowledge to Query-Driven Learned Databases
Venue: arXiv cs.DB 2023
Year: 2023

In recent years, \emph{learned cardinality estimation} has emerged as an alternative to traditional query optimization methods: by training machine learning models over observed query performance, learned cardinality estimation techniques can accurately predict query cardinalities and costs -- accounting for skew, correlated predicates, and many other factors that traditional methods struggle to capture. However, query-driven learned cardinality estimators are dependent on sample workloads, requiring vast amounts of labeled queries. Further, we show that state-of-the-art query-driven techniques can make significant and unpredictable errors on queries that are outside the distribution of their training set. We show that these out-of-distribution errors can be mitigated by incorporating the \emph{domain knowledge} used in traditional query optimizers: \emph{constraints} on values and cardinalities (e.g., based on key-foreign-key relationships, range predicates, and more generally on inclusion and functional dependencies). We develop methods for \emph{semi-supervised} query-driven learned query optimization, based on constraints, and we experimentally demonstrate that such techniques can increase a learned query optimizer's accuracy in cardinality estimation, reduce the reliance on massive labeled queries, and improve the robustness of query end-to-end performance.
PROFL: A Privacy-Preserving Federated Learning Method with Stringent Defense Against Poisoning Attacks
Venue: arXiv cs.CR 2023
Year: 2023

Federated Learning (FL) faces two major issues: privacy leakage and poisoning attacks, which may seriously undermine the reliability and security of the system. Overcoming them simultaneously poses a great challenge. This is because privacy protection policies prohibit access to users' local gradients to avoid privacy leakage, while Byzantine-robust methods necessitate access to these gradients to defend against poisoning attacks. To address these problems, we propose a novel privacy-preserving Byzantine-robust FL framework PROFL. PROFL is based on the two-trapdoor additional homomorphic encryption algorithm and blinding techniques to ensure the data privacy of the entire FL process. During the defense process, PROFL first utilize secure Multi-Krum algorithm to remove malicious gradients at the user level. Then, according to the Pauta criterion, we innovatively propose a statistic-based privacy-preserving defense algorithm to eliminate outlier interference at the feature level and resist impersonation poisoning attacks with stronger concealment. Detailed theoretical analysis proves the security and efficiency of the proposed method. We conducted extensive experiments on two benchmark datasets, and PROFL improved accuracy by 39% to 75% across different attack settings compared to similar privacy-preserving robust methods, demonstrating its significant advantage in robustness.
RobustCalib: Robust Lidar-Camera Extrinsic Calibration with Consistency Learning
Venue: arXiv cs.CV 2023
Year: 2023

Current traditional methods for LiDAR-camera extrinsics estimation depend on offline targets and human efforts, while learning-based approaches resort to iterative refinement for calibration results, posing constraints on their generalization and application in on-board systems. In this paper, we propose a novel approach to address the extrinsic calibration problem in a robust, automatic, and single-shot manner. Instead of directly optimizing extrinsics, we leverage the consistency learning between LiDAR and camera to implement implicit re-calibartion. Specially, we introduce an appearance-consistency loss and a geometric-consistency loss to minimizing the inconsitency between the attrbutes (e.g., intensity and depth) of projected LiDAR points and the predicted ones. This design not only enhances adaptability to various scenarios but also enables a simple and efficient formulation during inference. We conduct comprehensive experiments on different datasets, and the results demonstrate that our method achieves accurate and robust performance. To promote further research and development in this area, we will release our model and code.
Demystifying DeFi MEV Activities in Flashbots Bundle
Venue: arXiv cs.CR 2023
Year: 2023

Decentralized Finance, mushrooming in permissionless blockchains, has attracted a recent surge in popularity. Due to the transparency of permissionless blockchains, opportunistic traders can compete to earn revenue by extracting Miner Extractable Value (MEV), which undermines both the consensus security and efficiency of blockchain systems. The Flashbots bundle mechanism further aggravates the MEV competition because it empowers opportunistic traders with the capability of designing more sophisticated MEV extraction. In this paper, we conduct the first systematic study on DeFi MEV activities in Flashbots bundle by developing ActLifter, a novel automated tool for accurately identifying DeFi actions in transactions of each bundle, and ActCluster, a new approach that leverages iterative clustering to facilitate us to discover known/unknown DeFi MEV activities. Extensive experimental results show that ActLifter can achieve nearly 100% precision and recall in DeFi action identification, significantly outperforming state-of-the-art techniques. Moreover, with the help of ActCluster, we obtain many new observations and discover 17 new kinds of DeFi MEV activities, which occur in 53.12% of bundles but have not been reported in existing studies