Reviewerly/exHarmony huggingface dataset:
Learning to rank approach for refining image retrieval in visual arts

Modern content-based image retrieval systems demonstrate rather good performance in identifying visually similar artworks. However, this task becomes more challenging when art history specialists aim to refine the list of similar artworks based on their criteria, thus we need to train the model to reproduce this refinement. In this paper, we propose an approach for improving the list of similar paintings according to specific simulated criteria. By this approach, we retrieve paintings similar to a request image using ResNet50 model and ANNOY algorithm. Then, we simulate re-ranking based on the two criteria, and use the re-ranked lists for training LambdaMART model. Finally, we demonstrate that the trained model reproduces the re-ranking for the query painting by the specific criteria. We plan to use the proposed approach for reproducing re-rankings made by art history specialists, when this data will be collected.
Non-Volatile III-V/Si Photonic Charge-Trap Flash Memory

We demonstrate, for the first time, non-volatile optical charge-trap memory (CTM) on a heterogeneous III-V/Si integrated photonics platform. It is promising to co-locate computing and reliable data storage on the same photonic layer or elements to enable highly-integrated, energy-efficient, low-latency neuromorphic computing.
DatasetEquity: Are All Samples Created Equal? In The Quest For Equity Within Datasets

Data imbalance is a well-known issue in the field of machine learning, attributable to the cost of data collection, the difficulty of labeling, and the geographical distribution of the data. In computer vision, bias in data distribution caused by image appearance remains highly unexplored. Compared to categorical distributions using class labels, image appearance reveals complex relationships between objects beyond what class labels provide. Clustering deep perceptual features extracted from raw pixels gives a richer representation of the data. This paper presents a novel method for addressing data imbalance in machine learning. The method computes sample likelihoods based on image appearance using deep perceptual embeddings and clustering. It then uses these likelihoods to weigh samples differently during training with a proposed Generalized Focal Loss function. This loss can be easily integrated with deep learning algorithms. Experiments validate the method’s effectiveness across autonomous driving vision datasets including KITTI and nuScenes. The loss function improves state-of-the-art 3D object detection methods, achieving over 200% AP gains on under-represented classes (Cyclist) in the KITTI dataset. The results demonstrate the method is generalizable, complements existing techniques, and is particularly beneficial for smaller datasets and rare classes. Code is available at: https://github.com/towardsautonomy/DatasetEquity
Experimental Evaluation and Expansion Suggestion of Digital Signatures Available on OP-TEE

OP-TEE is an open-source project that uses security concepts and cryptography to create a Trusted Execution Environment (TEE) based on software isolation. Although being open-source and one of the main TEE for Android smartphones, it lacks an in-depth performance/security analysis of available algorithms, especially digital signatures. Therefore, in order to support next-generation of security applications, this work faces the challenge of improving the security of this trusted computing environment by (I) surveying and analyzing the security of the natively implemented digital signature algorithms on OP-TEE; (II) embedding and suggesting modern/secure alternatives as expansion; and (III) conducting experiments to check the performance of each these digital signature algorithms. As a result, users who are interested in adopting OP-TEE will be able to evaluate the security and functionality of various digital signature algorithms. Furthermore, our analysis confirms the necessity to include new algorithms, particularly post-quantum.
Dynamic Type Misuse Detection and Analysis for Python-Based Edge Device Applications

Developing edge device applications in resource-constrained environments shows challenges in ensuring software stability and development efficiency. This paper addresses the need of enhancing software reliability by exploring the utilization of Python type checking techniques in the context of edge computing. We define seven types of dynamic type misuse frequently appeared in edge device applications and then develop a static type checking tool, PySteck, specifically designed to detect these dynamic type misuses. Our experiments used representative applications from different subdomains of edge computing to evaluate PySteck's effectiveness and efficiency. We also analyzed the distribution of dynamic type misuse instances across different subdomains in the field of edge computing. The results show that PySteck is an effective and efficient tool for detecting dynamic type misuse in Python-based edge device applications. Our study can improve the development of reliable and efficient edge device applications, which further promote the growth and deployment of edge computing solutions across industries.
MERGE: Multi-Entity Relational Reasoning Based Explanation in Visual Question Answering

To handle VQA tasks in complex scenarios involving multiple entities and obtain reliable explanations, models need to fully understand the high-level semantic information of visual and textual features. Existing VQA methods usually lack the exploration of entity relation features, resulting in insufficient answer prediction accuracy and generated explanations that are not sufficiently relevant to the image and question. To address this issue, we use visual relational reasoning to enhance the overall understanding of image scenes and improve the accuracy of predicted answers and explanations. Our proposed method, named Multi-Entity Relational Reasoning based Explanation (MERGE), leverages the construction of action, spatial, and attribute relations among the question-related entities in images. The contextual visual features are encoded through a graph attention mechanism and fused with question and answer embeddings to generate more accurate textual explanations. To validate the effectiveness of our method, we conducted extensive experiments on seven datasets, including VQA-CP, VQA-X, and CLEVR-X. The results demonstrate improved answer accuracy and high-quality explanations. Furthermore, our results show that the supervisory role of explanations can quantitatively improve the accuracy of answer prediction.
On-chip Nonlinear Activation and Gradient Functions for Photonic Backpropagation Training and Inference

We implement two neural nonlinear activation functions and their gradients on a silicon photonic platform by leveraging the nonlinear electro-optic response of a micro-disk modulator. The experimental results are incorporated into a neural network simulation, achieving classification accuracy of >97% on the MNIST dataset.
Targeted Adversarial Attacks on Generalizable Neural Radiance Fields

Neural Radiance Fields (NeRFs) have recently emerged as a powerful tool for 3D scene representation and rendering. These data-driven models can learn to synthesize high-quality images from sparse 2D observations, enabling realistic and interactive scene reconstructions. However, the growing usage of NeRFs in critical applications such as augmented reality, robotics, and virtual environments could be threatened by adversarial attacks.In this paper we present how generalizable NeRFs can be attacked by both low-intensity adversarial attacks and adversarial patches, where the later could be robust enough to be used in real world applications. We also demonstrate targeted attacks, where a specific, predefined output scene is generated by these attack with success.
Multi-Modal Correlated Network with Emotional Reasoning Knowledge for Social Intelligence Question-Answering

The capacity for social reasoning is essential to the development of social intelligence in humans, which we easily acquire through study and experience. The acquisition of such ability by machines, however, is still challenging, even with the diverse deep learning models that are currently available. Recent artificial social intelligence models have achieved state-of-the-art results in question-answering tasks by employing a variety of methods, including self-supervised setups, multi-modal inputs, and so on. However, there is still a gap in the literature regarding the introduction of commonsense knowledge when training the model in social intelligence tasks. In this paper, we propose a Multi-Modal Temporal Correlated Network with Emotional Social Cues (MMTC-ESC). In order to model cross-modal correlations, an attention-based mechanism is used, and contrastive learning is achieved using emotional social cues. Our findings indicate that combining multimodal inputs and using contrastive loss is advantageous for the performance of social intelligence learning.
Instant Continual Learning of Neural Radiance Fields

Neural radiance fields (NeRFs) have emerged as an effective method for novel-view synthesis and 3D scene reconstruction. However, conventional training methods require access to all training views during scene optimization. This assumption may be prohibitive in continual learning scenarios, where new data is acquired in a sequential manner and a continuous update of the NeRF is desired, as in automotive or remote sensing applications. When naively trained in such a continual setting, traditional scene representation frameworks suffer from catastrophic forgetting, where previously learned knowledge is corrupted after training on new data. Prior works in alleviating forgetting with NeRFs suffer from low reconstruction quality and high latency, making them impractical for real-world application. We propose a continual learning framework for training NeRFs that leverages replay-based methods combined with a hybrid explicit– implicit scene representation. Our method outperforms previous methods in reconstruction quality when trained in a continual setting, while having the additional benefit of being an order of magnitude faster.
Quantum MBSE and Quantum SysML

A critical need exists for a quantum reference architecture that adheres to INCOSE, OMG <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">6</sup> standards of model representation. This paper attempts to provide a reference framework for designs in Quantum Information Technology and Quantum Algorithmic protocols. Quantum Communications is seen as a part of this SySML model design.
Revisiting Generalizability in Deepfake Detection: Improving Metrics and Stabilizing Transfer

"Generalizability" is seen as the hallmark quality of a good deepfake detection model. However, standard out-of-domain evaluation datasets are very similar in form to the training data and lag behind the advancements in modern synthesis methods, making them highly insufficient metrics for robustness. We extend the study of transfer performance of three state-of-the-art methods (that use spatial, temporal, and lip-reading features respectively) on four newer fake types released within the last year. Depending on the artifact modes they were trained on, detection methods fail in different scenarios. On diffusion fakes, the aforementioned methods get 96%, 75%, and 51% AUC respectively, whereas on talking-head fakes, the same methods get 80%, 99%, and 92% AUC. We compare various methods of combining spatial and temporal modalities through joint training and feature fusion in order to stabilize generalization performance.We also propose a new, randomized algorithm to synthesize videos that emulate diverse, visually apparent artifacts with implausibilities in human facial-structure. By testing deepfake detectors on highly randomized artifacts, we can measure the level to which detection networks have learned a strong model for "reality", as opposed to memorizing subtle artifact patterns.
Field Trial In Progress for Measuring Global, Direct, Diffuse, and Ground-Reflected Irradiance Using a Static Sensor Array

We report on plans for a field trial now in progress to test measurement of global horizontal irradiance (GHI), direct normal irradiance (DNI), diffuse horizontal irradiance (DHI), and reflected horizontal irradiance (RHI) using an array of static sensors with no moving parts. As in our recent work in this area, the collection of static sensors includes reference cells in multiple orientations. In addition, our current system under test includes a modified reference cell with a collimation tube to admit only diffuse light contributions from a limited region of the sky. We are developing an analysis model to determine GHI, DNI, DHI, and RHI from the combined sensor data. Field trials have recently begun. Results will be published at a later date.
IPCert: Provably Robust Intellectual Property Protection for Machine Learning

Watermarking and fingerprinting are two popular methods to protect intellectual property (IP) of a model. In particular, a model owner can use them to detect whether a given model is a stolen version of its model. Robustness against perturbation added to a model is a key desired property for IP protection methods. In this work, we first show that existing IP protection methods are not robust against model perturbations in the worst-case scenarios as previously thought. Second, we propose a randomized smoothing based framework that can turn a watermarking/fingerprinting method to be provably robust against model perturbations. However, a straightforward application of randomized smoothing achieves suboptimal provable robustness. To address the challenge, we propose optimization strategies to enhance provable robustness. We evaluate our framework on multiple datasets to show its provable robustness.
SynDrone – Multi-modal UAV Dataset for Urban Scenarios

The development of computer vision algorithms for Unmanned Aerial Vehicles (UAVs) imagery heavily relies on the availability of annotated high-resolution aerial data. However, the scarcity of large-scale real datasets with pixel-level annotations poses a significant challenge to researchers as the limited number of images in existing datasets hinders the effectiveness of deep learning models that require a large amount of training data. In this paper, we propose a multimodal synthetic dataset containing both images and 3D data taken at multiple flying heights to address these limitations. In addition to object-level annotations, the provided data also include pixel-level labeling in 28 classes, enabling exploration of the potential advantages in tasks like semantic segmentation. In total, our dataset contains 72k labeled samples that allow for effective training of deep architectures showing promising results in synthetic-to-real adaptation. The dataset will be made publicly available to support the development of novel computer vision methods targeting UAV applications.
Data-Driven Constraint Mining for Realizable Adversarial Samples

Machine learning based malware detectors can identify new patterns in malware not considered by classic signature matching antivirus software, by processing features extracted from cyber artifacts. However, they must be evaluated for robustness against adversarial attacks that cause mis-classifications by making small modifications to the feature values to exploit weaknesses in the machine learning model’s decision surface. The process of feature extraction maps the problem space, or the actual cyber artifact (e.g. file, network flow data), to the feature space, often resulting in dependencies between features. To learn adversarial examples for malware detectors that can be realized in the real world, we need to be able to map feature-value perturbations back to the original problem space. This requires that 1) all feature value changes are consistent with dependencies between features that are inherent in the mapping from the problem space 2) the feature values are realizable in the problem space. While most current methods require expert knowledge to identify these constraints on the feature values, data-driven constraint mining is an automatic and generalized alternative. This work identifies Tabular Constraint Learning (TaCLe) as a method to automatically mine the dependencies between features via analysis of the feature space data. TaCLe is incorporated into a constraint aware adversarial attack strategy to identify the watermark selection space for a backdoor poisoning attack. We find that it enables the attack to find watermarks that are 70% more evasive than the state-of-the-art baseline.
Online Detection of AI-Generated Images

With advancements in AI-generated images coming on a continuous basis, it is increasingly difficult to distinguish traditionally-sourced images (e.g., photos, artwork) from AI-generated ones. Previous detection methods study the generalization from a single generator to another in isolation. However, in reality, new generators are released on a streaming basis. We study generalization in this setting, training on N models and testing on the next (N + k), following the historical release dates of well-known generation methods. Furthermore, images increasingly consist of both real and generated components, for example through image inpainting. Thus, we extend this approach to pixel prediction, demonstrating strong performance using automatically-generated inpainted data. In addition, for settings where commercial models are not publicly available for automatic data generation, we evaluate if pixel detectors can be trained solely on whole synthetic images.
Secure Optical Hashing for Information Compression in a Convolutional Neural Network

Here we introduce an optical hashing and compression scheme inspired by SWIFFT algorithms which enables high-speed processing of encrypted data using analog optical AI accelerators, offering potential performance gains and improved data security compared to electronic machine learning accelerators.
NCQS: Nonlinear Convex Quadrature Surrogate Hyperparameter Optimization

Deep learning has revolutionized artificial intelligence and enabled breakthroughs across various domains. However, as deep learning models continue to grow in scale and complexity, optimizing their hyperparameters for efficient resource utilization becomes a critical challenge. Traditional optimization techniques often assume smooth and continuous loss functions, limiting their effectiveness in this context. In this work, we propose a novel data-driven approach to hyperparameter optimization using a convex quadrature surrogate. By leveraging a set of sampled hyperparameters and their corresponding performance, our method fits a multivariate quadratic surrogate model to identify the optimal hyperparameters. We demonstrate the practicality and effectiveness of our approach by improving the efficiency and performance of various hyperparameter strategies on both closed and open set benchmarks across diverse vision and tabular datasets. Additionally, we showcase its applicability in automatic target recognition tasks. This research contributes to the broader objective of resource-efficient deep learning for computer vision, fostering advancements in model efficiency, computational memory constraints, and latency considerations. Code available here.
Distance Matters For Improving Performance Estimation Under Covariate Shift

Performance estimation under covariate shift is a crucial component of safe AI model deployment, especially for sensitive use-cases. Recently, several solutions were proposed to tackle this problem, most leveraging model predictions or softmax confidence to derive accuracy estimates. However, under dataset shifts confidence scores may become ill-calibrated if samples are too far from the training distribution. In this work, we show that taking into account distances of test samples to their expected training distribution can significantly improve performance estimation under covariate shift. Precisely, we introduce a "distance-check" to flag samples that lie too far from the expected distribution, to avoid relying on their untrustworthy model outputs in the accuracy estimation step. We demonstrate the effectiveness of this method on 13 image classification tasks, across a wide-range of natural and synthetic distribution shifts and hundreds of models, with a median relative MAE improvement of 27% over the best baseline across all tasks, and SOTA performance on 10 out of 13 tasks. Our code is publicly available at https://github.com/melanibe/distance_matters_performance_estimation.
Mapping Memes to Words for Multimodal Hateful Meme Classification

Multimodal image-text memes are prevalent on the internet, serving as a unique form of communication that combines visual and textual elements to convey humor, ideas, or emotions. However, some memes take a malicious turn, promoting hateful content and perpetuating discrimination. Detecting hateful memes within this multimodal context is a challenging task that requires understanding the intertwined meaning of text and images. In this work, we address this issue by proposing a novel approach named ISSUES for multimodal hateful meme classification. ISSUES leverages a pre-trained CLIP vision-language model and the textual inversion technique to effectively capture the multimodal semantic content of the memes. The experiments show that our method achieves state-of-the-art results on the Hateful Memes Challenge and HarMeme datasets. The code and the pre-trained models are publicly available at https://github.com/miccunifi/ISSUES
Neurosymbolic AI in Cybersecurity: Bridging Pattern Recognition and Symbolic Reasoning

In the face of escalating cyber threats, traditional security measures often fall short, prompting the need for more advanced and interpretable solutions. Neurosymbolic AI, which synergistically combines the pattern recognition capabilities of neural networks with the explicit reasoning of symbolic systems, presents a promising avenue in this realm. This paper offers a comprehensive exploration of Neurosymbolic AI and its potential in enhancing Intrusion Detection Systems (IDS). By integrating data-driven learning with structured reasoning, Neurosymbolic AI promises more robust, adaptive, and transparent cybersecurity solutions. However, challenges such as model interpretability, data requirements, and adaptability in dynamic threat landscapes persist. This paper provides an overview of these challenges, emphasizing the transformative potential of Neurosymbolic AI in fortifying the cybersecurity domain.