Reviewerly/exHarmony huggingface dataset:
A Deep Learning Solution for Arabic Words Sign Language Recognition in the Context of Sentences

Sign language recognition is used to facilitate the communication between the hearing and the deaf societies. This paper proposes a video-based approach to recognizing Arabic sign words in the context of sentences. This is important as in real life the start and end positions of the arms are not necessarily at the sides of the signer. Rather, the position depends on the previous and successive words in the same sentence. This paper proposes to use video coding techniques to generate Sum of Displaced Differences Images (SDDI) of sign words using optical flow followed by the summation of the prediction errors of video frames. Time-dependent feature vectors are then generated from these images using transfer learning. Consequently, model generation is performed using recurrent neural networks. Experimental results performed on 80 sign language words represented in 40 sentences revealed that the proposed solution results in a word recognition rate of 98.6%, which surpasses the accuracy of existing work that uses the same sign language dataset.
Adversarial Patch Attacks Against Aerial Imagery Object Detectors Based on Interpretable Guidance

Although image recognition and classification methods based on deep neural networks (DNNs) have accomplished remarkable achievements in the domain of computer vision, more and more research in recent years has shown that these models may have vulnerabilities that can be easily exploited by attackers, leading to incorrect output results. Especially in the realm of object detection, the incorporation of meticulously crafted patches onto images can potentially deceive DNN-based detectors, resulting in erroneous output outcomes. These patches can be carefully designed to greatly interfere with and mislead the output results of deep neural networks. These misleading output results may lead to serious security issues and losses, so it is becoming increasingly important to study and strengthen the robustness and security of DNNs based target detectors. However, previous methods did not consider the interpretability of the model, and this paper proposes a new mothed to adversarial patch attacks. First, we employ the model interpretable algorithm Grad-CAM to obtain the patch's location within the model's region of interest for the target. Subsequently, we calculate the precise patch placement and apply it accordingly. Following that, we utilize a loss function to optimize patch, ensuring its effectiveness. The experimental results obtained from the DOTA dataset indicate that our proposed method effectively degrades the performance of the target detection model. Secondly, we conducted experiments to improve the transferability of adversarial patches across different models, and verified that using patches to cover the regions of interest in the model improves the transferability of patches.
Optical Freespace Michelson Interferometric Reconfigurable Full Complex Convolution Module

Optical dynamic real-time signal processing can expedite computations due to its inherent parallelism, superior speed, and energy efficiency. Here we demonstrate simultaneous optical two-dimensional reconfigurable full complex convolution by Michelson Interferometric modulation methods with Spatial Light Modulators and discuss system fidelity and complex convolution performance.
Validation of Inverter Labeling with Plant Transfer Functions

The large quantity of data sources found within a utility scale photovoltaic plant presents data quality control challenges. One potential issue is mislabeling of the plant's component outputs (e.g. production measurements made at the combiner or inverter level). If a component's output is incorrectly labeled, it presents an obstacle to plant monitoring and maintenance, as operators will not know where fixes are needed. This study aims to demonstrate the possibility of utilizing the Cloud Advection Model to perform quality checks on the labeling of production outputs at a plant component level based on information about the plant's spatial layout. Results utilizing simulated data showed that the plant transfer function predicted by the CAM could provide discrimination between plant segments that are separated in the cloud motion direction. The discrimination occurred primarily through the phase of the transfer function, but in cases where the spatial dispersion of the plant varied significantly in the cloud motion direction, changes to the transfer function bandwidth were also observable. This methodology shows promise using the simulated plant data in this study, which warrants further study and practical validation of this method utilizing real plant data.
The Use of Biometric Technologies to Increase Security at Sporting Events

The topic of the article is consideration of the possibility of using biometric technologies to increase security during sports events. Security will always be among the most discussed topics, and biometric technologies are one of its modern elements. Currently, sporting events and spectator violence occurring at them are a much-discussed topics. Moreover, as a motive, violence between individual population groups is growing throughout Europe. The number of academic publications dealing with spectator violence is high, but most research in this area is dedicated to describing individual aspects of hooliganism and its causes. Unfortunately, less attention is paid to security measures that could be applied in the specific environment of sports venues. In addition, some sports clubs themselves are resisting the introduction of tougher security measures. They are looking for a possible outflow of participants. However, the experience of organizations that have not been afraid of this confirms that the opposite is true. After the outflow of dangerous fans, there was an increase in the participation of whole families, who no longer had such concerns for their safety. Therefore, the article will first provide an overview of the types of spectator violence and its history to orientate in the issue, and will further focus on the description of individual suitable biometric technologies and their use. It includes an overview of the technologies and experiences of sports clubs that have already included these elements in their preventive activities. In the end, a comparison of the success of selected technologies and recommendations for criteria for their selection and subsequent implementation will be presented.
Banking Fraud Identification and Prevention

The integration of information technologies in the banking sector and their widespread adoption in various financial activities have given rise to banking fraud and scams. As the foremost challenge in the banking sector, fraud and scams directly affect the well-being and prosperity of the global economy and individual nations. The magnitude of bank fraud cannot be known because much of it is undisclosed or undetected. To address this issue, our research focused on comprehending the technological dimensions and manifestations of bank fraud and scams, while also examining the response of governments and financial institutions in preventing and controlling such illicit activities. By surveying users of banking services, some of whom have fallen victim to fraud, and conducting interviews with experts from the financial industry, we successfully identified the most prevalent types of banking fraud and scams. We then compared the perspectives of these two groups to gain a comprehensive understanding. The findings of this study have significant implications for various stakeholders, including government agencies, citizens, corporate managers, financial and non-financial institutions, as well as academics seeking to enhance their knowledge about fraud detection and mitigation.
Design of Electronic Control of PV Tracking Independent of Weather Forecast

With over 28 Terawatt-hours of annual electricity produced globally, only 2% is generated by photovoltaics (PV). Solar tracking is one of the most efficient methods to increase the amount of clean energy produced by PV panels. It was shown in experimental studies that the solar day could become 30% longer, i.e., 30% more solar energy collected daily, if proper single-axis (or dual-axis) tracking systems are installed. However, many commercial systems are controlled by input from local weather forecasts, which rarely take the specific location of a solar farm into account. Thus, usage of pyranometers positioned close to solar farms produce accurate solar irradiance information, despite their delayed response and drawbacks with predicting solar production. In the current study we describe the design and operation of a miniature, low cost, electronic system which tracks the sun and instantly prescribes the optimal orientation of PV panels [1] for any local weather. The system consists of two stepper motors, where each carry three small photodetectors connected to its shaft. The total sensitivity of sensors covers the entire spectrum of silicon solar cells. Stepper motors control the movement of optical sensors scanning the skies in the vertical (elevation) and horizontal (azimuth) directions. The programmable microcontroller stores information of complete elevation and azimuthal scans. The algorithm then predicts the amount of power that could be generated by large PV panels, if their orientation replicates positions of motor shafts. Experimental testing of the novel system supports our method of efficient sun tracking.
Enhancing Low Light Images with VEViD Algorithm using Optical Systems Acceleration (ENVICE)

In this paper, we introduce ENVICE, a new approach aimed at enhancing the performance of the VEViD algorithm. Our approach leverages a real optical diffraction system to accelerate the algorithm, specifically targeting light-level and color enhancement. By incorporating the optical diffraction system, we propose a novel method to improve the speed and efficiency of VEViD, leading to enhanced image quality.
The Efficacy of Transformer-based Adversarial Attacks in Security Domains

Today, the security of many domains rely on the use of Machine Learning to detect threats, identify vulnerabilities, and safeguard systems from attacks. Recently, transformer architectures have improved the state-of-the-art performance on a wide range of tasks such as malware detection and network intrusion detection. But, before abandoning current approaches to transformers, it is crucial to understand their properties and implications on cybersecurity applications. In this paper, we evaluate the robustness of transformers to adversarial samples for system defenders (i.e., resiliency to adversarial perturbations generated on different types of architectures) and their adversarial strength for system attackers (i.e., transferability of adversarial samples generated by transformers to other target models). To that effect, we first fine-tune a set of pre-trained transformer, Convolutional Neural Network (CNN), and hybrid (an ensemble of transformer and CNN) models to solve different downstream image-based tasks. Then, we use an attack algorithm to craft 19,367 adversarial examples on each model for each task. The transferability of these adversarial examples is measured by evaluating each set on other models to determine which models offer more adversarial strength, and consequently, more robustness against these attacks. We find that the adversarial examples crafted on transformers offer the highest transferability rate (i.e., 25.7% higher than the average) onto other models. Similarly, adversarial examples crafted on other models have the lowest rate of transferability (i.e., 56.7% lower than the average) onto transformers. Our work emphasizes the importance of studying transformer architectures for attacking and defending models in security domains, and suggests using them as the primary architecture in transfer attack settings.
Private Membership Aggregation

We consider the problem of private membership aggregation (PMA), in which a user counts the number of times a certain element is stored in a system of independent parties that store arbitrary sets of elements from a universal alphabet. The parties are not allowed to learn which element is being counted by the user. Further, neither the user nor the other parties are allowed to learn the stored elements of each party involved in the process. PMA is a generalization of the recently introduced problem of K private set intersection (K-PSI). The K-PSI problem considers a set of M parties storing arbitrary sets of elements, and a user who wants to determine if a certain element is repeated at least at K parties out of the M parties without learning which party has the required element and which party does not. To solve the general problem of PMA, we dissect it into four categories based on the privacy requirement and the collusions among databases/parties. We map these problems into equivalent private information retrieval (PIR) problems. We propose achievable schemes for each of the four variants of the problem based on the concept of cross-subspace alignment (CSA). The proposed schemes achieve linear communication complexity as opposed to the state-of-the-art K-PSI scheme that requires exponential complexity even though our PMA problems contain more security and privacy constraints.
Channel Aware Adversarial Attacks are Not Robust

Adversarial Machine Learning (AML) has shown significant success when applied to deep learning models across various domains. This paper explores channel-aware adversarial attacks on DNN-based modulation classification models within wireless environments. Our investigation focuses on the robustness of these attacks with respect to channel distribution and path-loss parameters. We examine two scenarios: one in which the attacker has instantaneous channel knowledge and another in which the attacker relies on statistical channel data. In both cases, we study channels subject to Rayleigh fading alone, Rayleigh fading combined with shadowing, and Rayleigh fading combined with both shadowing and path loss. Our findings reveal that the distance between the attacker and the legitimate receiver largely dictates the success of an AML attack. Without precise distance estimation, adversarial attacks are likely to fail.
Volumetric Fast Fourier Convolution for Detecting Ink on the Carbonized Herculaneum Papyri

Recent advancements in Digital Document Restoration (DDR) have led to significant breakthroughs in analyzing highly damaged written artifacts. Among those, there has been an increasing interest in applying Artificial Intelligence techniques for virtually unwrapping and automatically detecting ink on the Herculaneum papyri collection. This collection consists of carbonized scrolls and fragments of documents, which have been digitized via X-ray tomography to allow the development of ad-hoc deep learning-based DDR solutions. In this work, we propose a modification of the Fast Fourier Convolution operator for volumetric data and apply it in a segmentation architecture for ink detection on the challenging Herculaneum papyri, demonstrating its suitability via deep experimental analysis. To encourage the research on this task and the application of the proposed operator to other tasks involving volumetric data, we will release our implementation (https://github.com/aimagelab/vffc).
Geometric Superpixel Representations for Efficient Image Classification with Graph Neural Networks

While Convolutional Neural Networks and Vision Transformers are the go-to solutions for image classification, their model sizes make them expensive to train and deploy. Alternatively, input complexity can be reduced following the intuition that adjacent similar pixels contain redundant information. This prior can be exploited by clustering such pixels into superpixels and connecting adjacent superpixels with edges, resulting in a sparse graph representation on which Graph Neural Networks (GNNs) can operate efficiently. Although previous work clearly highlights the computational efficiency of this approach, this prior can be overly restrictive and, as a result, performance is lacking compared to contemporary dense vision methods. In this work, we propose to extend this prior by incorporating shape information into the individual super-pixel representations. This is achieved through a separate, patch-level GNN. Together with enriching the previously explored appearance and pose information of superpixels and further architectural changes, our best model, ShapeGNN, surpasses the previous state-of-the-art in superpixel-based image classification on CIFAR-10 by a significant margin. We also present an optimised pipeline for efficient image-to-graph transformation and show the viability of training end-to-end on high-resolution images on ImageNet-1k. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
Noise Aware Design Enables Robust Diffractive Deep Neural Network Designs in Visible Wavelengths

Diffractive deep neural networks (D2NNs) are prone to various fabrication errors. Our work shows that including these errors as noise sources in training, ensures the proper functioning and increases the robustness of the fabricated D2NNs.
FedRCIL: Federated Knowledge Distillation for Representation based Contrastive Incremental Learning

The present work proposes a holistic approach to address catastrophic forgetting in the field of computer vision during the process of incremental learning. More specifically, it suggests a series of steps for effective learning of models in distributed environments, based on extracting meaningful representations, modeling them into actual knowledge, and transferring it through a continual distillation mechanism. Additionally, it introduces a federated learning algorithm tailored to the problem, eliminating the need for central model transfer, by proposing an approach based on multi-scale representation learning, coupled with a Knowledge Distillation technique. Finally, inspired by the current trend, it modifies a contrastive learning technique combining existing knowledge with previous states, aiming to preserve previously learned knowledge while incorporating new knowledge. Thorough experimentation has been conducted to provide a comprehensive analysis of the issue at hand, highlighting the great potential of the proposed method, achieving great results in a federated environment with reduced communication cost and a robust performance within highly distributed incremental scenarios.
Quantum Interference of Identical Photons from Remote GaAs Quantum Dots

We present near-unity two-photon interference visibilities for single-photon states generated from two separate GaAs quantum dots. This level of interference visibility is the first of its kind among solid-state quantum emitters and matches the performance of pure platforms such as atoms and ions.
Gradient Estimation for Unseen Domain Risk Minimization with Pre-Trained Models

Domain generalization aims to build generalized models that perform well on unseen domains when only source domains are available for model optimization. Recent studies have shown that large-scale pre-trained models can enhance domain generalization by leveraging their generalization power. However, these pre-trained models lack target task-specific knowledge yet due to discrepancies between the pre-training objectives and the target task. Although the task-specific knowledge could be learned from source domains by fine-tuning, this hurts the generalization power of pre-trained models due to gradient bias toward the source domains. To alleviate this problem, we propose a new domain generalization method that estimates unobservable gradients that reduce potential risks in unseen domains using a large-scale pre-trained model. These estimated unobservable gradients allow the pre-trained model to learn task-specific knowledge further while preserving its generalization ability by relieving the gradient bias. Our experimental results show that our method outperforms baseline methods on DOMAINBED, a standard benchmark in domain generalization. We also provide extensive analyses to demonstrate that the pre-trained model can learn task-specific knowledge without sacrificing its generalization power.
Joint Optimization of E2E Latency, FPS, Energy, and Confidentiality for Surveillance UAV

In this paper, we investigate an unmanned aerial vehicles (UAVs) with edge computing (EC) servers for surveillance applications. We address the challenges of object detection using deep neural networks (DNNs) under constraints of limited computational capability and energy resources on UAVs. To tackle the challenges, we propose the UAV Surveillance Optimization Framework (USOF) algorithm, by employing a Lyapunov optimization framework to jointly optimize four key performance metrics: end-to-end latency, frames per second (FPS), energy consumption, and data confidentiality during computational offloading. Numerical results verify that our proposed USOF algorithm achieves a balanced performance for surveillance UAVs.
Waveform Manipulation Against DNN-based Modulation Classification Attacks

In this paper we propose a method for defending against an eavesdropper that uses a Deep Neural Network (DNN) for learning the modulation of wireless communication signals. Our method is based on manipulating the emitted waveform with the aid of a continuous time frequency-modulated (FM) obfuscating signal that is mixed with the modulated data. The resulting waveform allows a legitimate receiver (LRx) to demodulate the data but it increases the test error of a pre-trained or adversarially-trained DNN classifier at the eavesdropper. The scheme works for analog modulation and digital single carrier and multi carrier orthogonal frequency division multiplexing (OFDM) waveforms, while it can implemented in frame-based wireless protocols. The results indicate that careful selection of the parameters of the obfuscating waveform can drop classification performance at the eavesdropper to less than 10% in AWGN and fading channels with no performance loss at the LRx.
Multimodal Neurons in Pretrained Text-Only Transformers

Language models demonstrate remarkable capacity to generalize representations learned in one modality to downstream tasks in other modalities. Can we trace this ability to individual neurons? We study the case where a frozen text transformer is augmented with vision using a self-supervised visual encoder and a single linear projection learned on an image-to-text task. Outputs of the projection layer are not immediately decodable into language describing image content; instead, we find that translation between modalities occurs deeper within the transformer. We introduce a procedure for identifying "multimodal neurons" that convert visual representations into corresponding text, and decoding the concepts they inject into the model’s residual stream. In a series of experiments, we show that multimodal neurons operate on specific visual concepts across inputs, and have a systematic causal effect on image captioning. Project page: mmns.csail.mit.edu