Reviewerly/exHarmony huggingface dataset:
An Affordance-Based Intersubjectivity Mechanism to Infer the Behaviour of Other Agents

We introduce an architecture based on interactionist principles which enables autonomous agents to infer the behavioural patterns of other agents, without prior knowledge. Previous works have shown that agents can leverage the relations between entities and sensorimotor abilities afforded by them to build a model of the environment which supports behaviours satisfying innate motivational principles. However, efficient collaborative or competitive behaviours relies on the ability to infer hidden motivational principles of other agents by observing their behaviour. In this paper, we introduce a novel architecture able to identify mobile affordances generated by other agents, infer their behavioural preferences and predict their movements. We validate our proposition though experiments showing that a predator agent can learn to infer that its prey is attracted by specific elements of the environment.
Research on Sustainable Design Evaluation of Smart Education Based on the Internet of Things

This study aims to explore the sustainable design evaluation method for Internet of Things (IoT) smart education through practical cases. With the development of information technology, IoT-based smart education has become increasingly popular. Over the past three years, the global working from home mode caused by the COVID-19 pandemic has led to a rapid rise in smart education at a faster-than-expected rate, becoming the mainstream form of education in advance. The advantages of the IoT smart education are apparent, while the disadvantages of the IoT smart education are undeniable. It is urgent to find and solve these problems. From the industrial design perspective, the essence of IoT smart education is a product-service system. This article explores the sustainable design evaluation method suitable for IoT smart education by sorting out the relevant concepts and product-service systems. It uses the new evaluation method in real smart education scenarios to discover problems and propose strategies. Through comparative research and comprehensive evaluation, this article verifies that the proposed five-dimensional sustainable design evaluation method helps to comprehensively examine the product-service system design of IoT smart education from multiple dimensions of sustainability to better promote the sustainable development of education.
What You See is what You Grasp: User-Friendly Grasping Guided by Near-eye-tracking

This study introduces an advanced human-robot interface designed to discern and execute manipulation tasks based solely on visual cues. The interface combines eye-tracking technology and robotic manipulation, facilitating actions like grasping or pick-and-place tasks. We have developed a head-mounted device for tracking eye movements, allowing the system to determine the user's focus and initiate sight-driven manipulation. Enhancing grasping efficiency, the system incorporates a transformer-based model, utilizing attention blocks for feature extraction and optimizing both channel capacity and spatial resolution of the feature maps. Our experiments confirm the system's capability in aiding users to perform tasks using only their gaze, suggesting significant implications for assistive robots in helping people with upper limb disabilities or the elderly with everyday activities.
Impact of Free-carrier Nonlinearities on Silicon Microring-based Reservoir Computing

We quantify the impact of thermo-optic and free-carrier effects on time-delay reservoir computing using a silicon microring resonator. We identify pump power and frequency detuning ranges with NMSE less than 0.05 for the NARMA-10 task depending on the time constants of the two considered effects.
An Architecture for Predictive Resource Allocation in Cloud Capacity Management Using Neural Networks

Cloud service providers (CSP) offer shared environments to meet customers' fluctuating requirements. The challenge is to use mechanisms capable of optimizing the use of resources and ensuring that the performance of these services continues to comply with quality metrics and the respective Service Level Agreements (SLA). CSPs must offer autonomous mechanisms to readily promote resource scalability, while customers must be confident in the performance and costs involved. In Cloud Capacity Management, the target of research in Business-driven IT Management (BDIM), predictive approaches have already been proposed to overcome the limitations of reactive methods. Such techniques have yet to demonstrate satisfactory results. This research suggests an Autonomous Architecture for Predictive Resource Allocation in Cloud Capacity Management using Neural Networks, combining reactive and predictive approaches for resource scheduling. A deep learning-based method supported predictive resource provisioning through the SLSTM (Stacked long short-term memory) neural network. This proposal adopted an autonomic approach using the MAPE-K (Monitor-Analyze-Plan-Execute over a Knowledge base) loop. A comparison between statistical methods, classical neural networks LSTM (Long short-term memory), and SLSTM demonstrated the proposal's feasibility.
A Novel Fast Video Fragment Matching Algorithm for Copyright Protection

The rise of the short-form video industry marks a revolutionary change in the digital media landscape. This has opened up new opportunities for content distribution and creation but has also brought about several challenges, one of which is the difficulty of effectively protecting video copyrights. In the industrial sector, for example, the industrial digital design, manufacturing, and production process often generate a large amount of video data. If these videos are stolen, copied, or tampered with by unauthorized third parties, infringement issues can arise. To effectively monitor and protect the video copyrights of users or enterprises, this paper proposes a lightweight video clip fast matching (KMP-FVFM) algorithm. The algorithm utilizes the idea of Knuth-Morris-Pratt (KMP) algorithm, which takes into account the inter-frame fragment relationship of the query video to improve the video matching speed. At the same time, the duplicate segment locations are pinpointed in the original video. This paper also implements a plain method for video fragment matching, and the experimental results show that KMP-FVFM can effectively improve the efficiency of video fragment matching.
Analysis and Identification of Measurement Uncertainty Sources of a LED Sun Simulator with Double-Side Illumination for Bifacial PV Module Power Rating

This paper gives an understanding of measurement uncertainty sources of LED sun simulators beyond uncertainty calculation for pulsed Xenon lamp systems. Technology-specific uncertainty sources will be identified, and an uncertainty budget calculation of a single-side and a double-side LED solar simulator will be presented. Results show significant differences between the two light sources. The expanded uncertainty (k=2) for power measurements at 1000 W/m <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> (front) and 200 W/m <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> (rear) is ±8.4%. Due to the 648 individual powered LED channels of each light source, the spectral distribution, the spatial non-uniformity, and the light stability during the I-V sweep are some of the main contributors to the expanded uncertainty.
Hybrid Quantum-Classical Multilevel Approach for Maximum Cuts on Graphs

Combinatorial optimization is one of the fields where near term quantum devices are being utilized with hybrid quantum-classical algorithms to demonstrate potentially practical applications of quantum computing. One of the most well studied problems in combinatorial optimization is the Max-Cut problem. The problem is also highly relevant to quantum and other types of “post Moore” architectures due to its similarity with the Ising model and other reasons. In this paper, we introduce a scalable hybrid multilevel approach to solve large instances of Max-Cut using both classical only solvers and quantum approximate optimization algorithm (QAOA). We compare the results of our solver to existing state of the art large-scale Max-Cut solvers. We demonstrate excellent performance of both classical and hybrid quantum-classical approaches and show that using QAOA within our framework is comparable to classical approaches. Reproducibility: Our solver is publicly available at https://github.com/angone/MLMax-cut.
On the Computational Complexities of Complex-Valued Neural Networks

Complex-valued neural networks (CVNNs) are nonlinear filters used in the digital signal processing of complex-domain data. Compared with real-valued neural networks (RVNNs), CVNNs can directly handle complex-valued input and output signals due to their complex domain parameters and activation functions. With the trend toward low-power systems, computational complexity analysis has become essential for measuring an algorithm's power consumption. Therefore, this paper presents both the quantitative and asymptotic computational complexities of CVNNs. This is a crucial tool in deciding which algorithm to implement. The mathematical operations are described in terms of the number of real-valued multiplications, as these are the most demanding operations. To determine which CVNN can be implemented in a low-power system, quantitative computational complexities can be used to accurately estimate the number of floating-point operations. We have also investigated the computational complexities of CVNNs discussed in some studies presented in the literature.
Stealth Spectrum Sensing Data Falsification Attacks Affecting IoT Spectrum Monitors on the Battlefield

Resource-constrained spectrum sensors from the Internet of Battlefield Things (IoBT) monitor the frequency spectrum to communicate over unoccupied bands, intercept enemy transmissions, or decode valuable information. However, they are also susceptible to Spectrum Sensing Data Falsification (SSDF) attacks manipulating the sensing data and impacting the previous services. Detection systems based on fingerprinting and machine learning have shown promising performance while detecting existing SSDF attacks. However, novel attacks reducing their impact on sensors behaviors have not been analyzed yet. Thus, this work redesigns and reimplements seven SSDF attacks by modifying spectrum data in the sensor memory instead of at later stages in the file system. Several experiments with current intelligent detection systems demonstrated that more effort is needed from the defensive perspective since the new SSDF attacks evade their detection. In this sense, literature-based detection methods achieve less than a 0.50 True Positive Rate when detecting the new implementations of the attacks.
A Dual Perspective of Human Motion Analysis - 3D Pose Estimation and 2D Trajectory Prediction

Anticipating human motion based on given sequences is a challenging and crucial task in computer vision and machine learning, enabling machines to understand human behaviors effectively. Precise prediction of human pose and motion trajectory holds great significance for various applications, including autonomous driving, robotics, and virtual reality. This paper presents a novel approach to address the interconnected tasks of estimating human motion, represented as 3D poses or 2D trajectories, and predicting future motions using 2D images and human pose/position sequences jointly. We propose an encoder-decoder architecture that leverages Transformer networks with a self-attention mechanism, utilizing visual context features, combined with an LSTM to model human motion kinematics. Our approach demonstrates consistent and remarkable improvements over existing methods, both quantitatively and qualitatively. Extensive experiments conducted on diverse public datasets, such as GTA-IM and PROX for 3D human pose estimation, and ETH and UCY combined datasets for 2D trajectory prediction, showcase that our method substantially reduces prediction errors compared to the current state-of-the-art methods.
Compact and Low-Loss PCM-based Silicon Photonic MZIs for Photonic Neural Networks

We present an optimized Mach–Zehnder Interferometer (MZI) with phase change materials for photonic neural networks (PNNs). With 0.2 dB loss, −38 dB crosstalk, and length of 52 µm, the designed MZI significantly improves the scalability and accuracy of PNNs under loss and crosstalk.
When Layers Play the Lottery, all Tickets Win at Initialization

Pruning is a standard technique for reducing the computational cost of deep networks. Many advances in pruning leverage concepts from the Lottery Ticket Hypothesis (LTH). LTH reveals that inside a trained dense network exists sparse subnetworks (tickets) able to achieve similar accuracy (i.e., win the lottery – winning tickets). Pruning at initialization focuses on finding winning tickets without training a dense network. Studies on these concepts share the trend that subnetworks come from weight or filter pruning. In this work, we investigate LTH and pruning at initialization from the lens of layer pruning. First, we confirm the existence of winning tickets when the pruning process removes layers. Leveraged by this observation, we propose to discover these winning tickets at initialization, eliminating the requirement of heavy computational resources for training the initial (over-parameterized) dense network. Extensive experiments show that our winning tickets notably speed up the training phase and reduce up to 51% of carbon emission, an important step towards democratization and green Artificial Intelligence. Beyond computational benefits, our winning tickets exhibit robustness against adversarial and out-of-distribution examples. Finally, we show that our subnetworks easily win the lottery at initialization while tickets from filter removal (the standard structured LTH) hardly become winning tickets.
Towards Fixing Clever-Hans Predictors with Counterfactual Knowledge Distillation

This paper introduces a novel technique called counterfactual knowledge distillation (CFKD) to detect and remove reliance on confounders in deep learning models with the help of human expert feedback. Confounders are spurious features that models tend to rely on, which can result in unexpected errors in regulated or safety-critical domains. The paper highlights the benefit of CFKD in such domains and shows some advantages of counterfactual explanations over other types of explanations. We propose an experiment scheme to quantitatively evaluate the success of CFKD and different teachers that can give feedback to the model. We also introduce a new metric that is better correlated with true test performance than validation accuracy. The paper demonstrates the effectiveness of CFKD on synthetically augmented datasets and on real-world histopathological datasets.
Memory-augmented Variational Adaptation for Online Few-shot Segmentation

In this paper, we investigate online few-shot segmentation, which learns to make mask prediction for novel classes while observing samples sequentially. The main challenge in such an online scenario is the sample diversity in the sequence, resulting in models learned from previous samples that do not generalize well to future samples. To this end, we propose a memory-augmented variational adaptation network, which learns to adapt the model to each new sample that arrives sequentially. Specifically, we first introduce a contextual prototypical memory, which retains category knowledge from previous contextual information to facilitate the model adaptation to future samples. The adaptation to each new sample is then formulated as a variational Bayesian inference problem, which strives to generate sample-specific model parameters by conditioning the sample and the prototypical memory. Furthermore, we propose a feature customization module to learn sample-specific feature representation for better model adaptation to each sample in the sequence. With extensive experiments, we show that the proposed method effectively adapts to each sample from the online sample sequence, thus achieving state-of-the-art performance on both natural image and medical image datasets.
Surround the Nonlinearity: Inserting Foldable Convolutional Autoencoders to Reduce Activation Footprint

Modern deep learning architectures, while highly successful, are characterized by substantial computational and memory demands due to their large number of parameters or the storing of activations. That is why it is hard to adapt a neural network to the constraints of hardware, especially at the edge. This paper presents an investigation into a novel approach for activation compression, which we term ’Projection-based compression on channels’ or ’ProChan’. Our method involves interposing projection layers into a pretrained network around the nonlinearity, reducing the channel dimensionality through compression operations and then expanding it back. Our module is made to be then totally fused with the convolutions around it, guaranteeing no overhead, and maximum FLOPs reduction. We studied its absorption of the cost of quantization, to combine the two approaches for footprint reduction. Our findings indicate that the projections likely perform an ’adaptive stretching’ operation on the feature space, enabling the preservation of essential information when constrained by dimensional limitations. We also perform an ablation study on the different possible strategies for a stable and quick training, and analyse the interactions with different quantization paradigms, namely PACT for activations and post-training quantization (PTQ) methods for weights.
PAT: Position-Aware Transformer for Dense Multi-Label Action Detection

We present PAT, a transformer-based network that learns complex temporal co-occurrence action dependencies in a video by exploiting multi-scale temporal features. In existing methods, the self-attention mechanism in transformers loses the temporal positional information, which is essential for robust action detection. To address this issue, we (i) embed relative positional encoding in the self-attention mechanism and (ii) exploit multi-scale temporal relationships by designing a novel non-hierarchical network, in contrast to the recent transformer-based approaches that use a hierarchical structure. We argue that joining the self-attention mechanism with multiple sub-sampling processes in the hierarchical approaches results in increased loss of positional information. We evaluate the performance of our proposed approach on two challenging dense multi-label benchmark datasets, and show that PAT improves the current state-of-the-art result by 1.1% and 0.6% mAP on the Charades and MultiTHUMOS datasets, respectively, thereby achieving the new state-of-the-art mAP at 26.5% and 44.6%, respectively. We also perform extensive ablation studies to examine the impact of the different components of our proposed network.
Black-Box Attacks on Image Activity Prediction and its Natural Language Explanations

Explainable AI (XAI) methods aim to describe the decision process of deep neural networks. Early XAI methods produced visual explanations, whereas more recent techniques generate multimodal explanations that include textual information and visual representations. Visual XAI methods have been shown to be vulnerable to white-box and gray-box adversarial attacks, with an attacker having full or partial knowledge of and access to the target system. As the vulnerabilities of multimodal XAI models have not been examined, in this paper we assess for the first time the robustness to black-box attacks of the natural language explanations generated by a self-rationalizing image-based activity recognition model. We generate unrestricted, spatially variant perturbations that disrupt the association between the predictions and the corresponding explanations to mislead the model into generating unfaithful explanations. We show that we can create adversarial images that manipulate the explanations of an activity recognition model by having access only to its final output.
Adapt Your Teacher: Improving Knowledge Distillation for Exemplar-free Continual Learning

In this work, we investigate exemplar-free class incremental learning (CIL) with knowledge distillation (KD) as a regularization strategy, aiming to prevent forgetting. KD-based methods are successfully used in CIL, but they often struggle to regularize the model without access to exemplars of the training data from previous tasks. Our analysis reveals that this issue originates from substantial representation shifts in the teacher network when dealing with out-of-distribution data. This causes large errors in the KD loss component, leading to performance degradation in CIL. Inspired by recent test-time adaptation methods, we introduce Teacher Adaptation (TA), a method that concurrently updates the teacher and the main model during incremental training. Our method seamlessly integrates with KD-based CIL approaches and allows for consistent enhancement of their performance across multiple exemplar-free CIL benchmarks.
Quantized Generative Models for Solving Inverse Problems

Generative priors have been shown to be highly successful in solving inverse problems. In this paper, we consider quantized generative models i.e., the generator network weights come from a learnt finite alphabet. Quantized neural networks are efficient in terms of memory and computation. They are ideally suited for deployment in a practical setting involving low-precision hardware. In this paper, we solve non-linear inverse problems using quantized generative models. We introduce a new meta-learning framework that makes use of proximal operators and jointly optimizes the quantized weights of the generative model, parameters of the sensing network, and the latent-space representation. Experimental validation is carried out using standard datasets – MNIST, CIFAR10, SVHN, and STL10. The results show that the performance of 32-bit networks can be achieved using 4-bit networks. The performance of 1-bit networks is about 0.7 to 2 dB inferior, while saving significantly (32×) on the model size.
Two-Stage Deep Learning Segmentation for Tiny Brain Regions

Accurate segmentation of brain regions has become increasingly important in the early diagnosis of brain diseases. Widely used methods for brain region segmentation usually rely on atlases and deformations, which require manual intervention and do not focus on tiny object segmentation. To address the challenge of tiny brain regions segmentation, we propose a two-stage segmentation network based on deep learning, using both 2D and 3D convolution. We first introduce the concept of the Small Object Distribution Map (SODM), allowing the model to perform coarse-to-fine segmentation for objects of different scales. Then, a contrastive loss function is implemented to automatically mine difficult negative samples, and two attention modules are added to assist in the accurate generation of the small object distribution map. Experimental results on a dataset of 120 brain MRI demonstrate that our method outperforms existing approaches in terms of objective evaluation metrics and subjective visual effects and shows promising potential for assisting in the diagnosis of brain diseases.