Reviewerly/exHarmony huggingface dataset:
Efficient Multi-Purpose Cross-Attention Based Image Alignment Block for Edge Devices

Image alignment, also known as image registration, is a critical block used in many computer vision problems. One of the key factors in alignment is efficiency, as inefficient aligners can cause significant overhead to the overall problem. In the literature, there are some blocks that appear to do the alignment operation, although most do not focus on efficiency. Therefore, an image alignment block which can both work in time and/or space and can work on edge devices would be beneficial for almost all networks dealing with multiple images. Given its wide usage and importance, we propose an efficient, cross-attention-based, multipurpose image alignment block (XABA) suitable to work within edge devices. Using cross-attention, we exploit the relationships between features extracted from images. To make cross-attention feasible for real-time image alignment problems and handle large motions, we provide a pyramidal block based cross-attention scheme. This also captures local relationships besides reducing memory requirements and number of operations. Efficient XABA models achieve real-time requirements of running above 20 FPS performance on NVIDIA Jetson Xavier with 30W power consumption compared to other powerful computers. Used as a sub-block in a larger network, XABA also improves multi-image super-resolution network performance in comparison to other alignment methods.
Neural Image Recolorization for Creative Domains

We present a self-supervised approach to recolorization of images from design-oriented domains. Our approach can recolor images based on image exemplars or target color palettes provided by a user. In contrast with previous approaches, our method can reproduce color palettes with luminance distributions that differ significantly from input, and our method is the first palette-based approach to distinguish between recolorings that match reflectance and those that match illumination, making it particularly well-suited to visualizing different aesthetic decisions in design applications. The key to our approach is first to learn latent representations for texture and color in a setting where self-supervision is especially straightforward, and then to learn a mapping to our color representation from input color palettes and scene illumination, which offers a more intuitive space for controlling and exploring recolorization.
Is synthetic voice detection research going into the right direction?

Machine Learning, and in general Artificial Intelligence approaches, brought a great advance in each and every field of Computer Science increasing accuracy levels of predictors in any known problem. Indeed, this evolution enabled the construction of effective frameworks and solutions able to be used in investigative and forensics scenarios for detection of fakes and, in general, manipulations in multimedia contents. On the other hand, can we trust these systems? Is research activity going in the right direction? Are we just taking the low-hanging fruit without taking into account many real-case-in-the-wild situations? The purpose of this paper is to raise an alert to the research community in the specific context of synthetic voice detection, where data available for training is not big enough to give sufficient trust in the techniques available in the literature. To this aim, an exploratory investigation of the most common voice spoofing dataset was carried out and it was surprisingly easy to build simple classifiers without any Deep Learning techniques. Simple considerations on bitrate were sufficient to achieve an effective detection performance.
SymDNN: Simple &amp; Effective Adversarial Robustness for Embedded Systems

We propose SymDNN, a Deep Neural Network (DNN) inference scheme, to segment an input image into small patches, replace those patches with representative symbols, and use the reconstructed image for CNN inference. This approach of deconstruction of images, and the reconstruction from cluster centroids trained on clean images, enhances robustness against adversarial attacks. The input transform used in SymDNN is learned from very large datasets, making it difficult to approximate for adaptive adversarial attacks. For example, SymDNN achieves 23% and 42% robust accuracy at L <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</inf> attack strengths of 8/255 and 4/255 respectively, against BPDA under a complete white box setting, where most input processing based defenses break completely. SymDNN is not a future-proof adversarial defense that can defend any attack, but it is one of the few readily usable defenses in resource-limited embedded systems that defends against a wide range of attacks. Our code is available at: https://github.com/swadeykgp/SymDNN.
Natural Language-Based Vehicle Retrieval with Explicit Cross-Modal Representation Learning

On the account of the explosive growth in the large-scale transportation videos, vehicle retrieval plays an important role in the public transportation security and the intelligent transport system recently. Most vehicle retrieval algorithms are vision-based and consist of vehicle re-identification and vehicle tracking. However, the performance of vision-based vehicle retrieval algorithms is constrained as the limited information provided by traffic video streams. In this paper, we propose a contrastive cross-modal vehicle retrieval solution, maximizing the value of the complementation between natural language representation and vision representation. The framework of the proposed solution includes: (1) Preprocess a source video in four ways for generating local motional semantics and global motional semantics; (2) Correspondingly, preprocess relevant description sentences in two ways, including Textual Local Instance Semantics Extraction (TLISE) and Textual Local Motional Semantics Extraction (TLMSE); (3) Use a two-stream architecture model with four visual encoders and four text encoders to extract visual features and textual embeddings; (4) Fuse visual features and textual embeddings respectively by concatenating them along the feature channel in the order of importance, and use them for retrieval. By using the proposed solution, we achieved MRR score of 33.20%, ranking the 7th place in the AI City Challenge 2022 Track 2. The code is publicly available at https://github.com/Katherinaxxx/2022AICITY_T2.
Epistemic Uncertainty-Weighted Loss for Visual Bias Mitigation

Deep neural networks are highly susceptible to learning biases in visual data. While various methods have been proposed to mitigate such bias, the majority require explicit knowledge of the biases present in the training data in order to mitigate. We argue the relevance of exploring methods which are completely ignorant of the presence of any bias, but are capable of identifying and mitigating them. Furthermore, we propose using Bayesian neural networks with an epistemic uncertainty-weighted loss function to dynamically identify potential bias in individual training samples and to weight them during training. We find a positive correlation between samples subject to bias and higher epistemic uncertainties. Finally, we show the method has potential to mitigate visual bias on a bias benchmark dataset and on a real-world face detection problem, and we consider the merits and weaknesses of our approach.
DA<sup>3</sup>: Dynamic Additive Attention Adaption for Memory-Efficient On-Device Multi-Domain Learning

Nowadays, one practical limitation of deep neural network (DNN) is its high degree of specialization to a single task or domain (e.g., one visual domain). It motivates re-searchers to develop algorithms that can adapt DNN model to multiple domains sequentially, while still performing well on the past domains, which is known as multi-domain learning. Almost all conventional methods only focus on improving accuracy with minimal parameter update, while ignoring high computing and memory cost during training, which makes it difficult to deploy multi-domain learning into more and more widely used resource-limited edge devices, like mobile phone, IoT, embedded system, etc. During our study in multi-domain training process, we observe that large memory used for activation storage is the bottleneck that largely limits the training time and cost on edge devices. To reduce training memory usage, while keeping the domain adaption accuracy performance, we propose Dynamic Additive Attention Adaption (DA <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> ), a novel memory-efficient on-device multi-domain learning method. DA <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> learns a novel additive attention adaptor module, while freezing the weights of the pre-trained backbone model for each domain. Differentiating from prior works, our proposed DA <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> module not only mitigates activation memory buffering for reducing memory usage during training, but also serves as dynamic gating mechanism to reduce the computation cost for fast inference. We validate DA <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> on multiple dataset against state-of-the-art methods, which shows great improvement in both accuracy and training time. Moreover, we deploy DA <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> into the popular NIVDIA Jetson Nano edge GPU, where the measured experimental results show our proposed DA <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> reduces the on-device training memory consumption by 5-37×, and training time by 2×, in comparison to the baseline methods (e.g., standard fine-tuning, Parallel and Series Res. adaptor, Piggyback and TinyTL).
Searching for Energy-Efficient Hybrid Adder-Convolution Neural Networks

As convolutional neural networks (CNNs) are more and more widely used in computer vision area, the energy consumption of CNNs has become the focus of researchers. For edge devices, both the battery life and the inference latency are critical and directly affect user experience. Recently, great progress has been made in the design of neural architectures and new operators. The emergence of neural architecture search technology has improved the performance of network step by step, and liberated the productivity of engineers to a certain extent. New operators, such as AdderNets, make it possible to further improve the energy efficiency of neural networks. In this paper, we explore the fusion of new adder operators and common convolution operators into state-of-the-art light-weight networks, GhostNet, to search for models with better energy efficiency and performance. Our proposed search equilibrium strategy ensures that the adder and convolution operators can be treated fairly in the search, and the resulting model achieves the same accuracy of 73.9% with GhostNet on the ImageNet dataset at an extremely low power consumption of 0.612 mJ. When keeping the same energy consumption, the accuracy reaches 74.3% which is 0.4% higher than original GhostNet.
Leveraging Unlabeled Data for Sketch-based Understanding

Sketch-based understanding is a critical component of human cognitive learning and is a primitive communication means between humans. This topic has recently attracted the interest of the computer vision community as sketching represents a powerful tool to express static objects and dynamic scenes. Unfortunately, despite its broad application domains, the current sketch-based models strongly rely on labels for supervised training, ignoring knowledge from unlabeled data, thus limiting the underlying generalization and the applicability. Therefore, we present a study about the use of unlabeled data to improve a sketch-based model. To this end, we evaluate variations of VAE and semi-supervised VAE, and present an extension of BYOL to deal with sketches. Our results show the superiority of sketch-BYOL, which outperforms other self-supervised approaches increasing the retrieval performance for known and unknown categories. Furthermore, we show how other tasks can benefit from our proposal.
Human Stools Classification for Gastrointestinal Health based on an Improved ResNet18 Model with Dual Attention Mechanism

The human stools are directly related to the health of human gastrointestinal function. Preliminary classification of the shape and colour of stools can diagnose the health status of peoples, therefore automatic recognition of stools is the current development direction of smart toilets. Due to the difficulty in identification with complex image content, this paper proposed a convolutional neural network called StoolNet to solve the current challenges. The architecture of StoolNet is based on ResNet and contains two output branches which perform colour and shape recognition, respectively. To improve the recognition performance, the dual attention mechanism was introduced into feature extraction stage. The accuracy value of our proposed model could achieve 99.7% and 94.4% for color and shape recognition on our test set, respectively. Experimental results show that, compared with other stool classification algorithms, our method possesses better capability of category discrimination on real dataset.
Semantically Grounded Visual Embeddings for Zero-Shot Learning

Zero-shot learning methods rely on fixed visual and semantic embeddings, extracted from independent vision and language models, both pre-trained for other large-scale tasks. This is a weakness of current zero-shot learning frameworks as such disjoint embeddings fail to adequately associate visual and textual information to their shared semantic content. Therefore, we propose to learn semantically grounded and enriched visual information by computing a joint image and text model with a two-stream network on a proxy task. To improve this alignment between image and textual representations, provided by attributes, we leverage ancillary captions to provide grounded semantic information. Our method, dubbed joint embeddings for zero-shot learning is evaluated on several benchmark datasets, improving the performance of existing state-of-the-art methods in both standard (+1.6% on aPY, +2.6% on FLO) and generalized (+2.1% on AWA2, +2.2% on CUB) zero-shot recognition.
Investigating the effect of training time for machine learning based photovoltaic power forecasting

The built-in capacity and therefore the importance of photovoltaic (PV) power generation is inevitably increasing in the past years resulting that it is less and less possible to turn a blind eye to the challenging problems that come with it. This article focuses on a practical issue regarding numerical weather prediction (NWP) based PV forecasting with machine learning, which is the effect of the training time. Most of the studies propose new high-level sophisticated algorithms which can reduce the errors with a given percentage, however few of them deals with practical questions regarding the training time therefore the operating time needed to reach the accuracy of physical models, and the optimal length of the training dataset is still generally unclear. This study is supposed to provide some suggestions introducing the test method and the results of the simulations that were done on 2-years data including spatial weather forecast and the power production of a Hungarian PV plant. Three fast but powerful ensemble methods, XGBoost, LightGBM and ExtraTrees regression were applied using different train length cases in the range of 2,4,6… 102 weeks, for all the 52 two-week intervals. This simulation allows to investigate not just the previously mentioned aspects, but also the seasonal difference and sensitivity for the forecasting accuracy. The results are shown in both day-ahead and intraday time horizons, where in case of latter, also some measured power values of the close-past has been added to the input variables.
LUUNU — Blockchain, MISP, Model Cards and Federated Learning Enabled Cyber Threat Intelligence Sharing Platform

Cyber Threat Intelligence (CTI) is a process of threat data collection, processing, and analysis to understand a threat actor’s motives, targets, and attack behaviors. CTI involves highly sensitive data and any inadvertent access to it can harm the affected organization’s reputation. More importantly, CTI sharing has a dangerous outcome of inadvertently revealing the security weaknesses or vulnerabilities in an organization’s infrastructure. It is important that any proposed system of CTI sharing guarantees preserving the privacy and anonymity of the partnering organizations. In this paper, we propose “Luunu,” a blockchain, MISP, Model Cards and Federated Learning enabled CTI sharing platform, to provide enhanced privacy, transparency, traceability, anonymity, and data provenance in a scalable manner. The self-sovereign identity (SSI) capability of Luunu ensures the anonymity of the participants in the CTI sharing. Further, we propose a blockchain-based federated learning system to analyze the collected CTI data from the participating organizations.
No-Reference Light Field Image Quality Assessment Method Based on a Long-Short Term Memory Neural Network

Light Field (LF) cameras capture angular and spatial information and, consequently, require a large amount of resources in memory and bandwidth. To reduce these requirements, LF contents generally need to undergo compression and transmission protocols. Since these techniques may introduce distortions, the design of Light-Field Image Quality Assessment (LFI-IQA) methods are important to monitor the quality of the LFI content at the user side. In this work, we present a No-Reference (NR) LFIIQA method that is based on a Long Short-Term Memory based Deep Neural Network (LSTM-DNN). The method is composed of two streams. The first stream extracts long-term dependent distortion related features from horizontal epipolar plane images, while the second stream processes bottleneck features of micro-lens images. The outputs of both streams are fused, and supplied to a regression operation that generates a scalar value as a predicted quality score. Results show that the proposed method is robust and accurate, outperforming several state-of-the-art LF-IQA methods.
Radix Trie improved Nahrain chaotic map-based image encryption model for effective image encryption process

As the intrinsic features of images include huge size and additional correlation prevailing among pixels, it is tedious to accomplish by using outdated models. The cryptographic features of Chaotic Maps (CM), including early criteria and random nature, aid in designing novel Image Encryption (IE) methods. In this paper, Nahrain Map with Radix Trie-based Image Encryption (NMRTIE) model is proposed to guarantee image encryption that resists known attacks that could be launched during its transmission in the cloud environment. The propounded NMRTIE model includes three essential phases: NCM-based Image Encryption (IE), Radix Trie (RT) based scrambling, and incessant diffusion. Initially, the NCM model uses more than one chaotic function to produce a sequence of keys. The RT model eases scrambling, where the rows and columns' pixels are swapped substantially. Lastly, the propounded diffusion scheme cooperates with NCM to produce key streams for remarkably hastening diffusion and dispersing effect. The propounded NMRTIE model is evaluated, and the results are confirmed based on numerous characteristics. The investigational consequence confirms that the propounded NMRTIE scheme is more appropriate than the associated approaches.
Unconventional data for policy

The paper explores the possibility to employ the source code of corporate websites as an information source for research in innovation studies. Research in this area is generally based on studies that collect data on patents or official data sources. Our paper links the standard economic information of the firm with web-based data and joins the ongoing debate with a threefold contribution. First, whereas the majority of the literature focused on the linguistic content of web-pages, we mostly use HTML tags. Second, we propose a method to assess the quality of the linkage of Web data to firm-level information. Third, we show that the data retrieved from corporate websites can aid to identify ‘innovative SMEs’.
A Cross Entropy Test Allows Quantitative Statistical Comparison of t-SNE and UMAP Representations

The advent of high-dimensional single cell data has necessitated the development of dimensionality-reduction tools. t-SNE and UMAP are the two most frequently used approaches, allowing clear visualisation of highly complex single cell datasets. Despite the ubiquity of these approaches and the clear need for quantitative comparison of single cell datasets, t-SNE and UMAP have largely remained data visualisation tools due to the lack of robust statistical approaches available. Here, we have derived a statistical test for evaluating the difference between dimensionality-reduced datasets, using the Kolmogorov-Smirnov test on the distributions of cross entropy. As the approach uses the interrelationship of single cells, the resulting statistic is robust and capable of distinguishing between true biological variation and rotational symmetry. Further, the test provides a valid distance between single cell datasets, allowing the organisation of multiple samples for quantitative comparison. These results demonstrate the potential of dimensionality-reduction tools for biomedical data analysis.
What Should Be Equivariant In Self-Supervised Learning

Self-supervised learning (SSL) aims to learn feature representation without human-annotated data. Existing methods approach this goal by encouraging the feature representations to be invariant under a set of task-irrelevant transformations and distortions defined a priori. However, multiple studies have shown that such an assumption often limits the expressive power of the representations and model would perform poorly when downstream tasks violate this assumption. For example, being invariant to rotations would prevent features from retaining enough information to estimate object rotation angles. This suggests additional manual work and domain knowledge are required for selecting augmentation types during SSL. In this work, we relax the transformation-invariance assumption by introducing a SSL framework that encourages the feature representations to preserve the order of transformation scale in embedding space for some transformations while maintaining invariance to other transformations. This allows the learned feature representations to retain information about task-relevant transformations. In addition, this framework gives rise to a handy mechanism to determine the augmentation types to which the features representations should be invariant and equivariant during SSL. We demonstrate the effectiveness of our method on various datasets such as Fruits 360, Caltech-UCSD Birds 200, and Blood cells dataset.
Adaptive Navigation Control of an Active Smart Walker

The rapid increase of elderly population all over the world has determined an enhanced attention to their needs: among the others, mobility is one of the most important human skills to support. In this work we propose a prototype of an active smart walker featuring obstacle avoidance and autonomous navigation capabilities, which is meant to assist the elderly people during the walk. The proposed system is able to plan and move in a known environment, adapting its behavior according to the presence of the user. The proposed solution has been widely tested in its different operating modes in a real setting.