Reviewerly/exHarmony huggingface dataset:
Photonic spiking neural networks with event-driven femtojoule optoelectronic neurons based on Izhikevich-inspired model

Photonic spiking neural networks (PSNNs) potentially offer exceptionally high throughput and energy efficiency compared to their electronic neuromorphic counterparts while maintaining their benefits in terms of event-driven computing capability. While state-of-the-art PSNN designs require a continuous laser pump, this paper presents a monolithic optoelectronic PSNN hardware design consisting of an MZI mesh incoherent network and event-driven laser spiking neurons. We designed, prototyped, and experimentally demonstrated this event-driven neuron inspired by the Izhikevich model incorporating both excitatory and inhibitory optical spiking inputs and producing optical spiking outputs accordingly. The optoelectronic neurons consist of two photodetectors for excitatory and inhibitory optical spiking inputs, electrical transistors' circuits providing spiking nonlinearity, and a laser for optical spiking outputs. Additional inclusion of capacitors and resistors complete the Izhikevich-inspired optoelectronic neurons, which receive excitatory and inhibitory optical spikes as inputs from other optoelectronic neurons. We developed a detailed optoelectronic neuron model in Verilog-A and simulated the circuit-level operation of various cases with excitatory input and inhibitory input signals. The experimental results closely resemble the simulated results and demonstrate how the excitatory inputs trigger the optical spiking outputs while the inhibitory inputs suppress the outputs. The nanoscale neuron designed in our monolithic PSNN utilizes quantum impedance conversion. It shows that estimated 21.09 fJ/spike input can trigger the output from on-chip nanolasers running at a maximum of 10 Gspike/second in the neural network. Utilizing the simulated neuron model, we conducted simulations on MNIST handwritten digits recognition using fully connected (FC) and convolutional neural networks (CNN). The simulation results show 90% accuracy on unsupervised learning and 97% accuracy on a supervised modified FC neural network. The benchmark shows our PSNN can achieve 50 TOP/J energy efficiency, which corresponds to 100 × throughputs and 1000 × energy-efficiency improvements compared to state-of-art electrical neuromorphic hardware such as Loihi and NeuroGrid.
Improved air quality in China can enhance solar-power performance and accelerate carbon-neutrality targets

<h2>Summary</h2> China forecasts that a 14-fold increase in photovoltaic installations is needed to meet 2060 carbon-neutrality targets. In light of the fact that air pollution impairs photovoltaic performance, pollution control could reduce the installation requirement, but research has not yet taken into account the coeval impact of unfavorable meteorological conditions, which also impair performance. Here, we employ a coupled model to determine the impact of air-pollution control policies on China's photovoltaic power output in the presence of varying meteorological conditions between 1995 and 2019. We find that the benefits of air-pollution control introduced in 2004 were only partially offset by unfavorable meteorological conditions (primarily in Central and South China) and resulted in solar-power performance improvement of 0.9%/decade from 2008 onward. Further analysis shows that solar-power output in 2020 was 1.7% higher thanks to air-pollution control and that more stringent air-quality targets could reduce the demand for photovoltaic installed capacity needed to meet the 2060 carbon-neutrality target.
Motion-aware Contrastive Video Representation Learning via Foreground-background Merging

In light of the success of contrastive learning in the image domain, current self-supervised video representation learning methods usually employ contrastive loss to facilitate video representation learning. When naively pulling two augmented views of a video closer, the model however tends to learn the common static background as a shortcut but fails to capture the motion information, a phenomenon dubbed as background bias. Such bias makes the model suffer from weak generalization ability, leading to worse performance on downstream tasks such as action recognition. To alleviate such bias, we propose Foreground-background Merging (FAME) to deliberately compose the moving foreground region of the selected video onto the static background of others. Specifically, without any off-the-shelf detector, we extract the moving fore-ground out of background regions via the frame difference and color statistics, and shuffle the background regions among the videos. By leveraging the semantic consistency between the original clips and the fused ones, the model focuses more on the motion patterns and is debiased from the background shortcut. Extensive experiments demonstrate that FAME can effectively resist background cheating and thus achieve the state-of-the-art performance on downstream tasks across UCF101, HMDB51, and Diving48 datasets. The code and configurations are released at https://github.com/Mark12Ding/FAME.
Blind Quality Assessment for in-the-Wild Images via Hierarchical Feature Fusion Strategy

Image quality assessment (IQA) is very important for both end-users and service-providers since a high-quality image can significantly improve the user's quality of experience (QoE). Most existing blind image quality assessment (BIQA) models were developed for synthetically distorted images, however, they perform poorly on in-the-wild images, which are widely existed in various practical applications. In this paper, inspired by perceptual visual quality being affected by both low-level visual features and high-level semantic information, we propose an effective BIQA model for in-the-wild images by considering rich features extracted from the convolution neural network (CNN). Specifically, we propose a staircase structure to hierarchically integrate the features from intermediate layers of the CNN into the quality-aware feature representation, which enables the model to make full use of visual information from low-level to high-level and are more suitable for the in-the-wild IQA task. Experimental results show that the proposed model outperforms other state-of-the-art BIQA models on six in-the-wild IQA databases by a large margin. Moreover, the proposed model is flexible and can be replaced with popular CNN models to meet the various needs of practical applications.
Holistic Prototype Activation for Few-Shot Segmentation

Conventional deep CNN-based segmentation approaches have achieved satisfactory performance in recent years, however, they are essentially Big Data-driven technologies and are difficult to generalize to unseen categories. Few-shot segmentation is subsequently developed to perform pertinent operations in a low-data regime. Unfortunately, due to the training paradigm and network architecture factors, existing methods are prone to overfit the targets of base categories and yield inaccurate segmentation boundaries, which impedes the research progress to some extent. In this paper, we propose a Holistic Prototype Activation (HPA) network to alleviate these problems. Its novel designs can be summarized in three aspects: 1) A training-free scheme to derive the prior representations of base categories. 2) Prototype Activation Module (PAM) that generates reliable activation maps and well-matched query features by filtering the objects of irrelevant classes with high confidence. 3) Cross-Referenced Decoder (CRD) for interacted feature reweighting and multi-level feature aggregation. Extensive experiments on standard few-shot segmentation benchmarks (PASCAL-5 i and COCO-20 i) verify the effectiveness of our method. On top of that, the superior performance on multiple extended tasks, such as weak-label segmentation, zero-shot segmentation, and video object segmentation, also illustrates its flexibility and versatility. Our code is publicly available at https://github.com/chunbolang/HPA.
In the eye of the beholder: A survey of gaze tracking techniques

Gaze tracking estimates and tracks the user’s gaze by analyzing facial or eye features, it is an important way to realize automated vision-based interaction. This paper introduces the visual information used in gaze tracking, and discusses the commonly used gaze estimation methods and their research dynamics, including: 2D mapping-based methods, 3D model-based methods, and appearance-based methods. In this way, some key issues that need to be solved in these methods are considered, and their research trends are discussed. Their characteristics in system configuration, personal calibration, head motion, gaze accuracy and robustness are also compared. Finally, the applications of gaze tracking techniques are analyzed from various application factors and fields. This paper reviews the latest development of gaze tracking, focuses more on various gaze tracking algorithms and their existing challenges. The development trends of gaze tracking are prospected, which provides ideas for future theoretical research and practical applications.
Multi-modal knowledge graphs representation learning via multi-headed self-attention

Traditional knowledge graphs (KG) representation learning focuses on the link information between entities, and the effectiveness of learning is influenced by the complexity of KGs. Considering a multi-modal knowledge graph (MKG), due to the introduction of considerable other modal information(such as images and texts), the complexity of KGs further increases, which degrades the effectiveness of representation learning. To resolve this solve the problem, this study proposed the multi-modal knowledge graphs representation learning via multi-head self-attention (MKGRL-MS) model, which improved the effectiveness of link prediction by adding rich multi-modal information to the entity. We first generated a single-modal feature vector corresponding to each entity. Then, we used multi-headed self-attention to obtain the attention degree of different modal features of entities in the process of semantic synthesis. In this manner, we learned the multi-modal feature representation of entities. New knowledge representation is the sum of traditional knowledge representation and an entity’s multi-modal feature representation. Simultaneously, we successfully train our model on two existing models and two different datasets and verified its versatility and effectiveness on the link prediction task.
Meta-optic accelerators for object classifiers

Rapid advances in deep learning have led to paradigm shifts in a number of fields, from medical image analysis to autonomous systems. These advances, however, have resulted in digital neural networks with large computational requirements, resulting in high energy consumption and limitations in real-time decision-making when computation resources are limited. Here, we demonstrate a meta-optic-based neural network accelerator that can off-load computationally expensive convolution operations into high-speed and low-power optics. In this architecture, metasurfaces enable both spatial multiplexing and additional information channels, such as polarization, in object classification. End-to-end design is used to co-optimize the optical and digital systems, resulting in a robust classifier that achieves 93.1% accurate classification of handwriting digits and 93.8% accuracy in classifying both the digit and its polarization state. This approach could enable compact, high-speed, and low-power image and information processing systems for a wide range of applications in machine vision and artificial intelligence.
Point Cloud Quality Assessment: Dataset Construction and Learning-based No-reference Metric

Full-reference (FR) point cloud quality assessment (PCQA) has achieved impressive progress in recent years. However, in many cases, obtaining the reference point clouds is difficult, so no-reference (NR) metrics have become a research hotspot. Few researches about NR-PCQA are carried out due to the lack of a large-scale PCQA dataset. In this article, we first build a large-scale PCQA dataset named LS-PCQA, which includes 104 reference point clouds and more than 22,000 distorted samples. In the dataset, each reference point cloud is augmented with 31 types of impairments (e.g., Gaussian noise, contrast distortion, local missing, and compression loss) at 7 distortion levels. Besides, each distorted point cloud is assigned with a pseudo-quality score as its substitute of Mean Opinion Score. Inspired by the hierarchical perception system and considering the intrinsic attributes of point clouds, we propose a NR metric ResSCNN based on sparse convolutional neural network (CNN) to accurately estimate the subjective quality of point clouds. We conduct several experiments to evaluate the performance of the proposed NR metric. The results demonstrate that ResSCNN exhibits the state-of-the-art performance among all the existing NR-PCQA metrics and even outperforms some FR metrics. The dataset presented in this work will be made publicly accessible at https://smt.sjtu.edu.cn . The source code for the proposed ResSCNN can be found at https://github.com/lyp22/ResSCNN .
A GAN-based hybrid sampling method for imbalanced customer classification

a b s t r a c tClass imbalance is a critical issue in customer classification, for which a plethora of techniques have been proposed in the current body of literature.In particular, generative adversarial network (GAN)-based oversampling can capture the true data distribution of minority class samples and generate new samples, and this approach has demonstrated an outstanding ability to address class imbalance.However, GAN-based oversampling suffers from the issue of class overlap.As a result, in this work, we propose a novel a novel GAN-based hybrid sampling method.The new approach first uses GAN-based oversampling to generate the initial balanced dataset and then applies a novel adaptive neighborhood-based weighted undersampling method to remove generated instances and original majority class instances.This approach not only produces instances that fit the actual data distribution but also significantly reduces the influence of class overlap.Experimental results on artificial data and real-world customer datasets show that the proposed GAN-based hybrid sampling method has better performance than other benchmark methods with both accuracy-based and profit-based evaluation metrics.
QASMBench: A Low-Level Quantum Benchmark Suite for NISQ Evaluation and Simulation

The rapid development of quantum computing (QC) in the NISQ era urgently demands a low-level benchmark suite and insightful evaluation metrics for characterizing the properties of prototype NISQ devices, the efficiency of QC programming compilers, schedulers and assemblers, and the capability of quantum system simulators in a classical computer. In this work, we fill this gap by proposing a low-level, easy-to-use benchmark suite called QASMBench based on the OpenQASM assembly representation. It consolidates commonly used quantum routines and kernels from a variety of domains including chemistry, simulation, linear algebra, searching, optimization, arithmetic, machine learning, fault tolerance, cryptography, and so on, trading-off between generality and usability. To analyze these kernels in terms of NISQ device execution, in addition to circuit width and depth, we propose four circuit metrics including gate density, retention lifespan, measurement density, and entanglement variance, to extract more insights about the execution efficiency, the susceptibility to NISQ error, and the potential gain from machine-specific optimizations. Applications in QASMBench can be launched and verified on several NISQ platforms, including IBM-Q, Rigetti, IonQ and Quantinuum. For evaluation, we measure the execution fidelity of a subset of QASMBench applications on 12 IBM-Q machines through density matrix state tomography, comprising 25K circuit evaluations. We also compare the fidelity of executions among the IBM-Q machines, the IonQ QPU and the Rigetti Aspen M-1 system. QASMBench is released at: http://github.com/pnnl/QASMBench .
Towards Self-Adaptive Metric Learning On the Fly

Good quality similarity metrics can significantly facilitate the performance of many large-scale, real-world applications. Existing studies have proposed various solutions to learn a Mahalanobis or bilinear metric in an online fashion by either restricting distances between similar (dissimilar) pairs to be smaller (larger) than a given lower (upper) bound or requiring similar instances to be separated from dissimilar instances with a given margin. However, these linear metrics learned by leveraging fixed bounds or margins may not perform well in real-world applications, especially when data distributions are complex. We aim to address the open challenge of "Online Adaptive Metric Learning" (OAML) for learning adaptive metric functions on-the-fly. Unlike traditional online metric learning methods, OAML is significantly more challenging since the learned metric could be non-linear and the model has to be self-adaptive as more instances are observed. In this paper, we present a new online metric learning framework that attempts to tackle the challenge by learning a ANN-based metric with adaptive model complexity from a stream of constraints. In particular, we propose a novel Adaptive-Bound Triplet Loss (ABTL) to effectively utilize the input constraints, and present a novel Adaptive Hedge Update (AHU) method for online updating the model parameters. We empirically validates the effectiveness and efficacy of our framework on various applications such as real-world image classification, facial verification, and image retrieval.
Vision-based method for semantic information extraction in construction by integrating deep learning object detection and image captioning

Recently, vision-based monitoring has been widely adopted in construction management to improve crew productivity, reduce safety risks, and facilitate site planning. However, automated retrieval of semantic information (e.g., objects, activities, and interactions between objects) from construction images remains challenging due to the complex nature of construction sites. This paper proposes a novel semantic information extraction method by integrating deep learning object detection and image captioning, which aims to explore salient information from construction images or videos. In the proposed method, object detection has been employed as an encoder to extract the feature maps of construction object zones and the holistic image. The image captioning has been selected as the decoder to extract the semantic information. A post-processing method has been proposed to parse the semantic information into a graph format for better accessibility and visualization. In experiments, the proposed method has achieved the Consensus-based Image Description Evaluation (CIDEr) of 1.84. By adopting the proposed method, semantic information behind construction images can be presented to construction managers to assist their decision-making.
Rotation, Translation and Scale Invariant Sign Word Recognition Using Deep Learning

Communication between people with disabilities and people who do not understand sign language is a growing social need and can be a tedious task.One of the main functions of sign language is to communicate with each other through hand gestures.Recognition of hand gestures has become an important challenge for the recognition of sign language.There are many existing models that can produce a good accuracy, but if the model test with rotated or translated images, they may face some difficulties to make good performance accuracy.To resolve these challenges of hand gesture recognition, we proposed a Rotation, Translation and Scale-invariant sign word recognition system using a convolutional neural network (CNN).We have followed three steps in our work: rotated, translated and scaled (RTS) version dataset generation, gesture segmentation, and sign word classification.Firstly, we have enlarged a benchmark dataset of 20 sign words by making different amounts of Rotation, Translation and Scale of the original images to create the RTS version dataset.Then we have applied the gesture segmentation technique.The segmentation consists of three levels, i) Otsu Thresholding with YCbCr, ii) Morphological analysis: dilation through opening morphology and iii) Watershed algorithm.Finally, our designed CNN model has been trained to classify the hand gesture as well as the sign word.Our model has been evaluated using the twenty sign word dataset, five sign word dataset and the RTS version of these datasets.We achieved 99.30% accuracy from the twenty sign word dataset evaluation, 99.10% accuracy from the RTS version of the twenty sign word evolution, 100% accuracy from the five sign word dataset evaluation, and 98.00% accuracy from the RTS version five sign word dataset evolution.Furthermore, the influence of our model exists in competitive results with state-of-the-art methods in sign word recognition.
Attention-based random forest and contamination model

A new approach called ABRF (the attention-based random forest) and its modifications for applying the attention mechanism to the random forest (RF) for regression and classification are proposed. The main idea behind the proposed ABRF models is to assign attention weights with trainable parameters to decision trees in a specific way. The attention weights depend on the distance between an instance, which falls into a corresponding leaf of a tree, and training instances, which fall in the same leaf. This idea stems from representation of the Nadaraya-Watson kernel regression in the form of a RF. Three modifications of the general approach are proposed. The first one is based on applying the Huber's contamination model and on computing the attention weights by solving quadratic or linear optimization problems. The second and the third modifications use the gradient-based algorithms for computing an extended set of the attention trainable parameters. Numerical experiments with various regression and classification datasets illustrate the proposed method. The code implementing the approach is publicly available.
Decoding Artificial Intelligence to Achieve Diagnostic Excellence

This Viewpoint describes 3 learning methods that form the basis of many systems of diagnostic artificial intelligence systems—learning from experts, examples, and experience—and their parallels to clinicians’ existing approaches to learning.
Online Teaching in K-12 Education in the United States: A Systematic Review

The transition to fully or partially online instruction for K–12 students necessitated by the 2020 COVID-19 pandemic highlighted the current lack of understanding of practices that support K–12 student learning in online settings in emergency situations but also, more troublingly, in K–12 online teaching and learning more generally. A systematic review of literature regarding K–12 online teaching and learning in the United States was therefore conducted to begin to fill this gap and to inform the work of policy makers, researchers, teacher educators, teachers, and administrators as they negotiate the changing role of online instruction in our nation’s educational systems. The review revealed a set of contextual conditions that are foundational to student learning in K–12 online settings (prepared educators, technology access and autonomy, students’ developmental needs and abilities, and students’ self-regulated learning skills). The literature also pointed to seven pillars of instructional practice that support student learning in these settings (evidence-based course organization and design, connected learners, accessibility, supportive learning environment, individualization, active learning, and real-time assessment).