Reviewerly/exHarmony huggingface dataset:
ALP-KD: Attention-Based Layer Projection for Knowledge Distillation

Knowledge distillation is considered as a training and compression strategy in which two neural networks, namely a teacher and a student, are coupled together during training. The teacher network is supposed to be a trustworthy predictor and the student tries to mimic its predictions. Usually, a student with a lighter architecture is selected so we can achieve compression and yet deliver high-quality results. In such a setting, distillation only happens for final predictions whereas the student could also benefit from teacher’s supervision for internal components. Motivated by this, we studied the problem of distillation for intermediate layers. Since there might not be a one-to-one alignment between student and teacher layers, existing techniques skip some teacher layers and only distill from a subset of them. This shortcoming directly impacts quality, so we instead propose a combinatorial technique which relies on attention. Our model fuses teacher-side information and takes each layer’s significance into consideration, then it performs distillation between combined teacher layers and those of the student. Using our technique, we distilled a 12-layer BERT (Devlin et al. 2019) into 6-, 4-, and 2-layer counterparts and evaluated them on GLUE tasks (Wang et al. 2018). Experimental results show that our combinatorial approach is able to outperform other existing techniques.
Federated Learning with Fair Averaging

Fairness has emerged as a critical problem in federated learning (FL). In this work, we identify a cause of unfairness in FL -- conflicting gradients with large differences in the magnitudes. To address this issue, we propose the federated fair averaging (FedFV) algorithm to mitigate potential conflicts among clients before averaging their gradients. We first use the cosine similarity to detect gradient conflicts, and then iteratively eliminate such conflicts by modifying both the direction and the magnitude of the gradients. We further show the theoretical foundation of FedFV to mitigate the issue conflicting gradients and converge to Pareto stationary solutions. Extensive experiments on a suite of federated datasets confirm that FedFV compares favorably against state-of-the-art methods in terms of fairness, accuracy and efficiency. The source code is available at https://github.com/WwZzz/easyFL.
Time-Domain-Multiplexed Measurement-Based Quantum Operations with 25-MHz Clock Frequency

Among various approaches toward quantum computation, measurement-based quantum computation (MBQC) multiplexed in time domain is currently a promising method for addressing the need for scalability. MBQC requires two components: cluster states and programmable measurements. With time-domain multiplexing, the former has been realized on an ultra-large-scale. The latter, however, has remained unrealized, leaving the large-scale cluster states unused. In this work, we make such a measurement system and use it to demonstrate basic quantum operations multiplexed in the time domain with 25 MHz clock frequency. We verify transformations of the input states and their nonclassicalities for single-step quantum operations and also observe multi-step quantum operations up to one hundred steps.
Paradigm Shift in Natural Language Processing

Abstract In the era of deep learning, modeling for most natural language processing (NLP) tasks has converged into several mainstream paradigms. For example, we usually adopt the sequence labeling paradigm to solve a bundle of tasks such as POS-tagging, named entity recognition (NER), and chunking, and adopt the classification paradigm to solve tasks like sentiment analysis. With the rapid progress of pre-trained language models, recent years have witnessed a rising trend of paradigm shift, which is solving one NLP task in a new paradigm by reformulating the task. The paradigm shift has achieved great success on many tasks and is becoming a promising way to improve model performance. Moreover, some of these paradigms have shown great potential to unify a large number of NLP tasks, making it possible to build a single model to handle diverse tasks. In this paper, we review such phenomenon of paradigm shifts in recent years, highlighting several paradigms that have the potential to solve different NLP tasks.
Arabic Offensive and Hate Speech Detection Using a Cross-Corpora Multi-Task Learning Model

As social media platforms offer a medium for opinion expression, social phenomena such as hatred, offensive language, racism, and all forms of verbal violence have increased spectacularly. These behaviors do not affect specific countries, groups, or communities only, extending beyond these areas into people’s everyday lives. This study investigates offensive and hate speech on Arab social media to build an accurate offensive and hate speech detection system. More precisely, we develop a classification system for determining offensive and hate speech using a multi-task learning (MTL) model built on top of a pre-trained Arabic language model. We train the MTL model on the same task using cross-corpora representing a variation in the offensive and hate context to learn global and dataset-specific contextual representations. The developed MTL model showed a significant performance and outperformed existing models in the literature on three out of four datasets for Arabic offensive and hate speech detection tasks.
Learning Implicit Sentiment in Aspect-based Sentiment Analysis with Supervised Contrastive Pre-Training

Aspect-based sentiment analysis aims to identify the sentiment polarity of a specific aspect in product reviews.We notice that about 30% of reviews do not contain obvious opinion words, but still convey clear human-aware sentiment orientation, which is known as implicit sentiment.However, recent neural networkbased approaches paid little attention to implicit sentiment entailed in the reviews.To overcome this issue, we adopt Supervised Contrastive Pre-training on large-scale sentimentannotated corpora retrieved from in-domain language resources.By aligning the representation of implicit sentiment expressions to those with the same sentiment label, the pretraining process leads to better capture of both implicit and explicit sentiment orientation towards aspects in reviews.Experimental results show that our method achieves state-of-theart performance on SemEval2014 benchmarks, and comprehensive analysis validates its effectiveness on learning implicit sentiment.
Small Data? No Problem! Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced Languages

Pretrained multilingual language models have been shown to work well on many languages for a variety of downstream NLP tasks.However, these models are known to require a lot of training data.This consequently leaves out a huge percentage of the world's languages as they are under-resourced.Furthermore, a major motivation behind these models is that lower-resource languages benefit from joint training with higher-resource languages.In this work, we challenge this assumption and present the first attempt at training a multilingual language model on only low-resource languages.We show that it is possible to train competitive multilingual language models on less than 1 GB of text.Our model, named AfriBERTa, covers 11 African languages, including the first language model for 4 of these languages.Evaluations on named entity recognition and text classification spanning 10 languages show that our model outperforms mBERT and XLM-R in several languages and is very competitive overall.Results suggest that our "small data" approach based on similar languages may sometimes work better than joint training on large datasets with high-resource languages.Code, data and models are released at https://github.com/keleog/afriberta.
Unified Authentication and Access Control for Future Mobile Communication-Based Lightweight IoT Systems Using Blockchain

The Internet of Things (IoT) is a new revolution defined by heterogeneous devices made up of intelligent, omnipresent items that are all hooked up to The internet. These devices are frequently implemented in different areas to offer innovative programs in various industrial applications, including intelligent urban, medicine, and societies. Such Internet of Things (IoT) equipment generates a large volume of private and safety information. Because IoT systems are resource-constrained in terms of operation, memory, and communication capability, safeguarding accessibility to them is a difficult task. In the blockchain concept, the majority, or even all network nodes, check the validity and accuracy of exchanged data before accepting and recording it, whether this data is related to financial transactions, measurements of a sensor, or an authentication message. In evaluating the validity of exchanged data, nodes must reach a consensus in order to perform a special action, in which case the opportunity to enter and record transactions and unreliable interactions with the system is significantly reduced. Recently, in order to share and access management of IoT devices’ information with a distributed attitude, a new authentication protocol based on blockchain has been proposed, and it is claimed that this protocol satisfies user privacy while preserving security. Today’s identification and authentication techniques have substantial shortcomings due to rapidly growing prevalence and implementation. As a result, the protection of such gadgets is critical to guarantee the program’s efficacy and safety. A decentralized authentication and access control method for lightweight IoT systems are proposed in this work and a blockchain-based system that enables identification and secures messaging with IoT nodes. The technique is built on fog information systems and the idea of a blockchain system; when contrasted to something like a blockchain-based verification system, the testing findings show that the suggested mechanism outperforms it. The authentication and verification system undergoes using the blockchain technique. Our method takes advantage of blockchain’s inherent advantages while also associated with development authentication systems. Our suggested blockchain-based approach, structure, and layout, in particular, provide for transparency, consistency, and provenance while also providing tamper-proof records. The article describes the general systems architectural style and the analysis and execution of a real scenario as just a prototype system. The authentication included give as protected prototype that can transmit data with secured protocol and achieves minimum error rate.
China's Public Health Policies in Response to COVID-19: From an “Authoritarian” Perspective

Background: China is generally regarded internationally as an “authoritarian” state. Traditional definitions have assigned many negative connotations surrounding the term of authoritarian. We realize that it might not be considered value-neutral in other countries. But authoritarian in the Chinese context emphasizes more on centralized decision making, collectivism, coordinating all activities of the nation, and public support, which is considered a value-neutral term. Therefore, it is adopted in this paper. We would like to clarify this. Authoritarian governance is considered an important mechanism for developing China's economy and solving social problems. The COVID-19 crisis is no exception. Most of the current research on crisis management and government crises focuses on advanced, democratic countries. However, the consequences of crisis management by authoritarian governments have not been fully appreciated. Although prior research has addressed authoritarian initiatives to manage crises in China, authoritarian interventions have rarely been theorized in public health emergencies. Methods: Based on a literature review and theoretical analysis, we use a descriptive and qualitative approach to assess public health policies and mechanisms from an authoritarian perspective in China. In light of the key events and intervention measures of China's government in response to COVID-19, the strategic practices of the Communist Party of China (CPC) to construct, embody, or set political goals through authoritarian intervention in public health crisis management are discussed. Results: China's government responded to the COVID-19 pandemic with a comprehensive authoritarian intervention, notably by establishing a top-down leadership mechanism, implementing a resolute lockdown, rapidly establishing square cabin hospitals, enhancing cooperation between different government departments, mobilizing a wide range of volunteer resources, enforcing the use of health codes, imposing mandatory quarantine on those returning from abroad, and implementing city-wide nucleic acid testing. These measures ensured that China was able to contain the outbreak quickly and reflect on the unique role of the Chinese authoritarian system in responding to public health crises. Conclusions: Our paper contributes to expanding the existing understanding of the relationship between crisis management and authoritarian system. China's response to COVID-19 exemplifies the unique strengths of authoritarian institutions in public health crisis management, which is a helpful and practical tool to further enhance the CPC's political legitimacy. As a socialist model of crisis management with Chinese characteristics, it may offer desirable experiences and lessons for other countries still ravaged by the epidemic.
Modeling and simulating the noisy behavior of near-term quantum computers

Noise dominates every aspect of near-term quantum computers, rendering it exceedingly difficult to carry out even small computations. In this paper we are concerned with the modelling of noise in Noisy Intermediate-Scale Quantum (NISQ) computers. We focus on three error groups that represent the main sources of noise during a computation and present quantum channels that model each source. We engineer a noise model that combines all three noise channels and simulates the evolution of the quantum computer using its calibrated error rates. We run various experiments of our model, showcasing its behaviour compared to other noise models and an IBM quantum computer. We find that our model provides a better approximation of the quantum computer's behaviour than the other models. Following this, we use a genetic algorithm to optimize the parameters used by our noise model, bringing the behaviour of the model even closer to the quantum computer. Finally, a comparison between the pre and postoptimization parameters reveals that, according to our model, certain operations can be more or less erroneous than the hardware-calibrated parameters show.
Enhanced credit card fraud detection based on attention mechanism and LSTM deep model

Abstract As credit card becomes the most popular payment mode particularly in the online sector, the fraudulent activities using credit card payment technologies are rapidly increasing as a result. For this end, it is obligatory for financial institutions to continuously improve their fraud detection systems to reduce huge losses. The purpose of this paper is to develop a novel system for credit card fraud detection based on sequential modeling of data, using attention mechanism and LSTM deep recurrent neural networks. The proposed model, compared to previous studies, considers the sequential nature of transactional data and allows the classifier to identify the most important transactions in the input sequence that predict at higher accuracy fraudulent transactions. Precisely, the robustness of our model is built by combining the strength of three sub-methods; the uniform manifold approximation and projection (UMAP) for selecting the most useful predictive features, the Long Short Term Memory (LSTM) networks for incorporating transaction sequences and the attention mechanism to enhance LSTM performances. The experimentations of our model give strong results in terms of efficiency and effectiveness.
Blockchain-Based Deep Learning to Process IoT Data Acquisition in Cognitive Data

Remote health monitoring can help prevent disease at the earlier stages. The Internet of Things (IoT) concepts have recently advanced, enabling omnipresent monitoring. Easily accessible biomarkers for neurodegenerative disorders, namely, Alzheimer’s disease (AD) are needed urgently to assist the diagnoses at its early stages. Due to the severe situations, these systems demand high-quality qualities including availability and accuracy. Deep learning algorithms are promising in such health applications when a large amount of data is available. These solutions are ideal for a distributed blockchain-based IoT system. A good Internet connection is critical to the speed of these system responses. Due to their limited processing capabilities, smart gateway devices cannot implement deep learning algorithms. In this paper, we investigate the use of blockchain-based deep neural networks for higher speed and delivery of healthcare data in a healthcare management system. The study exhibits a real-time health monitoring for classification and assesses the response time and accuracy. The deep learning model classifies the brain diseases as benign or malignant. The study takes into account three different classes to predict the brain disease as benign or malignant that includes AD, mild cognitive impairment, and normal cognitive level. The study involves a series of processing where most of the data are utilized for training these classifiers and ensemble model with a metaclassifier classifying the resultant class. The simulation is conducted to test the efficacy of the model over that of the OASIS-3 dataset, which is a longitudinal neuroimaging, cognitive, clinical, and biomarker dataset for normal aging and AD, and it is further trained and tested on the UDS dataset from ADNI. The results show that the proposed method accurately (98%) responds to the query with high speed retrieval of classified results with an increased training accuracy of 0.539 and testing accuracy of 0.559.
ICT Use, Digital Skills and Students’ Academic Performance: Exploring the Digital Divide

Information and communication technologies (ICTs) are an integral part of our environment, and their uses vary across generations and among individuals. Today’s student population is made up of “digital natives” who have grown up under the ubiquitous influence of digital technologies, and for whom the use of ICT is common and whose daily activities are structured around media use. The aim of this study is to examine the impact of ICT use and digital skills on students’ academic performance and to explore the digital divide in France. Data were collected through face-to-face questionnaires administered to 1323 students enrolled in three French universities. Principal component analysis, a non-hierarchical k-means clustering approach and multilevel ordered logistic regression were used for data analysis and provide four main findings: first, poor investment in ICT affects students’ results; second, the ICT training offered by universities has little impact on students’ results; third, student performance improves with the innovative and collaborative use of ICTs; fourth, the acquisition of digital skills increases students’ academic performance. The results show that the digital divide still exists, and this raises questions about the effectiveness of education policies in France. They suggest also that organizational change in universities is essential to enable an exploitation of ICT.
TransVOD: End-to-End Video Object Detection With Spatial-Temporal Transformers

Detection Transformer (DETR) and Deformable DETR have been proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance as previous complex hand-crafted detectors. However, their performance on Video Object Detection (VOD) has not been well explored. In this paper, we present TransVOD, the first end-to-end video object detection system based on simple yet effective spatial-temporal Transformer architectures. The first goal of this paper is to streamline the pipeline of current VOD, effectively removing the need for many hand-crafted components for feature aggregation, e.g., optical flow model, relation networks. Besides, benefited from the object query design in DETR, our method does not need post-processing methods such as Seq-NMS. In particular, we present a temporal Transformer to aggregate both the spatial object queries and the feature memories of each frame. Our temporal transformer consists of two components: Temporal Query Encoder (TQE) to fuse object queries, and Temporal Deformable Transformer Decoder (TDTD) to obtain current frame detection results. These designs boost the strong baseline deformable DETR by a significant margin (3 %-4 % mAP) on the ImageNet VID dataset. TransVOD yields comparable performances on the benchmark of ImageNet VID. Then, we present two improved versions of TransVOD including TransVOD++ and TransVOD Lite. The former fuses object-level information into object query via dynamic convolution while the latter models the entire video clips as the output to speed up the inference time. We give detailed analysis of all three models in the experiment part. In particular, our proposed TransVOD++ sets a new state-of-the-art record in terms of accuracy on ImageNet VID with 90.0 % mAP. Our proposed TransVOD Lite also achieves the best speed and accuracy trade-off with 83.7 % mAP while running at around 30 FPS on a single V100 GPU device. Code and models are available at https://github.com/SJTU-LuHe/TransVOD.
Hybrid deep CNN-SVR algorithm for solar radiation prediction problems in Queensland, Australia

This study proposes a new hybrid deep learning (DL) model, the called CSVR, for Global Solar Radiation (GSR) predictions by integrating Convolutional Neural Network (CNN) with Support Vector Regression (SVR) approach. First, the CNN algorithm is used to extract local patterns as well as common features that occur recurrently in time series data at different intervals. Then, the SVR is subsequently adopted to replace the fully connected CNN layers to predict the daily GSR time series data at six solar farms in Queensland, Australia. To develop the hybrid CSVR model, we adopt the most pertinent meteorological variables from Global Climate Model and Scientific Information for Landowners database. From a pool of Global Climate Models variables and ground-based observations, the optimal features are selected through a metaheuristic Feature Selection algorithm, an Atom Search Optimization method. The hyperparameters of the proposed CSVR are optimized by mean of the HyperOpt method, and the overall performance of the objective algorithm is benchmarked against eight alternative DL methods, and some of the other Machine Learning approaches (LSTM, DBN, RBF, BRF, MARS, WKNNR, GPML and M5TREE) methods. The results obtained shows that the proposed CSVR model can offer several predictive advantages over the alternative DL models, as well as the conventional ML models. Specifically, we note that the CSVR model recorded a root mean square error/mean absolute error ranging between ≈ 2.172–3.305 MJ m2/1.624–2.370 MJ m2 over the six tested solar farms compared to ≈ 2.514–3.879 MJ m2/1.939–2.866 MJ m2 from alternative ML and DL algorithms. Consistent with this predicted error, the correlation between the measured and the predicted GSR, including the Willmott’s, Nash-Sutcliffe’s coefficient and Legates & McCabe’s Index was relatively higher for the proposed CSVR model compared to other DL and Machine Learning methods for all of the study sites. Accordingly, this study advocates the merits of CSVR model to provide a viable alternative to accurately predict GSR for renewable energy exploitation, energy demand or other forecasting-based applications.