Reviewerly/exHarmony huggingface dataset:
Softmax policy gradient methods can take exponential time to converge

Abstract The softmax policy gradient (PG) method, which performs gradient ascent under softmax policy parameterization, is arguably one of the de facto implementations of policy optimization in modern reinforcement learning. For $$\gamma $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>γ</mml:mi></mml:math> -discounted infinite-horizon tabular Markov decision processes (MDPs), remarkable progress has recently been achieved towards establishing global convergence of softmax PG methods in finding a near-optimal policy. However, prior results fall short of delineating clear dependencies of convergence rates on salient parameters such as the cardinality of the state space $${\mathcal {S}}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>S</mml:mi></mml:math> and the effective horizon $$\frac{1}{1-\gamma }$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mfrac><mml:mn>1</mml:mn><mml:mrow><mml:mn>1</mml:mn><mml:mo>-</mml:mo><mml:mi>γ</mml:mi></mml:mrow></mml:mfrac></mml:math> , both of which could be excessively large. In this paper, we deliver a pessimistic message regarding the iteration complexity of softmax PG methods, despite assuming access to exact gradient computation. Specifically, we demonstrate that the softmax PG method with stepsize $$\eta $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>η</mml:mi></mml:math> can take $$\begin{aligned} \frac{1}{\eta } |{\mathcal {S}}|^{2^{\Omega \big (\frac{1}{1-\gamma }\big )}} ~\text {iterations} \end{aligned}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mtable><mml:mtr><mml:mtd><mml:mrow><mml:mfrac><mml:mn>1</mml:mn><mml:mi>η</mml:mi></mml:mfrac><mml:msup><mml:mrow><mml:mo>|</mml:mo><mml:mi>S</mml:mi><mml:mo>|</mml:mo></mml:mrow><mml:msup><mml:mn>2</mml:mn><mml:mrow><mml:mi>Ω</mml:mi><mml:mrow><mml:mo>(</mml:mo></mml:mrow><mml:mfrac><mml:mn>1</mml:mn><mml:mrow><mml:mn>1</mml:mn><mml:mo>-</mml:mo><mml:mi>γ</mml:mi></mml:mrow></mml:mfrac><mml:mrow><mml:mo>)</mml:mo></mml:mrow></mml:mrow></mml:msup></mml:msup><mml:mspace /><mml:mtext>iterations</mml:mtext></mml:mrow></mml:mtd></mml:mtr></mml:mtable></mml:mrow></mml:math> to converge, even in the presence of a benign policy initialization and an initial state distribution amenable to exploration (so that the distribution mismatch coefficient is not exceedingly large). This is accomplished by characterizing the algorithmic dynamics over a carefully-constructed MDP containing only three actions. Our exponential lower bound hints at the necessity of carefully adjusting update rules or enforcing proper regularization in accelerating PG methods.
Evaluation of the Influence of Interface Symbols on User Hand-Gestures in Augmented Reality

Effective, efficient and satisfactory human-computer interface is an important research content of natural human-computer interaction in augmented reality. Augmented reality (AR) interface requires users to face both real and virtual interface, which leads to information overload and cognitive burden. Without properly designed visual and related information, users will misunderstand the intention of interaction, resulting in misoperation of gesture input. This work classifies symbolic elements and spatial locations by studying visual interfaces in the context of augmented reality. Using user heuristic analysis to summarize the existing common gestures. The task completion time and PSSUQ scale were used to evaluate the effect of user interface on gesture interaction, and then to improve the augmented reality interface. This research is helpful to reduce the visual buttons in the augmented reality human-computer interface, improve the visual interface of gesture interaction and reduce the cognitive burden. Application of augmented reality in education and training.
Unsupervised Visual Attention and Invariance for Reinforcement Learning

Vision-based reinforcement learning (RL) is successful, but how to generalize it to unknown test environments remains challenging. Existing methods focus on training an RL policy that is universal to changing visual domains, whereas we focus on extracting visual foreground that is universal, feeding clean invariant vision to the RL policy learner. Our method is completely unsupervised, without manual annotations or access to environment internals.Given videos of actions in a training environment, we learn how to extract foregrounds with unsupervised keypoint detection, followed by unsupervised visual attention to automatically generate a foreground mask per video frame. We can then introduce artificial distractors and train a model to reconstruct the clean foreground mask from noisy observations. Only this learned model is needed during test to provide distraction-free visual input to the RL policy learner.Our Visual Attention and Invariance (VAI) method significantly outperforms the state-of-the-art on visual domain generalization, gaining 15~49% (61~229%) more cumulative rewards per episode on DeepMind Control (our DrawerWorld Manipulation) benchmarks. Our results demonstrate that it is not only possible to learn domain-invariant vision without any supervision, but freeing RL from visual distractions also makes the policy more focused and thus far better.
News Popularity Prediction Using Topic Modelling

The tasks of Natural Language Processing (hereinafter - NLP) are very topical today, in particular in the context of the application of machine learning and artificial intelligence models. In this work the approach allowing to carry out classification of texts with the minimal required manual markup is considered. The problem of prediction of resonance of news in the Kazakhstani media space where resonance is understood as abnormal interest of the public to the publication within the limits of a concrete information source is considered. The proposed model is implemented and tested within an informational system.
Anomalous Detection System in Crowded Environment using Deep Learning

In recent years, surveillance systems have become very important due to security concerns. These systems are widely used in many applications such as airports, railway stations, shopping malls, crowded sports arenas, military etc., [1]. The wide deployment of surveillance systems has made the detection of anomalous behavioral patterns in video streams to become increasingly important. An anomalous event can be considered as a deviation from the regular scene; however, the distribution of normal and anomalous events is severely imbalanced, since the anomalous behavior events do not frequently occur, hence it is imperative to accurately detect anomalous behavioral pattern from a normal pattern in a surveillance system. This paper proposes a Convolutional Neural Network and Long Short-Term Memory (CNN-LSTM) technique. The CNN is used to extract the features from the image frames and the LSTM is used as a mechanism for remembrance to make quick and accurate detection. Experiments are done on the University of California San Diego dataset using the proposed anomalous behavioral pattern detection system. Compared with other existing methods, experimental analysis demonstrates that CNN-LSTM technique has high accuracy with better parameters tuning. Different analyses were conducted using the publicly available dataset repository that has been used by many researchers in the field of computer vision in the detection of anomalous behavior. The results obtained show that CNN-LSTM outperforms the others with overall F1-score of 0.94; AUC of 0.891 and accuracy of 89%. This result shows that the deployment of the proposed technique in a surveillance detection system can assist the security personnel to detect an anomalous behavioral pattern in a crowded environment.
“Alexa, What is That Sound?” A Video Analysis of Child-Agent Communication From Two Amazon Alexa Games

The rapid adoption of smart home speakers into households has enabled young children to verbally communicate with voice assistants (VAs). VAs, such as Amazon Alexa, allow children to use their voice to play various game-based activities and offer opportunities to understand children's communication during child-agent communication. This qualitative study describes findings from video analysis of young children between three and six years old who engaged in voice-based gameplay with two commercial Alexa Skills, Animal Sounds and Animal Game. We report findings about verbal behaviors and communication breakdowns and repairs between children and Alexa, and discuss key considerations for designing voice games for children.
Putting Trust back in IP Licensing: DLT Smart Licenses for the Internet of Things

Our proposal aims to help solving a trust problem between licensors and licensees that occurs during the active life of license agreements. We particularly focus on licensing of proprietary intellectual property (IP) that is embedded in Internet of Things (IoT) devices and services (e.g. patented technologies). To achieve this we propose to encode the logic of license agreements into smart licenses (SL). We define a SL as a `digital twin' of a licensing contract, i.e. one or more smart contracts that represent the full or relevant parts of a licensing agreement in machine readable and executable code. As SL are self enforcing, the royalty computation and execution of payments can be fully automated in a tamper free and trustworthy way. This of course, requires to employ a Distributed Ledger Technology (DLT). Such an Automated Licensing Payment System (ALPS) can thus automate an established business process and solve a longstanding trust issue in licensing markets. It renders traditional costly audits obsolete, lowers entry barriers for those who want to participate in licensing markets, and enables novel business models too complex with traditional approaches.
Small Drone Detection Using Airborne Weather Radar

As small commercial and hobby drones increase in popularity, there is a desire to monitor and avoid these drones. This paper focuses on how small drones such as the DJI Phantom 3 and DJI Inspire could be detected using existing weather radar hardware already installed in many airplanes. Garmin's GWX 80 airborne weather radar was used to demonstrate this potential. Only software modifications were made to the aviation certified hardware for the feasibility of this approach.
New and improved algorithms for unordered tree inclusion

The tree inclusion problem is, given two node-labeled trees P and T (the "pattern tree" and the "target tree"), to locate every minimal subtree in T (if any) that can be obtained by applying a sequence of node insertion operations to P. Although the ordered tree inclusion problem is solvable in polynomial time, the unordered tree inclusion problem is NP-hard. The currently fastest algorithm for the latter is a classic algorithm by Kilpeläinen and Mannila from 1995 that runs in O(d22dmn) time, where m and n are the sizes of the pattern and target trees, respectively, and d is the degree of the pattern tree. Here, we develop a new algorithm that runs in O(d2dmn2) time, improving the exponential factor from 22d to 2d by considering a particular type of ancestor-descendant relationships that is suitable for dynamic programming. We also study restricted variants of the unordered tree inclusion problem.
The Role of Interactive Visualization in Developing Stakeholders’ Understanding of Natural Resource Decision-Making: A Case Study of Te Waihora/Lake Ellesmere, Canterbury, New Zealand

The purpose of this study was to investigate whether stakeholders’ understanding of natural resource decision-making was influenced by the form that visualization of environmental changes were presented, using Te Waihora/Lake Ellesmere, New Zealand as a case study. An interactive visualization (I-Vis) tool “ElleVis,” developed by (Otinpong 2013), shows the effects of different water levels on the flora and fauna living in and around the lake. An experimental design was used in which 40 stakeholders were randomly assigned to a presentation of this information using I-Vis or a non-interactive paper form of the visualization (NI-Vis). While stakeholders in both groups significantly improved their understanding of information, stakeholders presented with information using I-Vis made a statistically significant greater gain in understanding than those presented information using NI-Vis (t(38)=2.663, p<.05). Semi-structured discussions with participants confirmed the importance stakeholders placed on the interactive nature of the visualizations. The findings inform discussions about how I-Vis tools might contribute to an understanding of environmental management situations where contested resources and/or a multiplicity of interests are involved.
Students’ emotion recognition and psychological stress during the exam

Through the identification of students’ emotions and psychological pressure during the exam, scientific and effective psychological counseling strategies can be formulated for students to improve their academic performance. At present, the methods for identifying the emotional and psychological stress of students during examinations are mostly carried out through questionnaires, but the practical effect is not obvious. This paper combines machine learning algorithms to recognize the facial features of students during the exam, and then transforms the recognition results into emotion recognition to judge student emotions, and quantifies it as the corresponding psychological pressure of students. Moreover, this paper combines the algorithm to construct the student’s emotional and psychological pressure recognition model during the exam, and constructs the basic structure and algorithm flow of the model according to actual needs. In addition, this paper designs experiments to verify the practical effects of the model. The research results show that the model constructed in this paper has obvious practical effects and timely reflects the students’ mental state problems, so it can provide a reference for subsequent teaching strategies.
Communication-Efficient Tracking of Unknown, Spatially Correlated Signals in Ad-Hoc Wireless Sensor Networks: Two Machine Learning Approaches

A low-cost machine learning (ML) algorithm is proposed and discussed for spatial tracking of unknown, correlated signals in localized, ad-hoc wireless sensor networks. Each sensor is modeled as one neuron and a selected subset of these neurons are called to identify the spatial signal. The algorithm is implemented in two phases of spatial modeling and spatial tracking. The spatial signal is modeled using its M iso-contour lines at levels {ℓj}j=1M and those sensors that their sensor observations are in Δ margin of any of these levels report their sensor observations to the fusion center (FC) for spatial signal reconstruction. In spatial modeling phase, the number of these contour lines, their levels and a proper Δ are identified. In this phase, the algorithm may either use adaptive-weight stochastic gradient or scaled stochastic gradient method to select a proper Δ. Additive white Gaussian noise (AWGN) with zero mean is assumed along with the sensor observations. To reduce the observation noise’s effect, each sensor applies moving average filter on its observation to drastically reduce the effect of noise. The modeling performance, the cost and the convergence of the algorithm are discussed based on extensive computer simulations and reasoning. The algorithm is proposed for climate and environmental monitoring. In this paper, the percentage of wireless sensors that initiate a communication attempt is assumed as cost. The performance evaluation results show that the proposed spatial tracking approach is low-cost and can model the spatial signal over time with the same performance as that of spatial modeling.
Identifying the Optimal Refactoring Dependencies Using Heuristic Search Algorithms to Maximize Maintainability

Bad smells represent imperfection in the design of the software system and trigger the urge to refactor the source code. The quality of object-oriented software has always been a major concern for the developer team and refactoring techniques help them to focus on this aspect by transforming the code in a way such that the behavior of the software can be preserved. Rigorous research has been done in this field to improve the quality of the software using various techniques. But, one of the issues still remains unsettled, i.e. the overhead effort to refactor the code in order to yield the maximum maintainability value. In this paper, a quantitative evaluation method has been proposed to improve the maintainability value by identifying the most optimum refactoring dependencies in advance with the help of various meta-heuristic algorithms, including A * , AO * , Hill-Climbing and Greedy approaches. A comparison has been done between the maintainability values of the software used, before and after applying the proposed methodology. The results of this study show that the Greedy algorithm is the most promising algorithm amongst all the algorithms in determining the most optimum refactoring sequence resulting in 18.56% and 9.90% improvements in the maintainability values of jTDS and ArtOfIllusion projects, respectively. Further, this study would be beneficial for the software maintenance team as refactoring sequences will be available beforehand, thereby helping the team in maintaining the software with much ease to enhance the maintainability of the software. The proposed methodology will help the maintenance team to focus on a limited portion of the software due to prioritization of the classes, in turn helping them in completing their work within the budget and time constraints.
Using Augmented Reality in programming learning: A systematic mapping study

Coding skills have become the new language of communication for the tech world. At an educational level, applying the concepts and logic of programming is a complex task for the student. The investigation of this problem was carried out with the intention of knowing if there are tools that could help the student understand programming by using augmented reality technology. For this purpose, a systematic mapping study was carried out to identify, filter and classify the information through a query applied in different ways of research. As a result, 34 articles were selected and classified. The main results show that: a) programming learning is not limited in terms of the student's age; b) Augmented reality has potential advantages in programming learning; c) Due to the extensive content of the programming, applications focused on specific topics were found according to the level of studies; d) The software used in the development of AR applications, mostly uses Unity with Vuforia; and, e) Augmented reality contributes to different learning techniques and styles that improve the way information is perceived and visualized. In conclusion, augmented reality technology has proven to have positive consequences in the programming learning process, providing a starting point for the development of a tool that contributes to the programming learning based on the characteristics found and analyzed in this document.
Hierarchical Cost-Sensitive Techniques for Class Imbalance Learning

In the traditional classification, the classification of a new example is based on maximum of posterior probability (minimizing the error rate). This implicitly assumes that all misclassification errors cost equally. However, in real-world applications, such as face recognition, medical diagnosis, spam filtering, these assumptions are not reasonable, since costs between different misclassification errors can be quite different. Cost-sensitive learning has been developed for many years to solve this problem, but most of existing methods are developed for two-class cases, yet how to extend these methods and theories to multi-class cases is not straightforward. While several multiclass cost-sensitive algorithms have been proposed, they cannot leverage the minority classes efficiently since they treat each minority class individually. Aiming at addressing this issue, we first investigate multi-class cost-sensitive kernel logistic regression (mcKLR), which is derived from Bayes decision theory and cost-sensitive learning theory. Furthermore, based on the ideas of mcKLR, we also propose a hierarchical approach to cost-sensitive, and study its effectiveness in the context of kernel logistic regression, the experimental results justify the effectiveness and efficiency of our algorithm.
Application of big data and artificial intelligence approaches in diagnosis and treatment of neuropsychiatric diseases

In recent years, with the rapid development of big data and artificial intelligence, a large number of biological and clinical data have been generated and collected at an unprecedented speed and scale, which have been increasingly used in biomedical and healthcare informatics research, as well as in the field of neuropsychiatry. This paper introduces the application of big data and artificial intelligence in the field of neuropsychiatry and the related data types and algorithms frequently used in recent years. We took autism, schizophrenia, Alzheimer’s disease, epilepsy, and other neuropsychiatric disorders as examples for specific analysis, and summarized the characteristics of the application of big data and artificial intelligence in the early warning, diagnosis, and prognosis of these diseases. Then we analyzed the development trend of big data and artificial intelligence in the field of neuropsychiatry. This chapter further points out that the exchange and sharing of medical data and the improvement of data storage and analysis capability are important directions for the future development of this field.
A digital “Yellow Card” for securely recording vaccinations using Community PKI certificates

The Yellow Card or carte jaune is a paper booklet in a standard format set by the World Health Organisation in which a person's vaccinations are recorded by healthcare officials. Numerous initiatives are striving to create both digital vaccination records and new digital identities for people with little or no official documentation; i.e. “low doc” persons. Yet there is no globally agreed model for identity, nor any standardized way to establish identity. Nevertheless, field workers today are able by and large to establish the bona fides of Yellow Card holders with adequate certainty for the paper-based system to function most of the time. This paper contends that vaccinations should be digitized without introducing new identity systems, since a lack of formal identification is obviously not preventing Yellow Cards today. This paper describes a new digital Yellow Card, deployable on most regular mobile phones, in which public key certificates represent vaccinations and other credentials, vouched for by officials or field workers. The design has practical benefits for the digital engagement and privacy of low doc persons. It also shows how traditionally hierarchical public key infrastructure can be deployed without dictating identification protocols to communities, thus avoiding some of the controversies that plague this technology. The PKI security function can remain centralized while certificate issuance is decentralized, which leaves community organizations free to carry on their business as usual.