Reviewerly/exHarmony huggingface dataset:
Aesthetic Evaluation and Guidance for Mobile Photography

Nowadays, almost everyone can shoot photos using smart phones. However, not everyone can take good photos. We propose to use computational aesthetics to automatically teach people without photography training to take excellent photos. We present Aesthetic Dashboard: a system of rich aesthetic evaluation and guidance for mobile photography. We take 2 most used types of photos: landscapes and portraits into consideration. When people take photos in the preview mode, for landscapes, we show the overall aesthetic score and scores of 3 basic attributes: light, composition and color usage. Meanwhile, the matching scores of the 3 basic attributes of current preview to typical templates are shown, which can help users to adjust 3 basic attributes accordingly. For portraits, besides the above basic attributes, the facial appearance, the guidance of face light, body pose and the garment color are also shown to the users. This is the first system that can teach mobile users to shoot good photos in the form of aesthetic dashboard, through which, users can adjust several aesthetic attributes to take good photos easily.
Predicting Resource Availability in Local Mobile Crowd Computing Using Convolutional GRU

In mobile crowd computing (MCC), people's smart mobile devices (SMDs) are utilized as computing resources.Considering the ever-growing computing capabilities of today's SMDs, a collection of them can offer significantly high-performance computing services.In a local MCC, the SMDs are typically connected to a local Wi-Fi network.Organizations and institutions can leverage the SMDs available within the campus to form local MCCs to cater to their computing needs without any financial and operational burden.Though it offers an economical and sustainable computing solution, users' mobility poses a serious issue in the QoS of MCC.To address this, before submitting a job to an SMD, we suggest estimating that particular SMD's availability in the network until the job is finished.For this, we propose a convolutional GRU-based prediction model to assess how long an SMD is likely to be available in the network from any given point of time.For experimental purposes, we collected real users' mobility data (in-time and outtime) with respect to a Wi-Fi access point.To build the prediction model, we presented a novel feature extraction method to be applied to the time-series data.The experimental results prove that the proposed convolutional GRU model outperforms the conventional GRU model.
Poisoning Attacks on Fair Machine Learning

Both fair machine learning and adversarial learning have been extensively studied. However, attacking fair machine learning models has received less attention. In this paper, we present a framework that seeks to effectively generate poisoning samples to attack both model accuracy and algorithmic fairness. Our attacking framework can target fair machine learning models trained with a variety of group based fairness notions such as demographic parity and equalized odds. We develop three online attacks, adversarial sampling, adversarial labeling, and adversarial feature modification. All three attacks effectively and efficiently produce poisoning samples via sampling, labeling, or modifying a fraction of training data in order to reduce the test accuracy. Our framework enables attackers to flexibly adjust the attack's focus on prediction accuracy or fairness and accurately quantify the impact of each candidate point to both accuracy loss and fairness violation, thus producing effective poisoning samples. Experiments on two real datasets demonstrate the effectiveness and efficiency of our framework.
Let Me Ask You This: How Can a Voice Assistant Elicit Explicit User Feedback?

Voice assistants offer users access to an increasing variety of personalized functionalities. Researchers and engineers who build these experiences rely on various signals from users to create the machine learning models powering them. One type of signal is explicit feedback. While collecting explicit user feedback in situ via voice assistants would help improve and inspect the underlying models, from a user perspective it can be disruptive to the overall experience, and the user might not feel compelled to respond. However, careful design can help alleviate the friction in the experience. In this paper, we explore the opportunities and the design space for voice assistant explicit feedback elicitation. First, we present four usage categories of explicit feedback in situ for model evaluation and improvement, derived from interviews with machine learning practitioners. Then, using realistic scenarios generated for each category, we conducted an online study to evaluate multiple voice assistant designs. Our results reveal that when the voice assistant is introduced as a learner or a collaborator, users were more willing to respond to its request for feedback and felt less disruptive. In addition, giving users instructions on how to initiate feedback themselves can reduce the perceived disruptiveness compared to asking users for feedback directly. Based on our findings, we discuss the implications and potential future directions for designing voice assistants to elicit user feedback for personalized voice experiences.
Weakly supervised action segmentation with effective use of attention and self-attention

This paper generates human action sequences using a novel hybrid sequence-to-sequence model that outputs a sequence of actions in the chronological order of the actions being performed in the longer activity of a given video. At test time, our models are able to generate action for each frame using weak supervision. We evaluate several sequence-to-sequence models to solve this task and demonstrate that they are able to solve action segment generation tasks on three challenging action recognition datasets. We present how to use self-attention and standard attention mechanisms with known sequence-to-sequence models for weakly supervised video action segmentation. Our new architecture is effective for weakly supervised action segmentation that uses a combination of recurrent and transformer-based sequence-to-sequence models. Our architecture consists of Transformers and GRU encoders to encode temporal information and we use self-attention and standard attention during the decoding process. We introduce an effective positional weight prior to further improve action segmentation performance. Using this architecture and two types of attention along with positional weight priors, we obtain state-of-the-art results on Breakfast and 50Salads datasets for weakly supervised action segmentation.
Pedestrian Detection Algorithm Based on Improved YOLOv3_tiny

In view of the low detection accuracy of YOLOv3_tiny algorithm on small pedestrian target, a pedestrian detection method to improve YOLOv3_tiny is proposed. Firstly, the head and shoulder of the pedestrian are taken as the detection object. The K-means++ algorithm is used to improve the k-means algorithm and cluster the data set to get the anchor frame with higher accuracy. Secondly, add a target prediction layer with a resolution of 52 × 52 to the multi-scale prediction section, which improves the detection accuracy of the small target. In addition, the backbone structure of YOLOv3_tiny is light-weighted, which further compresses the model memory and reduces the deployment burden of the model while maintaining accuracy. A comparative experiment is conducted on the self-made pedestrian dataset which tags the head and shoulder. The results show that the recall of the improved algorithm is 0.8743, which is 11.78% higher than that of the original algorithm; The average accuracy is 77.69%, which is better than the classical algorithms, such as SSD, fast RCNN and Yolo v3; the model size is 26M, and the test time of single image is 0.0789s, which can achieve efficient and real-time detection.
Measuring Cognitive Status from Speech in a Smart Home Environment

The population is aging, and becoming more tech-savvy. The United Nations predicts that by 2050, one in six people in the world will be over age 65 (up from one in 11 in 2019), and this increases to one in four in Europe and Northern America. Meanwhile, the proportion of American adults over 65 who own a smartphone has risen 24 percentage points from 2013-2017, and the majority have Internet in their homes. Smart devices and smart home technology have profound potential to transform how people age, their ability to live independently in later years, and their interactions with their circle of care. Cognitive health is a key component to independence and well-being in old age, and smart homes present many opportunities to measure cognitive status in a continuous, unobtrusive manner. In this article, we focus on speech as a measurement instrument for cognitive health. Existing methods of cognitive assessment suffer from a number of limitations that could be addressed through smart home speech sensing technologies. We begin with a brief tutorial on measuring cognitive status from speech, including some pointers to useful open-source software toolboxes for the interested reader. We then present an overview of the preliminary results from pilot studies on active and passive smart home speech sensing for the measurement of cognitive health, and conclude with some recommendations and challenge statements for the next wave of work in this area, to help overcome both technical and ethical barriers to success.
Space Detumbling Robot Arm Deployment Path Planning Based on Bi-FMT* Algorithm

In order to avoid damage to service satellites and targets during space missions and improve safety and reliability, it is necessary to study how to eliminate or reduce the rotation of targets. This paper focused on a space detumbling robot and studied the space detumbling robot dynamics and robot arm deployment path planning. Firstly, a certain space detumbling robot with a 'platform + manipulator + end effector' configuration is proposed. By considering the end effector as a translational joint, the entire space detumbling robot is equivalent to a link system containing six rotating joints and three translational joints, and the detailed derivation process of the kinematic and dynamic model is presented. Then, ADAMS and MATLAB were used to simulate the model, and the MATLAB results were compared with the ADAMS results to verify the correctness of the model. After that, the robot arm deployment problem was analyzed in detail from the aspects of problem description, constraint analysis and algorithm implementation. An algorithm of robot arm deployment path planning based on the Bi-FMT* algorithm is proposed, and the effectiveness of the algorithm is verified by simulation.
Automated Multi-Modal Video Editing for Ads Video

Video advertising is one of the most effective forms of advertisements because videos are more attractive, more persuasive, and more informative than images or texts. Increasing amounts of video advertisements requires faster and more intelligent technologies for video generation. We have developed a multi-modal video editing approach that can automatically generate advertising video clips from any source videos. We conduct chronological boundary segmentation of the original video and construct a weighted directed graph to assemble different segments. Experiments on our video editing datasets validate success of the proposed method in producing compelling and consistent advertising videos.
A synchronized heterogeneous autoencoder with feature-level and label-level knowledge distillation for the recommendation

Motivated by the success of deep learning techniques, there are numerous deep learning models developed for recommender systems. User-oriented autoencoder (UAE) and item-oriented autoencoder (IAE) are two typical approaches developed for the recommendation by learning features from items or users respectively. Existing works demonstrate that IAE model outperforms UAE model for rating prediction tasks, while UAE model outperforms IAE model for top-N recommendation task. This fact motivates us to develop a new synchronized heterogeneous autoencoder (SHAE) for top-N recommendation by considering both features learned by IAE and UAE models. Especially, we develop two novel heterogeneous knowledge distillation methods in feature-level and label-level to build relations between IAE and UAE models. Compared with matrix factorization approaches, our methods are more efficient with acceptable computational complexity for the recommendation. We conduct comprehensive experiments on public datasets to compare with several state-of-the-art approaches. Experimental results demonstrate that the proposed method significantly outperforms other comparisons for top-N recommendation tasks.
Tracker Terrain Loss Part Two

Trackers on variable terrain can incur electric mismatch losses from row-to-row shading even with backtracking. Tracker terrain loss is the difference between the performance of trackers on horizontal ground and that on variable terrain. SolarFarmer was used to study tracker terrain loss by simulating the Hopewell Friends Solar power plant, which has an average 4% southwest slope. The results yielded a tracker terrain loss of −2% with standard backtracking and slope-aware backtracking completely recovered the 2% loss. By subdividing the site from one to three layouts, the tracker terrain loss decreased 0.5%. The 1-h versus 5-min input data did not significantly affect the tracker terrain loss. This study is a continuation of a previous study that prompted improvements in SolarFarmer's 3-D tracker shading algorithm. The results of this study demonstrate that SolarFarmer can now be used to calculate tracker terrain loss. A comparison of the SolarFarmer results with a separate uneven terrain model developed by DNV using PVsyst produced similar results.
Graph-Fraudster: Adversarial Attacks on Graph Neural Network-Based Vertical Federated Learning

Graph neural network (GNN) has achieved great success on graph representation learning. Challenged by large-scale private data collected from user side, GNN may not be able to reflect the excellent performance, without rich features and complete adjacent relationships. Addressing the problem, vertical federated learning (VFL) is proposed to implement local data protection through training a global model collaboratively. Consequently, for graph-structured data, it is a natural idea to construct a GNN-based VFL (GVFL) framework. However, GNN has been proven vulnerable to adversarial attacks. Whether the vulnerability will be brought into the GVFL has not been studied. This is the first study of adversarial attacks on GVFL. A novel adversarial attack method is proposed, named Graph-Fraudster. It generates adversarial perturbations based on the noise-added global node embeddings via the privacy leakage and the gradient of pairwise node. Specifically, first, Graph-Fraudster steals the global node embeddings and sets up a shadow model of the server for the attack generator. Second, noise is added into node embeddings to confuse the shadow model. Finally, the gradient of pairwise node is used to generate attacks with the guidance of noise-added node embeddings. Extensive experiments on five benchmark datasets demonstrate that Graph-Fraudster achieves the state-of-the-art attack performance compared with baselines in different GNN based GVFLs. Furthermore, Graph-Fraudster can remain a threat to GVFL even if two possible defense mechanisms are applied. In addition, some suggestions are put forward for the future work to improve the robustness of GVFL. The code and datasets can be downloaded at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/hgh0545/Graph-Fraudster</uri> .
Hierarchical Learning from Demonstrations for Long-Horizon Tasks

Although reinforcement learning (RL) has achieved great success in robotic manipulation skills learning, it is still challenging for long-horizon tasks. Combining RL with demonstrations is an effective solution. In this paper, we propose a novel hierarchical learning from demonstrations method for long-horizon tasks, which leverages (i) object-centered segmentation of demonstrations to automatically segment the teaching trajectories into episodes. (ii) a bi-level hierarchical imitation learning method with a parallel training mechanism to train the two-level policies simultaneously. Experimental results on three challenging long-horizon tasks with sparse rewards show that our proposed method significantly outperforms state-of-art approaches in terms of both sample-efficiency and success rate. Moreover, our method is the only one which achieves satisfactory performance in tasks of multi-object stack and multi-object push&stack.
LSTC: Boosting Atomic Action Detection with Long-Short-Term Context

In this paper, we place the atomic action detection problem intoa Long-Short Term Context (LSTC) to analyze how the temporalreliance among video signals affect the action detection results. Todo this, we decompose the action recognition pipeline into short-term and long-term reliance, in terms of the hypothesis that the twokinds of context are conditionally independent given the objectiveaction instance. Within our design, a local aggregation branch isutilized to gather dense and informative short-term cues, while ahigh order long-term inference branch is designed to reason theobjective action class from high-order interaction between actor andother person or person pairs. Both branches independently predictthe context-specific actions and the results are merged in the end.We demonstrate that both temporal grains are beneficial to atomicaction recognition. On the mainstream benchmarks of atomic actiondetection, our design can bring significant performance gain fromthe existing state-of-the-art pipeline.
A Solution to Multi-modal Ads Video Tagging Challenge

In this paper, we present our solution to the Multi-modal Ads Video Tagging Challenge of Tencent Advertising Algorithm Competition in ACM Multimedia 2021 Grand Challenges. We extend the baseline model by redesigning the visual feature extraction procedure and we modify the loss function to cope with sparse positive targets. Moreover, we propose Semi-supervised Learning with Negative Masking to leverage both labeled data and unlabeled data from the preliminary contest which effectively enhances the training process. We further utilize Cross-Class Relevance Learning to boost the performance. We achieve 0.8237 GAP score via model ensemble and rank the second place among all submissions in the challenge.
Morpho Evolution With Learning Using a Controller Archive as an Inheritance Mechanism

Most work in evolutionary robotics centers on evolving a controller for a fixed body plan. However, previous studies suggest that simultaneously evolving both controller and body plan could open up many interesting possibilities. However, the joint optimization of body plan and control via evolutionary processes can be challenging in rich morphological spaces. This is because offspring can have body plans that are very different from either of their parents, leading to a potential mismatch between the structure of an inherited neural controller and the new body. To address this, we propose a framework that combines an evolutionary algorithm to generate body plans and a learning algorithm to optimize the parameters of a neural controller. The topology of this controller is created once the body plan of each offspring has been generated. The key novelty of the approach is to add an external archive for storing learned controllers that map to explicit “types” of robots (where this is defined with respect to the features of the body plan). By initiating learning from a controller with an appropriate structure inherited from the archive, rather than from a randomly initialized one, we show that both the speed and magnitude of learning increase over time when compared to an approach that starts from scratch, using two tasks and three environments. The framework also provides new insights into the complex interactions between evolution and learning.
Deep-learning-based adaptive camera calibration for various defocusing degrees

Camera calibration tends to suffer from the low-quality target image acquisition, which would yield inaccurate or inadequate extracted features, resulting in imprecise or even failed parameter estimation. To address this problem, this Letter proposes a novel deep-learning-based adaptive calibration method robust to defocus and noise, which could significantly enhance the image quality and effectively improve the calibration result. Our work provides a convenient multi-quality target dataset generation strategy and introduces a multi-scale deep learning framework that successfully recovers a sharp target image from a deteriorated one. Free from capturing additional patterns or using special calibration targets, the proposed method allows for a more reliable calibration based on the poor-quality acquired images. In this study, an initial training dataset can be easily established containing only 68 images captured by a smartphone. Based on the augmented dataset, the superior performance and flexible transferable ability of the proposed method are validated on another camera in the calibration experiments.
Reservoir Based Edge Training on RF Data To Deliver Intelligent and Efficient IoT Spectrum Sensors

Current radio frequency (RF) sensors at the Edge lack the computational resources to support practical, in-situ training for intelligent spectrum monitoring. This is true for sensor data classification in general. We propose a solution via Deep Delay Loop Reservoir Computing (DLR), a processing architecture that supports general machine learning algorithms on compact mobile devices by leveraging delay-loop reservoir computing in combination with innovative electro-optical hardware. With both digital and photonic realizations of our design of the loops, DLR delivers reductions in form factor, hardware complexity and latency, compared to the State-of-the-Art (SoA). The main impact of the reservoir is to project the input data into a higher dimensional space of reservoir state vectors in order to linearly separate the input classes. Once the classes are well separated, traditionally complex, power-hungry classification models are no longer needed for the learning process. Yet, even with simple classifiers based on Ridge regression (RR), the complexity grows at least quadratically with the input size. Hence, the hardware reduction required for training on compact devices is in contradiction with the large dimension of state vectors. DLR employs a RR-based classifier to exceed the SoA accuracy, while further reducing power consumption by leveraging the architecture of parallel (split) loops. We present DLR architectures composed of multiple smaller loops whose state vectors are linearly combined to create a lower dimensional input into Ridge regression. We demonstrate the advantages of using DLR for two distinct applications: RF Specific Emitter Identification (SEI) for IoT authentication, and wireless protocol recognition for IoT situational awareness.