Reviewerly/exHarmony huggingface dataset:
Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks

The selection of initial parameter values for gradient-based optimization of deep neural networks is one of the most impactful hyperparameter choices in deep learning systems, affecting both convergence times and model performance. Yet despite significant empirical and theoretical analysis, relatively little has been proved about the concrete effects of different initialization schemes. In this work, we analyze the effect of initialization in deep linear networks, and provide for the first time a rigorous proof that drawing the initial weights from the orthogonal group speeds up convergence relative to the standard Gaussian initialization with iid weights. We show that for deep networks, the width needed for efficient convergence to a global minimum with orthogonal initializations is independent of the depth, whereas the width needed for efficient convergence with Gaussian initializations scales linearly in the depth. Our results demonstrate how the benefits of a good initialization can persist throughout learning, suggesting an explanation for the recent empirical successes found by initializing very deep non-linear networks according to the principle of dynamical isometry.
Neural Architecture Search Using Deep Neural Networks and Monte Carlo Tree Search

Neural Architecture Search (NAS) has shown great success in automating the design of neural networks, but the prohibitive amount of computations behind current NAS methods requires further investigations in improving the sample efficiency and the network evaluation cost to get better results in a shorter time. In this paper, we present a novel scalable Monte Carlo Tree Search (MCTS) based NAS agent, named AlphaX, to tackle these two aspects. AlphaX improves the search efficiency by adaptively balancing the exploration and exploitation at the state level, and by a Meta-Deep Neural Network (DNN) to predict network accuracies for biasing the search toward a promising region. To amortize the network evaluation cost, AlphaX accelerates MCTS rollouts with a distributed design and reduces the number of epochs in evaluating a network by transfer learning, which is guided with the tree structure in MCTS. In 12 GPU days and 1000 samples, AlphaX found an architecture that reaches 97.84% top-1 accuracy on CIFAR-10, and 75.5% top-1 accuracy on ImageNet, exceeding SOTA NAS methods in both the accuracy and sampling efficiency. Particularly, we also evaluate AlphaX on NASBench-101, a large scale NAS dataset; AlphaX is 3x and 2.8x more sample efficient than Random Search and Regularized Evolution in finding the global optimum. Finally, we show the searched architecture improves a variety of vision applications from Neural Style Transfer, to Image Captioning and Object Detection.
Asymmetrical Hierarchical Networks with Attentive Interactions for Interpretable Review-Based Recommendation

Recently, recommender systems have been able to emit substantially improved recommendations by leveraging user-provided reviews. Existing methods typically merge all reviews of a given user (item) into a long document, and then process user and item documents in the same manner. In practice, however, these two sets of reviews are notably different: users' reviews reflect a variety of items that they have bought and are hence very heterogeneous in their topics, while an item's reviews pertain only to that single item and are thus topically homogeneous. In this work, we develop a novel neural network model that properly accounts for this important difference by means of asymmetric attentive modules. The user module learns to attend to only those signals that are relevant with respect to the target item, whereas the item module learns to extract the most salient contents with regard to properties of the item. Our multi-hierarchical paradigm accounts for the fact that neither are all reviews equally useful, nor are all sentences within each review equally pertinent. Extensive experimental results on a variety of real datasets demonstrate the effectiveness of our method.
Deep Discriminative CNN with Temporal Ensembling for Ambiguously-Labeled Image Classification

In this paper, we study the problem of image classification where training images are ambiguously annotated with multiple candidate labels, among which only one is correct but is not accessible during the training phase. Due to the adopted non-deep framework and improper disambiguation strategies, traditional approaches are usually short of the representation ability and discrimination ability, so their performances are still to be improved. To remedy these two shortcomings, this paper proposes a novel approach termed “Deep Discriminative CNN” (D2CNN) with temporal ensembling. Specifically, to improve the representation ability, we innovatively employ the deep convolutional neural networks for ambiguously-labeled image classification, in which the well-known ResNet is adopted as our backbone. To enhance the discrimination ability, we design an entropy-based regularizer to maximize the margin between the potentially correct label and the unlikely ones of each image. In addition, we utilize the temporally assembled predictions of different epochs to guide the training process so that the latent groundtruth label can be confidently highlighted. This is much superior to the traditional disambiguation operations which treat all candidate labels equally and identify the hidden groundtruth label via some heuristic ways. Thorough experimental results on multiple datasets firmly demonstrate the effectiveness of our proposed D2CNN when compared with other existing state-of-the-art approaches.
Leveraging multiple features for document sentiment classification

Sentiment classification is an important research task in Natural Language Processing. To fulfill this type of classification, previous works have focused on leveraging task-specific features. However, they only notice part of the related features. Also, state-of-the-art methods based on neural networks often ignore traditional features. This paper proposes a novel text sentiment classification method that learns the representation of texts by hierarchically incorporating multiple features. More specifically, we design different representations for sentiment words according to the polarity of labeled texts and whether negation exists; we distinguish words with different part-of-speech tags; emoticons, if there are, are to optimize the word vectors obtained in the previous step; apart from word embeddings, character embeddings are also trained. We use a deep neural network to get a sentence-level representation from both word and character sequence. For documents with at least two sentences, we use a hierarchical structure and design a rule to give more weight to import sentences empirically to get a document-level representation. Experimental results on open datasets demonstrate that our method could effectively improve the sentiment classification performance compared with the basic models and state-of-the-art methods.
Parameter Selection for Linear Support Vector Regression

In linear support vector regression (SVR), the regularization and error sensitivity parameters are used to avoid overfitting the training data. A proper selection of parameters is very essential for obtaining a good model, but the search process may be complicated and time-consuming. In an earlier work by Chu et al. (2015), an effective parameter-selection procedure by using warm-start techniques to solve a sequence of optimization problems has been proposed for linear classification. We extend their techniques to linear SVR, but address some new and challenging issues. In particular, linear classification involves only the regularization parameter, but linear SVR has an extra error sensitivity parameter. We investigate the effective range of each parameter and the sequence in checking the two parameters. Based on this work, an effective tool for the selection of parameters for linear SVR has been available for public use.
A dynamic group decision making process for high number of alternatives using hesitant Fuzzy Ontologies and sentiment analysis

The high spread of Internet and social networks have completely changed the way that Group Decision Making methods are designed, developed and implemented. Experts now operate in environments where a large amount of information is available and new ideas and participants can appear at any time; this results in a dynamically changing decision environment. In this paper, a novel group decision making method for dynamic contexts with a high number of decision alternatives is presented. As the main component of the proposal, a perceptual computing scheme is used in order to extract information from the experts. In the process, sentiment analysis is used when analyzing the debate texts in order to obtain information for selecting the best alternatives on each round. Moreover, interval type-2 hesitant Fuzzy Ontologies are used in order to store the information related to alternatives. By combining interval type-2 and hesitant fuzzy sets, imprecise information can be represented in a comfortable and intuitive way within the ontology.
Blockchain-Based Distributed Firmware Update Architecture for IoT Devices

The Internet of Things (IoT) which creates a hyper-connected society is playing a major role in the 4th industrial revolution. The IoT is being leveraged across various fields of business globally and the number of IoT devices is causing serious security concerns. Since the firmware update of an IoT device is necessary for its lifecycle, secure firmware update of the IoT device is being brought as the first step in IoT security. The Internet Engineering Task Force (IETF) Software Updates for Internet of Things (SUIT) working group has started to specify a software update architecture for IoT devices. However, the current SUIT working group adopts a traditional client-server model to distribute firmware images, which potentially causes security risks. The current approach of the SUIT working group is unable to solve a targeting issue and an author-disappearing issue, which is suggested in this paper. Therefore, in this work, we introduce a distributed firmware update architecture based on the SUIT firmware update architecture applying blockchain. Our update architecture can prevent the issues we concern through the characteristics of blockchain, such as decentralization, transparency, and irreversibility. The blockchain network has registration nodes that process registration of manifest and firmware image files from authors, and retrieval nodes that process downloading manifest and firmware image files. The firmware image files are stored in a distributed file system and the hash values of firmware image chunks are stored on the blockchain with manifest files. The proposed architecture in this paper enables the irreversible downloads even in the author-disappearing state and tolerant to a single point of failure.
Mendelian evolutionary theory optimization algorithm

This study presented a new multi-species binary coded algorithm, Mendelian evolutionary theory optimization (METO), inspired by the plant genetics. This framework mainly consists of three concepts: first, the “denaturation” of DNA’s of two different species to produce the hybrid “offspring DNA”. Second, the Mendelian evolutionary theory of genetic inheritance, which explains how the dominant and recessive traits appear in two successive generations. Third, the Epimutation, through which organism resist for natural mutation. The above concepts are reconfigured in order to design the binary meta-heuristic evolutionary search technique. Based on this framework, four evolutionary operators—(1) Flipper, (2) Pollination, (3) Breeding, and (4) Epimutation—are created in the binary domain. In this paper, METO is compared with well-known evolutionary and swarm optimizers: (1) binary hybrid GA, (2) bio-geography-based optimization, (3) invasive weed optimization, (4) shuffled frog leap algorithm, (5) teaching–learning-based optimization, (6) cuckoo search, (7) bat algorithm, (8) gravitational search algorithm, (9) covariance matrix adaptation evolution strategy, (10) differential evolution, (11) firefly algorithm and (12) social learning PSO. This comparison is evaluated on 30 and 100 variables benchmark test functions, including noisy, rotated, and hybrid composite functions. Kruskal–Wallis statistical rank-based nonparametric H-test is utilized to determine the statistically significant differences between the output distributions of the optimizer, which are the result of the 100 independent runs. The statistical analysis shows that METO is a significantly better algorithm for complex and multi-modal problems with many local extremes.
ResNet Can Be Pruned 60×: Introducing Network Purification and Unused Path Removal (P-RM) after Weight Pruning

The state-of-art DNN structures involve high computation and great demand for memory storage which pose intensive challenge on DNN framework resources. To mitigate the challenges, weight pruning techniques has been studied. However, high accuracy solution for extreme structured pruning that combines different types of structured sparsity still waiting for unraveling due to the extremely reduced weights in DNN networks. In this paper, we propose a DNN framework which combines two different types of structured weight pruning (filter and column prune) by incorporating alternating direction method of multipliers (ADMM) algorithm for better prune performance. We are the first to find nonoptimality of ADMM process and unused weights in a structured pruned model, and further design an optimization framework which contains the first proposed Network Purification and Unused Path Removal algorithms which are dedicated to post-processing an structured pruned model after ADMM steps. Some high lights shows we achieve 232× compression on LeNet-5, 60× compression on ResNet-18 CIFAR-10 and over 5× compression on AlexNet. We share our models at anonymous link http://bit.ly/2VJ5ktv.
Binary rewriting without control flow recovery

Static binary rewriting has many important applications in software security and systems, such as hardening, repair, patching, instrumentation, and debugging. While many different static binary rewriting tools have been proposed, most rely on recovering control flow information from the input binary. The recovery step is necessary since the rewriting process may move instructions, meaning that the set of jump targets in the rewritten binary needs to be adjusted accordingly. Since the static recovery of control flow information is a hard problem in general, most tools rely on a set of simplifying heuristics or assumptions, such as specific compilers, specific source languages, or binary file meta information. However, the reliance on assumptions or heuristics tends to scale poorly in practice, and most state-of-the-art static binary rewriting tools cannot handle very large/complex programs such as web browsers.
Improving Listening Skills with Extensive Listening Using Podcasts and Vodcasts

The present study investigated the potential of podcasting and vodcasting technology in promoting extensive listening and improving overall L2 listening skills. Forty-nine college-level EFL students took part in this year-long study. Data, coming from listening log assignments, listening progress tests, proficiency tests and a listening log questionnaire, were both quantitatively and qualitatively analyzed and interpreted. Results show that students spent approximately one hour per week on extensive listening with podcasts or vodcasts outside the classroom. Vodcasts were found to be less commonly preferred than podcasts by students. Further, as measured by listening progress tests and proficiency tests, students were able to make significant progress in their overall listening skills by the end of the year. Additionally, students found extensive listening practice with podcasts and vodcasts highly effective not only in helping them improve as L2 listeners but also enhancing their pronunciation abilities and knowledge of words and phrases. Yet, students reported being occasionally frustrated with the pace of speech in podcasts and vodcasts. Overall, this study suggests that developing certain language skills can be boosted with digital technologies at our fingertips.
Translationese as a Language in “Multilingual” NMT

Machine translation has an undesirable propensity to produce "translationese" artifacts, which can lead to higher BLEU scores while being liked less by human raters.Motivated by this, we model translationese and original (i.e.natural) text as separate languages in a multilingual model, and pose the question: can we perform zero-shot translation between original source text and original target text?There is no data with original source and original target, so we train a sentence-level classifier to distinguish translationese from original target text, and use this classifier to tag the training data for an NMT model.Using this technique we bias the model to produce more natural outputs at test time, yielding gains in human evaluation scores on both adequacy and fluency.Additionally, we demonstrate that it is possible to bias the model to produce translationese and game the BLEU score, increasing it while decreasing human-rated quality.We analyze these outputs using metrics measuring the degree of translationese, and present an analysis of the volatility of heuristic-based train-data tagging.
Deepfakes and image manipulation: criminalisation and control

Deepfakes are a form of human image synthesis where an existing picture or image is superimposed into a video to change the identity of those depicted in the video. The technology relies on machine learning or artificial intelligence to map an existing image, usually a photo of a person's face, to transfer that image to an existing video image. The technology emerged into the latter part of 2017, and has since given rise to apps and other programmes that allow users to create their own deepfakes. We already use filters and emojis to alter images by consent, however, deepfakes are particularly problematic because they allow for production of videos that are highly convincing, taken to be a real video of the person depicted. Deepfakes provide for the manipulation of all manner of video, but particular risks include videos produced to incite political deception, voter manipulation, commercial fraud, and ‘revenge porn’. The production of deepfake ‘revenge porn’ presents as especially insidious given the ability to transfer the face of any person onto an already existing pornographic video. Harm is exacerbated where that video is then disseminated, via the internet or by social media.
Adaptive deformable convolutional network

Deformable Convolutional Networks (DCNs) are proposed to solve the inherent limited geometric transformation in CNNs, showing outstanding performance on sophisticated computer vision tasks. Though they can rule out irrelevant image content and focus on region of interest to some degree, the adaptive learning of the deformation is still limited. In this paper, we delve it from the aspects of deformable modules and deformable organizations to extend the scope of deformation ability. Concretely, on the one hand, we reformulate the deformable convolution and RoIpooling by reconsidering spatial-wise attention, channel-wise attention and spatial-channel interdependency, to improve the single convolution’s ability to focus on pertinent image contents. On the other hand, an empirical study is conducted on various and general arrangements of deformable convolutions (e.g., connection type) in DCNs. Especially on semantic segmentation, the study yields significant findings for a proper combination of deformable convolutions. To verify the effectiveness and superiority of our proposed deformable modules, we also provide extensive ablation study for them and compare them with other previous versions. With the proposed contribution, our refined Deformable ConvNets achieve state-of-the-art performance on two semantic segmentation benchmarks (PASCAL VOC 2012 and Cityscapes) and an object detection benchmark (MS COCO).
The role of entrepreneurship in different economic phases

• Innovation positively affects entrepreneurial activity mainly in expansion phases. • Institutions positively affect economic growth and business activity in all periods. • The relationship between social climate and entrepreneurial activity is positive. • Institutions affect growth directly and indirectly through entrepreneurship.
Teaching Digital Competence and Eco-Responsible Use of Technologies: Development and Validation of a Scale

The environmental impact produced by digital technologies is one of the fundamental contents to be developed by teachers of the 21st century. Different investigations raise the need to redesign education towards sustainable models and promote a critical look at digital technologies. The main objective of this manuscript is to design a valid, reliable and useful scale to measure self-perceived teacher digital competence regarding the eco-responsible use of technologies. The instrument has been designed based on a detailed analysis of the main frameworks for the development of digital competence in Europe: INTEF and DigCompEdu. A content validation process has been followed through the expert judgment method. Subsequently, its reliability and validity are estimated using structural equation modeling techniques. The results obtained guarantee the reliability and validity of the model. Therefore, the need to establish environmental teacher training plans and more awareness about the eco-responsible use of technologies is established.
<i>VATLD</i>: A <i>V</i>isual <i>A</i>nalytics System to Assess, Understand and Improve <i>T</i>raffic <i>L</i>ight <i>D</i>etection

Traffic light detection is crucial for environment perception and decision-making in autonomous driving. State-of-the-art detectors are built upon deep Convolutional Neural Networks (CNNs) and have exhibited promising performance. However, one looming concern with CNN based detectors is how to thoroughly evaluate the performance of accuracy and robustness before they can be deployed to autonomous vehicles. In this work, we propose a visual analytics system, VATLD, equipped with a disentangled representation learning and semantic adversarial learning, to assess, understand, and improve the accuracy and robustness of traffic light detectors in autonomous driving applications. The disentangled representation learning extracts data semantics to augment human cognition with human-friendly visual summarization, and the semantic adversarial learning efficiently exposes interpretable robustness risks and enables minimal human interaction for actionable insights. We also demonstrate the effectiveness of various performance improvement strategies derived from actionable insights with our visual analytics system, VATLD, and illustrate some practical implications for safety-critical applications in autonomous driving.