OpenRaiser/Intern-Atlas huggingface dataset:
Training Large-Scale News Recommenders with Pretrained Language Models in the Loop
Venue: arXiv cs.IR 2021
Year: 2021

News recommendation calls for deep insights of news articles' underlying semantics. Therefore, pretrained language models (PLMs), like BERT and RoBERTa, may substantially contribute to the recommendation quality. However, it's extremely challenging to have news recommenders trained together with such big models: the learning of news recommenders requires intensive news encoding operations, whose cost is prohibitive if PLMs are used as the news encoder. In this paper, we propose a novel framework, {SpeedyFeed}, which efficiently trains PLMs-based news recommenders of superior quality. SpeedyFeed is highlighted for its light-weighted encoding pipeline, which gives rise to three major advantages. Firstly, it makes the intermedia results fully reusable for the training workflow, which removes most of the repetitive but redundant encoding operations. Secondly, it improves the data efficiency of the training workflow, where non-informative data can be eliminated from encoding. Thirdly, it further saves the cost by leveraging simplified news encoding and compact news representation. Extensive experiments show that SpeedyFeed leads to more than 100$\times$ acceleration of the training process, which enables big models to be trained efficiently and effectively over massive user data. The well-trained PLMs-based model from SpeedyFeed demonstrates highly competitive performance, where it outperforms the state-of-the-art news recommenders with significant margins. SpeedyFeed is also a model-agnostic framework, which is potentially applicable to a wide spectrum of content-based recommender systems; therefore, the whole framework is open-sourced to facilitate the progress in related areas.
Reduced-Order Neural Network Synthesis with Robustness Guarantees
Venue: arXiv cs.LG 2021
Year: 2021

In the wake of the explosive growth in smartphones and cyberphysical systems, there has been an accelerating shift in how data is generated away from centralised data towards on-device generated data. In response, machine learning algorithms are being adapted to run locally on board, potentially hardware limited, devices to improve user privacy, reduce latency and be more energy efficient. However, our understanding of how these device orientated algorithms behave and should be trained is still fairly limited. To address this issue, a method to automatically synthesize reduced-order neural networks (having fewer neurons) approximating the input/output mapping of a larger one is introduced. The reduced-order neural network's weights and biases are generated from a convex semi-definite programme that minimises the worst-case approximation error with respect to the larger network. Worst-case bounds for this approximation error are obtained and the approach can be applied to a wide variety of neural networks architectures. What differentiates the proposed approach to existing methods for generating small neural networks, e.g. pruning, is the inclusion of the worst-case approximation error directly within the training cost function, which should add robustness. Numerical examples highlight the potential of the proposed approach. The overriding goal of this paper is to generalise recent results in the robustness analysis of neural networks to a robust synthesis problem for their weights and biases.
GradFreeBits: Gradient Free Bit Allocation for Dynamic Low Precision Neural Networks
Venue: arXiv cs.LG 2021
Year: 2021

Quantized neural networks (QNNs) are among the main approaches for deploying deep neural networks on low resource edge devices. Training QNNs using different levels of precision throughout the network (dynamic quantization) typically achieves superior trade-offs between performance and computational load. However, optimizing the different precision levels of QNNs can be complicated, as the values of the bit allocations are discrete and difficult to differentiate for. Also, adequately accounting for the dependencies between the bit allocation of different layers is not straight-forward. To meet these challenges, in this work we propose GradFreeBits: a novel joint optimization scheme for training dynamic QNNs, which alternates between gradient-based optimization for the weights, and gradient-free optimization for the bit allocation. Our method achieves better or on par performance with current state of the art low precision neural networks on CIFAR10/100 and ImageNet classification. Furthermore, our approach can be extended to a variety of other applications involving neural networks used in conjunction with parameters which are difficult to optimize for.
The CNAME of the Game: Large-scale Analysis of DNS-based Tracking Evasion
Venue: arXiv cs.CR 2021
Year: 2021

Online tracking is a whack-a-mole game between trackers who build and monetize behavioral user profiles through intrusive data collection, and anti-tracking mechanisms, deployed as a browser extension, built-in to the browser, or as a DNS resolver. As a response to pervasive and opaque online tracking, more and more users adopt anti-tracking tools to preserve their privacy. Consequently, as the information that trackers can gather on users is being curbed, some trackers are looking for ways to evade these tracking countermeasures. In this paper we report on a large-scale longitudinal evaluation of an anti-tracking evasion scheme that leverages CNAME records to include tracker resources in a same-site context, effectively bypassing anti-tracking measures that use fixed hostname-based block lists. Using historical HTTP Archive data we find that this tracking scheme is rapidly gaining traction, especially among high-traffic websites. Furthermore, we report on several privacy and security issues inherent to the technical setup of CNAME-based tracking that we detected through a combination of automated and manual analyses. We find that some trackers are using the technique against the Safari browser, which is known to include strict anti-tracking configurations. Our findings show that websites using CNAME trackers must take extra precautions to avoid leaking sensitive information to third parties.
Software startup education: gamifying growth hacking
Venue: arXiv cs.CY 2021
Year: 2021

Startups seek to create highly scalable business models. For startups, growth is thus vital. Growth hacking is a marketing strategy advocated by various startup practitioner experts. It focuses on using low cost practices while utilizing existing platforms in creative ways to gain more users for the service. Though topics related to growth hacking such as marketing on a general level have been extensively studied in the past, growth hacking as a practitioner-born topic has not seen much interesting among the academia. To both spark interest in growth hacking, and to facilitate teaching growth hacking in the academia, we present two board games intended to serve as an engaging introduction to growth hacking for students.
How do students test software units?
Venue: arXiv cs.CY 2021
Year: 2021

We gained insight into ideas and beliefs on testing of students who finished an introductory course on programming without any formal education on testing. We asked students to fill in a small survey, to do four exercises and to fill in a second survey. We interviewed eleven of these students in semi-structured interviews, to obtain more in-depth insight. The main outcome is that students do not test systematically, while most of them think they do test systematically. One of the misconceptions we found is that most students can only think of test cases based on programming code. Even if no code was provided (black-box testing), students try to come up with code to base their test cases on.
Risk Framework for Bitcoin Custody Operation with the Revault Protocol
Venue: arXiv cs.CY 2021
Year: 2021

Our contributions with this paper are twofold. First, we elucidate the methodological requirements for a risk framework of custodial operations and argue for the value of this type of risk model as complementary with cryptographic and blockchain security models. Second, we present a risk model in the form of a library of attack-trees for Revault -- an open-source custody protocol. The model can be used by organisations as a risk quantification framework for a thorough security analysis in their specific deployment context. Our work exemplifies an approach that can be used independent of which custody protocol is being considered, including complex protocols with multiple stakeholders and active defence infrastructure.
Gradeer: An Open-Source Modular Hybrid Grader
Venue: arXiv cs.CY 2021
Year: 2021

Automated assessment has been shown to greatly simplify the process of assessing students' programs. However, manual assessment still offers benefits to both students and tutors. We introduce Gradeer, a hybrid assessment tool, which allows tutors to leverage the advantages of both automated and manual assessment. The tool features a modular design, allowing new grading functionality to be added. Gradeer directly assists manual grading, by automatically loading code inspectors, running students' programs, and allowing grading to be stopped and resumed in place at a later time. We used Gradeer to assess an end of year assignment for an introductory Java programming course, and found that its hybrid approach offers several benefits.
Deep Learning for Suicide and Depression Identification with Unsupervised Label Correction
Venue: arXiv cs.LG 2021
Year: 2021

Early detection of suicidal ideation in depressed individuals can allow for adequate medical attention and support, which in many cases is life-saving. Recent NLP research focuses on classifying, from a given piece of text, if an individual is suicidal or clinically healthy. However, there have been no major attempts to differentiate between depression and suicidal ideation, which is an important clinical challenge. Due to the scarce availability of EHR data, suicide notes, or other similar verified sources, web query data has emerged as a promising alternative. Online sources, such as Reddit, allow for anonymity that prompts honest disclosure of symptoms, making it a plausible source even in a clinical setting. However, these online datasets also result in lower performance, which can be attributed to the inherent noise in web-scraped labels, which necessitates a noise-removal process. Thus, we propose SDCNL, a suicide versus depression classification method through a deep learning approach. We utilize online content from Reddit to train our algorithm, and to verify and correct noisy labels, we propose a novel unsupervised label correction method which, unlike previous work, does not require prior noise distribution information. Our extensive experimentation with multiple deep word embedding models and classifiers display the strong performance of the method in anew, challenging classification application. We make our code and dataset available at https://github.com/ayaanzhaque/SDCNL
Interleaving classical and reversible
Venue: arXiv cs.PL 2021
Year: 2021

Given a simple recursive function, we show how to extract two interacting processes from it. The two processes can be described by means of iterative programs, one of which is intrinsically reversible, in a language that, up to minor details, belongs to the core of widely used imperative programming languages. We implement the two processes as interleaving synchronous JAVA threads whose interaction is equivalent to the recursive function they are extracted from.
A Bit Better? Quantifying Information for Bandit Learning
Venue: arXiv cs.LG 2021
Year: 2021

The information ratio offers an approach to assessing the efficacy with which an agent balances between exploration and exploitation. Originally, this was defined to be the ratio between squared expected regret and the mutual information between the environment and action-observation pair, which represents a measure of information gain. Recent work has inspired consideration of alternative information measures, particularly for use in analysis of bandit learning algorithms to arrive at tighter regret bounds. We investigate whether quantification of information via such alternatives can improve the realized performance of information-directed sampling, which aims to minimize the information ratio.
Entropy under disintegrations
Venue: arXiv cs.IT 2021
Year: 2021

We consider the differential entropy of probability measures absolutely continuous with respect to a given $σ$-finite reference measure on an arbitrary measurable space. We state the asymptotic equipartition property in this general case; the result is part of the folklore but our presentation is to some extent novel. Then we study a general framework under which such entropies satisfy a chain rule: disintegrations of measures. We give an asymptotic interpretation for conditional entropies in this case. Finally, we apply our result to Haar measures in canonical relation.
Attempted Blind Constrained Descent Experiments
Venue: arXiv cs.LG 2021
Year: 2021

Blind Descent uses constrained but, guided approach to learn the weights. The probability density function is non-zero in the infinite space of the dimension (case in point: Gaussians and normal probability distribution functions). In Blind Descent paper, some of the implicit ideas involving layer by layer training and filter by filter training (with different batch sizes) were proposed as probable greedy solutions. The results of similar experiments are discussed. Octave (and proposed PyTorch variants) source code of the experiments of this paper can be found at https://github.com/PrasadNR/Attempted-Blind-Constrained-Descent-Experiments-ABCDE- . This is compared against the ABCDE derivatives of the original PyTorch source code of https://github.com/akshat57/Blind-Descent .
Combinators and the Story of Computation
Venue: arXiv cs.LO 2021
Year: 2021

We discuss the role of combinators in the development of the modern conception of computation over the course of the past century. We describe how ideas about formalism and mathematical logic led to the introduction of combinators in 1920 as an extension of the discovery of Nand as a basis for basic logic. We then discuss how combinators informed lambda calculus and symbolic computation, and their relationship to the development of practical computation. We finally describe recent views of combinators in terms of the computational universe of possible programs, and a recent approach to the fundamental theory of physics.
SLIP Walking over Rough Terrain via H-LIP Stepping and Backstepping-Barrier Function Inspired Quadratic Program
Venue: arXiv cs.RO 2021
Year: 2021

We present an advanced and novel control method to enable actuated Spring Loaded Inverted Pendulum model to walk over rough and challenging terrains. The high-level philosophy is the decoupling of the controls of the vertical and horizontal states. The vertical state is controlled via Backstepping-Barrier Function (BBF) based quadratic programs: a combination of control Lyapunov backstepping and control barrier function, both of which provide inequality constraints on the inputs. The horizontal state is stabilized via Hybrid-Linear Inverted Pendulum (H-LIP) based stepping, which has a closed-form formulation. Therefore, the implementation is computationally-efficient. We evaluate our method in simulation, which demonstrates the aSLIP walking over various terrains, including slopes, stairs, and general rough terrains with uncertainties.
Permutation-Based SGD: Is Random Optimal?
Venue: arXiv cs.LG 2021
Year: 2021

A recent line of ground-breaking results for permutation-based SGD has corroborated a widely observed phenomenon: random permutations offer faster convergence than with-replacement sampling. However, is random optimal? We show that this depends heavily on what functions we are optimizing, and the convergence gap between optimal and random permutations can vary from exponential to nonexistent. We first show that for 1-dimensional strongly convex functions, with smooth second derivatives, there exist permutations that offer exponentially faster convergence compared to random. However, for general strongly convex functions, random permutations are optimal. Finally, we show that for quadratic, strongly-convex functions, there are easy-to-construct permutations that lead to accelerated convergence compared to random. Our results suggest that a general convergence characterization of optimal permutations cannot capture the nuances of individual function classes, and can mistakenly indicate that one cannot do much better than random.
Prioritize Crowdsourced Test Reports via Deep Screenshot Understanding
Venue: arXiv cs.SE 2021
Year: 2021

Crowdsourced testing is increasingly dominant in mobile application (app) testing, but it is a great burden for app developers to inspect the incredible number of test reports. Many researches have been proposed to deal with test reports based only on texts or additionally simple image features. However, in mobile app testing, texts contained in test reports are condensed and the information is inadequate. Many screenshots are included as complements that contain much richer information beyond texts. This trend motivates us to prioritize crowdsourced test reports based on a deep screenshot understanding. In this paper, we present a novel crowdsourced test report prioritization approach, namely DeepPrior. We first represent the crowdsourced test reports with a novelly introduced feature, namely DeepFeature, that includes all the widgets along with their texts, coordinates, types, and even intents based on the deep analysis of the app screenshots, and the textual descriptions in the crowdsourced test reports. DeepFeature includes the Bug Feature, which directly describes the bugs, and the Context Feature, which depicts the thorough context of the bug. The similarity of the DeepFeature is used to represent the test reports' similarity and prioritize the crowdsourced test reports. We formally define the similarity as DeepSimilarity. We also conduct an empirical experiment to evaluate the effectiveness of the proposed technique with a large dataset group. The results show that DeepPrior is promising, and it outperforms the state-of-the-art approach with less than half the overhead.
Serial-parallel Multi-Scale Feature Fusion for Anatomy-Oriented Hand Joint Detection
Venue: arXiv cs.CV 2021
Year: 2021

Accurate hand joints detection from images is a fundamental topic which is essential for many applications in computer vision and human computer interaction. This paper presents a two stage network for hand joints detection from single unmarked image by using serial-parallel multi-scale feature fusion. In stage I, the hand regions are located by a pre-trained network, and the features of each detected hand region are extracted by a shallow spatial hand features representation module. The extracted hand features are then fed into stage II, which consists of serially connected feature extraction modules with similar structures, called "multi-scale feature fusion" (MSFF). A MSFF contains parallel multi-scale feature extraction branches, which generate initial hand joint heatmaps. The initial heatmaps are then mutually reinforced by the anatomic relationship between hand joints. The experimental results on five hand joints datasets show that the proposed network overperforms the state-of-the-art methods.
A Deep Graph Wavelet Convolutional Neural Network for Semi-supervised Node Classification
Venue: arXiv cs.LG 2021
Year: 2021

Graph convolutional neural network provides good solutions for node classification and other tasks with non-Euclidean data. There are several graph convolutional models that attempt to develop deep networks but do not cause serious over-smoothing at the same time. Considering that the wavelet transform generally has a stronger ability to extract useful information than the Fourier transform, we propose a new deep graph wavelet convolutional network (DeepGWC) for semi-supervised node classification tasks. Based on the optimized static filtering matrix parameters of vanilla graph wavelet neural networks and the combination of Fourier bases and wavelet ones, DeepGWC is constructed together with the reuse of residual connection and identity mappings in network architectures. Extensive experiments on three benchmark datasets including Cora, Citeseer, and Pubmed are conducted. The experimental results demonstrate that our DeepGWC outperforms existing graph deep models with the help of additional wavelet bases and achieves new state-of-the-art performances eventually.
An Empirical Study on Measuring the Similarity of Sentential Arguments with Language Model Domain Adaptation
Venue: arXiv cs.CL 2021
Year: 2021

Measuring the similarity between two different sentential arguments is an important task in argument mining. However, one of the challenges in this field is that the dataset must be annotated using expertise in a variety of topics, making supervised learning with labeled data expensive. In this paper, we investigated whether this problem could be alleviated through transfer learning. We first adapted a pretrained language model to a domain of interest using self-supervised learning. Then, we fine-tuned the model to a task of measuring the similarity between sentences taken from different domains. Our approach improves a correlation with human-annotated similarity scores compared to competitive baseline models on the Argument Facet Similarity dataset in an unsupervised setting. Moreover, we achieve comparable performance to a fully supervised baseline model by using only about 60% of the labeled data samples. We believe that our work suggests the possibility of a generalized argument clustering model for various argumentative topics.
Hierarchical Recurrent Neural Networks for Conditional Melody Generation with Long-term Structure
Venue: arXiv cs.SD 2021
Year: 2021

The rise of deep learning technologies has quickly advanced many fields, including that of generative music systems. There exist a number of systems that allow for the generation of good sounding short snippets, yet, these generated snippets often lack an overarching, longer-term structure. In this work, we propose CM-HRNN: a conditional melody generation model based on a hierarchical recurrent neural network. This model allows us to generate melodies with long-term structures based on given chord accompaniments. We also propose a novel, concise event-based representation to encode musical lead sheets while retaining the notes' relative position within the bar with respect to the musical meter. With this new data representation, the proposed architecture can simultaneously model the rhythmic, as well as the pitch structures in an effective way. Melodies generated by the proposed model were extensively evaluated in quantitative experiments as well as a user study to ensure the musical quality of the output as well as to evaluate if they contain repeating patterns. We also compared the system with the state-of-the-art AttentionRNN. This comparison shows that melodies generated by CM-HRNN contain more repeated patterns (i.e., higher compression ratio) and a lower tonal tension (i.e., more tonally concise). Results from our listening test indicate that CM-HRNN outperforms AttentionRNN in terms of long-term structure and overall rating.
Kinematic Control of compliant serial manipulators composed of dual-triangles
Venue: arXiv cs.RO 2021
Year: 2021

The paper focuses on the kinematics control of a compliant serial manipulator composed of a new type of dualtriangle elastic segments. Some useful optimization techniques were applied to solve the geometric redundancy problem, ensure the stability of the manipulator configurations with respect to the external forces/torques applied to the endeffector. The efficiency of the developed control algorisms is confirmed by simulation.