OpenRaiser/Intern-Atlas huggingface dataset:
Predicting Missing Information of Vulnerability Reports
Authors: Hao Guo, Zhenchang Xing, Xiaohong Li
Venue: WWW
Year: 2020

We have found that there is a certain degree of missing information through studying the current exposure of software vulnerabilities. This problem is caused by incomplete information submitted by the vulnerability report submitter. In this paper, we extract the knowledge of software vulnerability in a fine-grained way, and design a machine learning method to complete the missing information through the other information of the vulnerability itself. Our method can predict and complete the missing types and causes information of vulnerability reports.
In Opinion Holders' Shoes: Modeling Cumulative Influence for View Change in Online Argumentation
Authors: Zhen Guo, Zhe Zhang, Munindar P. Singh
Venue: WWW
Year: 2020

Understanding how people change their views during multiparty argumentative discussions is important in applications that involve human communication, e.g., in social media and education. Existing research focuses on lexical features of individual comments, dynamics of discussions, or the personalities of participants but deemphasizes the cumulative influence of the interplay of comments by different participants on a participant’s mindset. We address the task of predicting the points where a user’s view changes given an entire discussion, thereby tackling the confusion due to multiple plausible alternatives when considering the entirety of a discussion. We make the following contributions. (1) Through a human study, we show that modeling a user’s perception of comments is crucial in predicting persuasiveness. (2) We present a sequential model for cumulative influence that captures the interplay between comments as both local and nonlocal dependencies, and demonstrate its capability of selecting the most effective information for changing views. (3) We identify contextual and interactive features and propose sequence structures to incorporate these features. Our empirical evaluation using a Reddit Change My View dataset shows that contextual and interactive features are valuable in predicting view changes, and a sequential model notably outperforms the nonsequential baseline models.
Challenges, Best Practices and Pitfalls in Evaluating Results of Online Controlled Experiments
Authors: Somit Gupta, Xiaolin Shi, Pavel A. Dmitriev, Xin Fu
Venue: WWW
Year: 2020

A/B Testing is the gold standard to estimate the causal relationship between a change in a product and its impact on key outcome measures. It is widely used in the industry to test changes ranging from simple copy change or UI change to more complex changes like using machine learning models to personalize user experience. The key aspect of A/B testing is evaluation of experiment results. Designing the right set of metrics - correct outcome measures, data quality indicators, guardrails that prevent harm to business, and a comprehensive set of supporting metrics to understand the “why” behind the key movements is the #1 challenge practitioners face when trying to scale their experimentation program 11, 14. On the technical side, improving sensitivity of experiment metrics is a hard problem and an active research area, with large practical implications as more and more small and medium size businesses are trying to adopt A/B testing and suffer from insufficient power. In this tutorial we will discuss challenges, best practices, and pitfalls in evaluating experiment results, focusing on both lessons learned and practical guidelines as well as open research questions. A version of this tutorial was also present at KDD 2019 23. It was attended by around 150 participants. This tutorial has also been accepted for the WSDM 2020 conference.
Adversarial Bandits Policy for Crawling Commercial Web Content
Authors: Shuguang Han, Michael Bendersky, Przemek Gajda, Sergey Novikov, Marc Najork, Bernhard Brodowsky, Alexandrin Popescul
Venue: WWW
Year: 2020

The rapid growth of commercial web content has driven the development of shopping search services to help users find product offers. Due to the dynamic nature of commercial content, an effective recrawl policy is a key component in a shopping search service; it ensures that users have access to the up-to-date product details. Most of the existing strategies either relied on simple heuristics, or overlooked the resource budgets. To address this, Azar et al. [5] recently proposed an optimization strategy LambdaCrawl aiming to maximize content freshness within a given resource budget. In this paper, we demonstrate that the effectiveness of LambdaCrawl is governed in large part by how well future content change rate can be estimated. By adopting the state-of-the-art deep learning models for change rate prediction, we obtain a substantial increase of content freshness over the common LambdaCrawl implementation with change rate estimated from the past history. Moreover, we demonstrate that while LambdaCrawl is a significant advancement upon existing recrawl strategies, it can be further improved upon by a unified multi-strategy recrawl policy. To this end, we adopt the K-armed adversarial bandits algorithm that can provably optimize the overall freshness by combining multiple strategies. Empirical results over a large-scale production dataset confirm its superiority to LambdaCrawl, especially under tight resource budgets.
Methods to Evaluate Temporal Cognitive Biases in Machine Learning Prediction Models
Authors: Christopher G. Harris
Venue: WWW
Year: 2020

When asked to rank or rate a list of items, humans are often affected by cognitive biases, which may lead to inconsistent decisions over time. These inconsistencies become part of machine learning prediction algorithms trained on human judgments, leading to misalignment and consequently affecting the metrics used to evaluate their correctness. In this paper, we propose new accuracy metrics, built upon commonly used statistics- and decision support-based metrics. Each of these metrics is designed to address the varying nature of human judgment and to evaluate the importance of decisions that change over time due to cognitive biases.
Mitigating Cognitive Biases in Machine Learning Algorithms for Decision Making
Authors: Christopher G. Harris
Venue: WWW
Year: 2020

Cognitive biases are an ingrained part of the human decision-making process. Nearly all machine learning algorithms that mimic human decision-making use human judgments as training data, which propagates these biases. In this paper, we conduct an empirical study in which 150 applicants are rated for suitability for three separate job openings. We develop an algorithm that learns from human judgments and consequently develops biases based on these human-generated inputs. Next, we explore and apply techniques to mitigate these algorithmic biases, using a combination of pre-processing, in-processing, and post-processing algorithms. The results from our study show that biases can be mitigated using these approaches but involve a tradeoff between complexity and effectiveness.
LOREM: Language-consistent Open Relation Extraction from Unstructured Text
Authors: Tom Harting, Sepideh Mesbah, Christoph Lofi
Venue: WWW
Year: 2020

We introduce a Language-consistent multi-lingual Open Relation Extraction Model (LOREM) for finding relation tuples of any type between entities in unstructured texts. LOREM does not rely on language-specific knowledge or external NLP tools such as translators or PoS-taggers, and exploits information and structures that are consistent over different languages. This allows our model to be easily extended with only limited training efforts to new languages, but also provides a boost to performance for a given single language. An extensive evaluation performed on 5 languages shows that LOREM outperforms state-of-the-art mono-lingual and cross-lingual open relation extractors. Moreover, experiments on languages with no or only little training data indicate that LOREM generalizes to other languages than the languages that it is trained on.
Towards Fine-grained Flow Forecasting: A Graph Attention Approach for Bike Sharing Systems
Authors: Suining He, Kang G. Shin
Venue: WWW
Year: 2020

As a healthy, efficient and green alternative to motorized urban travel, bike sharing has been increasingly popular, leading to wide deployment and use of bikes instead of cars. Accurate bike-flow prediction at the individual station level is essential for bike sharing service. Due to the spatial and temporal complexities of traffic networks and the lack of data-driven design for bike stations, existing methods cannot predict the fine-grained bike flows to/from each station. To remedy this problem, we propose a novel data-driven spatio-temporal Graph attention convolutional neural network for Bikestation-level flow prediction (GBikes). We develop data-driven and spatio-temporal designs, and model bike stations (nodes) and inter-station bike rides (edges) as a graph. In particular, we design a novel graph attention convolutional neural network (GACNN) with attention mechanisms capturing and differentiating station-to-station correlations. Multi-level temporal closeness, spatial distances and other external factors (e.g., weather and points of interest) are jointly considered for comprehensive learning and accurate prediction of bike flows at each station. Extensive experiments upon a total of over 11 million trips collected from three large-scale bike-sharing systems in New York City, Chicago, and Los Angeles have corroborated GBikes’s significant improvement of accuracy, robustness and effectiveness over prior work.
Constructing Knowledge Graph for Social Networks in A Deep and Holistic Way
Authors: Qi He, Jaewon Yang, Baoxu Shi
Venue: WWW
Year: 2020

Online social networks such as Facebook and LinkedIn have been an integrated part of everyday life. To improve the user experience and power the products around the social network, Knowledge Graphs (KG) are used as a standard way to extract and organize the knowledge in social networks. This tutorial focuses on how to build KGs for social networks by developing deep NLP models, and holistic optimization of KGs and the social network. Building KG for social networks poses two challenges: 1) input data for each member in the social network is noisy, implicit and in multilingual, so a deep understanding of the input data is needed; 2) KG and the social network influence each other via multiple organic feedback loops, so a holistic view on both networks is needed. We will share the lessons we learned from tackling the above challenges in the past seven years on building the Knowledge Graph for the LinkedIn social network. To address the first challenge of noisy and implicit input data, we present how to train high precision language understanding models by adding small clean data to the noisy data. By doing so, we enhance the-state-of-the-art NLP models such as BERT for building KG. To address multilingual aspect of the input data, we explain how to expand a single-language KG to multilingual KGs by applying transfer learning. For the second challenge of modeling interactions between social network and KG, we launch new products to get explicit feedback on KG from users, and refine KG by learning deep embeddings from the social network. Lastly, we present how we use our KG to empower more than 20+ products at LinkedIn with high business impacts.
Using Deep Learning for Temporal Forecasting of User Activity on Social Media: Challenges and Limitations
Authors: Anthony Hernandez, Kin Wai Ng, Adriana Iamnitchi
Venue: WWW
Year: 2020

The recent advances in neural network-based machine learning algorithms promise a revolution in prediction-based tasks in a variety of domains. Of these, forecasting user activity in social media is particularly relevant for problems such as modeling and predicting information diffusion and designing intervention techniques to mitigate disinformation campaigns. Social media seems an ideal context for applying neural network techniques, as they provide large datasets and challenging prediction objectives. Yet, our experiments find a number of limitations in the power of deep neural networks and traditional machine learning approaches in predicting user activity on social media platforms. These limitations are related to dataset characteristics due to temporal aspects of user behavior. This work describes the challenges we encountered while attempting to forecast user activity on two popular social interaction sites: Twitter and GitHub.
Expanding the Scope of Reproducibility Research Through Data Analysis Replications
Authors: Jake M. Hofman, Daniel G. Goldstein, Siddhartha Sen, Forough Poursabzi-Sangdeh
Venue: WWW
Year: 2020

In recent years, researchers in several scientific disciplines have become concerned with published studies replicating less often than expected. A positive side effect of this concern is an increased appreciation for replicating other researchers’ work as a vital part of the scientific process. To date, many such efforts have come from the experimental sciences, where replication entails running new experiments, generating new data, and analyzing it. In this article, we emphasize not experimental replication but data analysis replication. We do so for three reasons. First, experimental replication excludes entire classes of publications that do not run experiments or even collect original data (for example, papers that make use of economic data, census data, municipal data, and the like). Second, experimental replication may in some cases be a needlessly high bar: there is great value in replicating the data analyses of published experimental work. As analytical replications require a lower investment of time and money than experimental replications, their adoption should expand the number and variety of scientific reproducibility studies undertaken. Third, we propose educating undergraduate students to perform data analysis replications, which has scalable benefits for both the students themselves and the broader research community. In our talk we will provide details of a pilot program we created to teach undergraduates the skills necessary to conduct data analysis replications, and include a case study of the first set of students who completed this program and attempted to replicate a widely-cited social science paper on policing.
Hedging via Opinion-based Pair Trading Strategy
Authors: Ting-Wei Hsu, Chung-Chi Chen, Hen-Hsen Huang, Meng Chang Chen, Hsin-Hsi Chen
Venue: WWW
Year: 2020

Risk is an important component when constructing a trading strategy. However, most of the previous works that make the price movement prediction on the basis of the opinions on social media platforms do not take the risk into consideration. In order to hedge the market-risk, we propose an idea of an opinion-based pair trading strategy. Comparing with the task setting of the previous works, our experimental results show that the neural network models with the pair-wise task setting perform better in both accuracy and profitability metrics. That introduces a new research direction for future researches on opinion-based price movement predictions.
Heterogeneous Graph Transformer
Authors: Ziniu Hu, Yuxiao Dong, Kuansan Wang, Yizhou Sun
Venue: WWW
Year: 2020

Recent years have witnessed the emerging success of graph neural networks (GNNs) for modeling structured data. However, most GNNs are designed for homogeneous graphs, in which all nodes and edges belong to the same types, making it infeasible to represent heterogeneous structures. In this paper, we present the Heterogeneous Graph Transformer (HGT) architecture for modeling Web-scale heterogeneous graphs. To model heterogeneity, we design node- and edge-type dependent parameters to characterize the heterogeneous attention over each edge, empowering HGT to maintain dedicated representations for different types of nodes and edges. To handle Web-scale graph data, we design the heterogeneous mini-batch graph sampling algorithm—HGSampling—for efficient and scalable training. Extensive experiments on the Open Academic Graph of 179 million nodes and 2 billion edges show that the proposed HGT model consistently outperforms all the state-of-the-art GNN baselines by 9–21 on various downstream tasks. The dataset and source code of HGT are publicly available at https://github.com/acbull/pyHGT.
Crowdsourcing Detection of Sampling Biases in Image Datasets
Authors: Xiao Hu, Haobo Wang, Anirudh Vegesana, Somesh Dube, Kaiwen Yu, Gore Kao, Shuo-Han Chen, Yung-Hsiang Lu, George K. Thiruvathukal, Ming Yin
Venue: WWW
Year: 2020

Despite many exciting innovations in computer vision, recent studies reveal a number of risks in existing computer vision systems, suggesting results of such systems may be unfair and untrustworthy. Many of these risks can be partly attributed to the use of a training image dataset that exhibits sampling biases and thus does not accurately reflect the real visual world. Being able to detect potential sampling biases in the visual dataset prior to model development is thus essential for mitigating the fairness and trustworthy concerns in computer vision. In this paper, we propose a three-step crowdsourcing workflow to get humans into the loop for facilitating bias discovery in image datasets. Through two sets of evaluation studies, we find that the proposed workflow can effectively organize the crowd to detect sampling biases in both datasets that are artificially created with designed biases and real-world image datasets that are widely used in computer vision research and system development.
Learning Bi-directional Social Influence in Information Cascades using Graph Sequence Attention Networks
Authors: Zhenhua Huang, Zhenyu Wang, Rui Zhang, Yangyang Zhao, Fadong Zheng
Venue: WWW
Year: 2020

The online information cascades have spatial and temporal characteristics. Retweets in cascades have bi-directional social influence and temporal delay, e.g., a hub node influenced by a root node will further increase the exposure of information, enhance the influence of the root node, and cause subsequent retweets after a certain time. Existing deep learning approaches mostly consider only the decay effects of social influence, ignoring the bi-directional dependency and delay effects between graphs at different moments. Therefore, a novel method is presented here, namely the Graph Sequence Attention Networks (GSAN) , which addresses the bi-directional attention mechanism to learn temporal dynamics of social influence, as well as the cascading structure. A graph transformer block is designed to learn the complicated dependencies of spatial and temporal features in cascades. The proposed method could achieve state-of-the-art performance and large improvements in popularity prediction compared to strong baselines in real-world datasets.
Selecting Representative Thumbnail Image and Video Clip from a Video via Bullet Screen
Authors: Yun-Ya Huang, Tong-Yi Kuo, Hung-Hsuan Chen
Venue: WWW
Year: 2020

Finding representative thumbnails or key video clips in a long video usually requires laborious manual editing and compilation. Although there have been deep-learning-based methods that aim to understand videos and pictures, these approaches rely on a large number of computing resources and training data, and the results may still be unsatisfactory. This paper tackles the task of thumbnail selection and highlighting video selection from a different perspective — we leverage on the bullet screen, an emerging new feature on the online video streaming sites that are popular in East Asia, to select thumbnails and video clips. We compared the proposed method with a thumbnail and video clip selecting tool developed by KKStream, a leading streaming service provider in East Asia. We recruited 100 individuals to conduct subjective tests. The experimental results show that most participants are satisfied with the thumbnails and video clips selected by our method, suggesting that the bullet screen could be a valuable resource for video understanding.
Natura: Towards Conversational Analytics for Comparing and Contrasting Time Series
Authors: Shima Imani, Eamonn J. Keogh
Venue: WWW
Year: 2020

Comparing and contrasting two things is one of the most basic ideas in science. While existing tools allow such comparisons for pairs (e.g. texts, trees, graphs, histograms), there are currently no such tools for time series data. This is somewhat surprising, given the ubiquity of time series data in modern life. One could imagine a tool for comparing and contrasting two time series by reporting various summary statistics. However, scientists, engineers, and physicians typically communicate such findings in natural language. In this work we present Natura, a domain agnostic natural language framework for comparing and contrasting two time series, that aims to duplicate this human skill. Through case study, we demonstrate the effectiveness and utility of our framework.
ROSE: Role-based Signed Network Embedding
Authors: Amin Javari, Tyler Derr, Pouya Esmailian, Jiliang Tang, Kevin Chen-Chuan Chang
Venue: WWW
Year: 2020

In real-world networks, nodes might have more than one type of relationship. Signed networks are an important class of such networks consisting of two types of relations: positive and negative. Recently, embedding signed networks has attracted increasing attention and is more challenging than classic networks since nodes are connected by paths with multi-types of links. Existing works capture the complex relationships by relying on social theories. However, this approach has major drawbacks, including the incompleteness/inaccurateness of such theories. Thus, we propose network transformation based embedding to address these shortcomings. The core idea is that rather than directly finding the similarities of two nodes from the complex paths connecting them, we can obtain their similarities through simple paths connecting their different roles. We employ this idea to build our proposed embedding technique that can be described in three steps: (1) the input directed signed network is transformed into an unsigned bipartite network with each node mapped to a set of nodes we denote as role-nodes. Each role-node captures a certain role that a node in the original network plays; (2) the network of role-nodes is embedded; and (3) the original network is encoded by aggregating the embedding vectors of role-nodes. Our experiments show the novel proposed technique substantially outperforms existing models.
Structured Paragraph Embeddings of Financial Earnings Calls
Authors: Porter Jenkins
Venue: WWW
Year: 2020

Financial earnings calls contain rich information about the quarterly performance and future projections of public companies. Such information is highly relevant to developing trading strategies and understanding economic trends. However, due to the unstructured nature of call transcripts important signals can be difficult to extract. In this preliminary work, we propose a novel paragraph embedding method that leverages the structure inherent in the Q&A format of earnings calls. We show that the proposed method improves classification performance over more general methods and provides a useful measure of similarity between paragraphs.