OpenRaiser/Intern-Atlas huggingface dataset:
Learned token pruning for transformers
Authors: Sehoon Kim 0001, Sheng Shen 0001, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, Kurt Keutzer
Venue: Conference KDD 2022
Year: 2022

Efficient deployment of transformer models in practice is challenging due to their inference cost including memory footprint, latency, and power consumption, which scales quadratically with input sequence length. To address this, we present a novel token reduction method dubbed Learned Token Pruning (LTP) which adaptively removes unimportant tokens as an input sequence passes through transformer layers. In particular, LTP prunes tokens with an attention score below a threshold, whose value is learned for each layer during training. Our threshold-based method allows the length of the pruned sequence to vary adaptively based on the input sequence, and avoids algorithmically expensive operations such as top-k token selection. We extensively test the performance of LTP on GLUE and SQuAD tasks and show that our method outperforms the prior state-of-the-art token pruning methods by up to ∽2.5% higher accuracy with the same amount of FLOPs. In particular, LTP achieves up to 2.1× FLOPs reduction with less than 1% accuracy drop, which results in up to 1.9× and 2.0× throughput improvement on Intel Haswell CPUs and NVIDIA V100 GPUs. Furthermore, we demonstrate that LTP is more robust than prior methods to variations in input sequence lengths. Our code has been developed in PyTorch and open-sourced
Detecting online commercial intention (OCI
Authors: Honghua (Kathy) Dai, Lingzhi Zhao, Zaiqing Nie, Ji-Rong Wen, Lee Wang, Ying Li
Venue: Conference WWW 2006
Year: 2006

Understanding goals and preferences behind a user's online activities can greatly help information providers, such as search engine and E-Commerce web sites, to personalize contents and thus improve user satisfaction. Understanding a user's intention could also provide other business advantages to information providers. For example, information providers can decide whether to display commercial content based on user's intent to purchase. Previous work on Web search defines three major types of user search goals for search queries: navigational, informational and transactional or resource [1][7]. In this paper, we focus our attention on capturing commercial intention from search queries and Web pages, i.e., when a user submits the query or browse a Web page, whether he/she is about to commit or in the middle of a commercial activity, such as purchase, auction, selling, paid service, etc. We call the commercial intentions behind a user's online activities as OCI (Online Commercial Intention). We also propose the notion of "Commercial Activity Phase" (CAP), which identifies in which phase a user is in his/her commercial activities: Research or Commit. We present the framework of building machine learning models to learn OCI based on any Web page content. Based on that framework, we build models to detect OCI from search queries and Web pages. We train machine learning models from two types of data sources for a given search query: content of algorithmic search result page(s) and contents of top sites returned by a search engine. Our experiments show that the model based on the first data source achieved better performance. We also discover that frequent queries are more likely to have commercial intention. Finally we propose our future work in learning richer commercial intention behind users' online activities.
Read it twice: Towards faithfully interpretable fact verification by revisiting evidence
Authors: Xuming Hu, Zhaochen Hong, Zhijiang Guo, Lijie Wen, Philip S. Yu
Venue: Conference SIGIR 2023
Year: 2023

Real-world fact verification task aims to verify the factuality of a claim by retrieving evidence from the source document. The quality of the retrieved evidence plays an important role in claim verification. Ideally, the retrieved evidence should be faithful (reflecting the model's decision-making process in claim verification) and plausible (convincing to humans), and can improve the accuracy of verification task. Although existing approaches leverage the similarity measure of semantic or surface form between claims and documents to retrieve evidence, they all rely on certain heuristics that prevent them from satisfying all three requirements. In light of this, we propose a fact verification model named ReRead to retrieve evidence and verify claim that: (1) Train the evidence retriever to obtain interpretable evidence (i.e., faithfulness and plausibility criteria); (2) Train the claim verifier to revisit the evidence retrieved by the optimized evidence retriever to improve the accuracy. The proposed system is able to achieve significant improvements upon best-reported models under different settings.
Muser: A multi-step evidence retrieval enhancement framework for fake news detection
Authors: Hao Liao, Jiahao Peng, Zhanyi Huang, Wei Zhang 0242, Guanghua Li, Kai Shu, Xing Xie 0001
Venue: Conference KDD 2023
Year: 2023

The ease of spreading false information online enables individuals with malicious intent to manipulate public opinion and destabilize social stability. Recently, fake news detection based on evidence retrieval has gained popularity in an effort to identify fake news reliably and reduce its impact. Evidence retrieval-based methods can improve the reliability of fake news detection by computing the textual consistency between the evidence and the claim in the news. In this paper, we propose a framework for fake news detection based on MUlti- Step Evidence Retrieval enhancement (MUSER), which simulates the steps of human beings in the process of reading news, summarizing, consulting materials, and inferring whether the news is true or fake. Our model can explicitly model dependencies among multiple pieces of evidence, and perform multi-step associations for the evidence required for news verification through multi-step retrieval. In addition, our model is able to automatically collect existing evidence through paragraph retrieval and key evidence selection, which can save the tedious process of manual evidence collection. We conducted extensive experiments on real-world datasets in different languages, and the results demonstrate that our proposed model outperforms state-of-the-art baseline methods for detecting fake news by at least 3% in F1-Macro and 4% in F1-Micro. Furthermore, it provides interpretable evidence for end users.
Preventing and detecting misinformation generated by large language models
Authors: Aiwei Liu, Qiang Sheng, Xuming Hu
Venue: Conference SIGIR 2024
Year: 2024

As large language models (LLMs) become increasingly capable and widely deployed, the risk of them generating misinformation poses a critical challenge. Misinformation from LLMs can take various forms, from factual errors due to hallucination to intentionally deceptive content, and can have severe consequences in high-stakes domains.This tutorial covers comprehensive strategies to prevent and detect misinformation generated by LLMs. We first introduce the types of misinformation LLMs can produce and their root causes. We then explore two broad categories: Preventing misinformation generation: a) AI alignment training techniques to reduce LLMs' propensity for misinformation and refuse malicious instructions during model training. b) Training-free mitigation methods like prompt guardrails, retrieval-augmented generation (RAG), and decoding strategies to curb misinformation at inference time. Detecting misinformation after generation, including a) using LLMs themselves to detect misinformation through embedded knowledge or retrieval-enhanced judgments, and b) distinguishing LLM-generated text from human-written text through black-box approaches (e.g., classifiers, probability analysis) and white-box approaches (e.g., watermarking). We also discuss the challenges and limitations of detecting LLM-generated misinformation.
From group to individual labels using deep features
Authors: Dimitrios Kotzias, Misha Denil, Nando de Freitas, Padhraic Smyth
Venue: Conference KDD 2015
Year: 2015

In many classification problems labels are relatively scarce. One context in which this occurs is where we have labels for groups of instances but not for the instances themselves, as in multi-instance learning. Past work on this problem has typically focused on learning classifiers to make predictions at the group level. In this paper we focus on the problem of learning classifiers to make predictions at the instance level. To achieve this we propose a new objective function that encourages smoothness of inferred instance-level labels based on instance-level similarity, while at the same time respecting group-level label constraints. We apply this approach to the problem of predicting labels for sentences given labels for reviews, using a convolutional neural network to infer sentence similarity. The approach is evaluated using three large review data sets from IMDB, Yelp, and Amazon, and we demonstrate the proposed approach is both accurate and scalable compared to various alternatives.
Controlling patent text generation by structural metadata
Authors: Jieh-Sheng Lee
Venue: Conference CIKM 2020
Year: 2020

The ultimate goal of my long-term project is "Augmented Inventing." This work is a follow-up effort toward the goal. It leverages the structural metadata in patent documents and the text-to-text mappings between metadata. The structural metadata includes patent title, abstract, independent claim, and dependent claim. By using the structural metadata, it is possible to control what kind of patent text to generate. By using the text-to-text mapping, it is possible to let a generative model generate one type of patent text from another type of patent text. Furthermore, through multiple mappings, it is possible to build a text generation flow, for example, generating from a few words to a patent title, from the title to an abstract, from the abstract to an independent claim, and from the independent claim to multiple dependent claims. The text generation flow can also go backward after training with bi-directional mappings. In addition to those above, the contributions of this work include: (1) released four generative models trained with patent corpus from scratch, (2) released the sample code to demonstrate how to generate patent text bi-directionally, (3) measuring the performances of the models by ROGUE and Universal Sentence Encoder as preliminary evaluations of text generation quality.
Topic quality metrics based on distributed word representations
Authors: Sergey I. Nikolenko
Venue: Conference SIGIR 2016
Year: 2016

Automated evaluation of topic quality remains an important unsolved problem in topic modeling and represents a major obstacle for development and evaluation of new topic models. Previous attempts at the problem have been formulated as variations on the coherence and/or mutual information of top words in a topic. In this work, we propose several new metrics for evaluating topic quality with the help of distributed word representations; our experiments suggest that the new metrics are a better match for human judgement, which is the gold standard in this case, than previously developed approaches.
Enhanced social event detection through dynamically weighted meta-paths modeling
Authors: Congbo Ma, Zitai Qiu, Hu Wang, Jing Du, Shan Xue, Jia Wu, Jian Yang
Venue: Conference WWW 2025
Year: 2025

Social event detection (SED) identifies significant events in social networks by analyzing complex interactions using both textual data and structural relationships that often span multiple nodes. This requires exploring long-range dependencies, which increases computational costs, especially with many neighbors.Therefore, in this paper, we present the Dynamically Weighted Meta-Paths Modeling (DWMM) framework for large-scale social event detection. It includes three main modules: 1) a graph building module to convert social event data into Heterogeneous Information Networks (HINs); 2) a meta-path searching module to determine the significant meta-paths and their importance; 3) a model training module that uses weighted top-k meta-paths for social event detection. Extensive experiments on three widely used social event detection datasets show that DWMM enhances performance, demonstrating its effectiveness.
Building a large-scale corpus for evaluating event detection on twitter
Authors: Andrew James McMinn, Yashar Moshfeghi, Joemon M. Jose
Venue: Conference CIKM 2013
Year: 2013

Despite the popularity of Twitter for research, there are very few publicly available corpora, and those which are available are either too small or unsuitable for tasks such as event detection. This is partially due to a number of issues associated with the creation of Twitter corpora, including restrictions on the distribution of the tweets and the difficultly of creating relevance judgements at such a large scale. The difficulty of creating relevance judgements for the task of event detection is further hampered by ambiguity in the definition of event. In this paper, we propose a methodology for the creation of an event detection corpus. Specifically, we first create a new corpus that covers a period of 4 weeks and contains over 120 million tweets, which we make available for research. We then propose a definition of event which fits the characteristics of Twitter, and using this definition, we generate a set of relevance judgements aimed specifically at the task of event detection. To do so, we make use of existing state-of-the-art event detection approaches and Wikipedia to generate a set of candidate events with associated tweets. We then use crowdsourcing to gather relevance judgements, and discuss the quality of results, including how we ensured integrity and prevented spam. As a result of this process, along with our Twitter corpus, we release relevance judgements containing over 150,000 tweets, covering more than 500 events, which can be used for the evaluation of event detection approaches.
An efficient automatic meta-path selection for social event detection via hyperbolic space
Authors: Zitai Qiu, Congbo Ma, Jia Wu, Jian Yang
Venue: Conference WWW 2024
Year: 2024

Social events reflect changes in communities, such as natural disasters and emergencies. Detection of these situations can help residents and organizations in the community avoid danger and reduce losses. The complex nature of social messages makes social event detection on social media challenging. The challenges that have a greater impact on social media detection models are as follows: (1) the amount of social media data is huge but its availability is small; (2) social media data is a tree structure and traditional Euclidean space embedding will distort embedded features; and (3) the heterogeneity of social media networks makes existing models unable to capture rich information well. To solve the above challenges, we propose a Heterogeneous Information Graph representation via Hyperbolic space combined with an Automatic Meta-path selection (GraphHAM) model, an efficient framework that automatically selects the meta-path's weight and combines hyperbolic space to learn information on social media. In particular, we apply an efficient automatic meta-path selection technique and convert the selected meta-path into a vector, thereby reducing the requisite amount of labeled data for the model. We also design a novel Hyperbolic Multi-Layer Perceptron (HMLP) to further learn the semantic and structural information of social information. Extensive experiments show that GraphHAM can achieve outstanding performance on real-world data using only 20% of the whole dataset as the training set. Our code can be found on GitHub https://github.com/ZITAIQIU/GraphHAM.
Tnt-llm: Text mining at scale with large language models
Authors: Mengting Wan, Tara Safavi, Sujay Kumar Jauhar, Yujin Kim, Scott Counts, Jennifer Neville, Siddharth Suri, Chirag Shah 0001, Ryen W. White, Longqi Yang 0001, Reid Andersen, Georg Buscher, Dhruv Joshi, Nagu Rangan
Venue: Conference KDD 2024
Year: 2024

Transforming unstructured text into structured and meaningful forms, organized by useful category labels, is a fundamental step in text mining for downstream analysis and application. However, most existing methods for producing label taxonomies and building text-based label classifiers still rely heavily on domain expertise and manual curation, making the process expensive and time-consuming. This is particularly challenging when the label space is under-specified and large-scale data annotations are unavailable. In this paper, we address these challenges with Large Language Models (LLMs), whose prompt-based interface facilitates the induction and use of large-scale pseudo labels. We propose TnT-LLM, a two-phase framework that employs LLMs to automate the process of end-to-end label generation and assignment with minimal human effort for any given use-case. In the first phase, we introduce a zero-shot, multi-stage reasoning approach which enables LLMs to produce and refine a label taxonomy iteratively. In the second phase, LLMs are used as data labelers that yield training samples so that lightweight supervised classifiers can be reliably built, deployed, and served at scale. We apply TnT-LLM to the analysis of user intent and conversational domain for Bing Copilot (formerly Bing Chat), an open-domain chat-based search engine. Extensive experiments using both human and automatic evaluation metrics demonstrate that TnT-LLM generates more accurate and relevant label taxonomies when compared against state-of-the-art baselines, and achieves a favorable balance between accuracy and efficiency for classification at scale. We also share our practical experiences and insights on the challenges and opportunities of using LLMs for large-scale text mining in real-world applications.
A scalable laplace approximation for neural networks
Authors: Hippolyt Ritter, Aleksandar Botev, David Barber
Venue: Conference ICLR 2018
Year: 2018

We leverage recent insights from second-order optimisation for neural networks to construct a Kronecker factored Laplace approximation to the posterior over the weights of a trained network. Our approximation requires no modification of the training procedure, enabling practitioners to estimate the uncertainty of their models currently used in production without having to retrain them. We extensively compare our method to using Dropout and a diagonal Laplace approximation for estimating the uncertainty of a network. We demonstrate that our Kronecker factored method leads to better uncertainty estimates on out-of-distribution data and is more robust to simple adversarial attacks. Our approach only requires calculating two square curvature factor matrices for each layer. Their size is equal to the respective square of the input and output size of the layer, making the method efficient both computationally and in terms of memory usage. We illustrate its scalability by applying it to a state-of-the-art convolutional network architecture.
You CAN Teach an Old Dog New Tricks! On Training Knowledge Graph Embeddings
Authors: Daniel Ruffinelli, Samuel Broscheit, Rainer Gemulla
Venue: Conference ICLR 2020
Year: 2020

Knowledge graph embedding (KGE) models learn algebraic representations of the entities and relations in a knowledge graph. A vast number of KGE techniques for multi-relational link prediction have been proposed in the recent literature, often with state-of-the-art performance. These approaches differ along a number of dimensions, including different model architectures, different training strategies, and different approaches to hyperparameter optimization. In this paper, we take a step back and aim to summarize and quantify empirically the impact of each of these dimensions on model performance. We report on the results of an extensive experimental study with popular model architectures and training strategies across a wide range of hyperparameter settings. We found that when trained appropriately, the relative performance differences between various model architectures often shrinks and sometimes even reverses when compared to prior results. For example, RESCAL~\\citep{nickel2011three}, one of the first KGE models, showed strong performance when trained with state-of-the-art techniques; it was competitive to or outperformed more recent architectures. We also found that good (and often superior to prior studies) model configurations can be found by exploring relatively few random samples from a large hyperparameter space. Our results suggest that many of the more advanced architectures and techniques proposed in the literature should be revisited to reassess their individual benefits. To foster further reproducible research, we provide all our implementations and experimental results as part of the open source LibKGE framework.
ML-LJP: Multi-Law Aware Legal Judgment Prediction
Authors: Yifei Liu, Yiquan Wu, Yating Zhang, Changlong Sun, Weiming Lu, Fei Wu, Kun Kuang
Venue: Conference SIGIR 2023
Year: 2023

Legal judgment prediction (LJP) is a significant task in legal intelligence, which aims to assist the judges and determine the judgment result based on the case's fact description. The judgment result consists of law articles, charge, and prison term. The law articles serve as the basis for the charge and the prison term, which can be divided into two types, named as charge-related law article and term-related law article, respectively. Recently, many methods have been proposed and made tremendous progress in LJP. However, the existing methods only focus on the prediction of the charge-related law articles, ignoring the term-related law articles (e.g., laws about lenient treatment), which limits the performance in the prison term prediction. In this paper, following the actual legal process, we expand the law article prediction as a multi-label classification task that includes both the charge-related law articles and term-related law articles and propose a novel multi-law aware LJP (ML-LJP) method to improve the performance of LJP. Given the case's fact description, firstly, the label (e.g., law article and charge) definitions in the Code of Law are used to transform the representation of the fact into several label-specific representations and make the prediction of the law articles and the charge. To distinguish the similar content of different label definitions, contrastive learning is conducted in the training. Then, a graph attention network (GAT) is applied to learn the interactions among the multiple law articles for the prediction of the prison term. Since numbers (e.g., amount of theft and weight of drugs) are important for LJP but often ignored by conventional encoders, we design a corresponding number representation method to locate and better represent these effective numbers. Extensive experiments on real-world dataset show that our method achieves the best results compared to the state-of-the-art models, especially in the task of prison term prediction where ML-LJP achieves a 10.07% relative improvement over the best baseline.