OpenRaiser/Intern-Atlas huggingface dataset:
Towards semantic retrieval of hashtags in microblogs
Authors: Piyush Bansal, Somay Jain, Vasudeva Varma
Venue: Conference WWW 2015
Year: 2015

On various microblogging platforms like Twitter, the users post short text messages ranging from news and information to thoughts and daily chatter. These messages often contain keywords called Hashtags, which are semantico-syntactic constructs that enable topical classification of the microblog posts. In this poster, we propose and evaluate a novel method of semantic enrichment of microblogs for a particular type of entity search -- retrieving a ranked list of the top-k hashtags relevant to a user's query Q. Such a list can help the users track posts of their general interest. We show that our technique significantly improved microblog retrieval as well. We tested our approach on the publicly available Stanford sentiment analysis tweet corpus. We observed an improvement of more than 10% in NDCG for microblog retrieval task, and around 11% in mean average precision for hashtag retrieval task.
Latent User Intent Modeling for Sequential Recommenders
Authors: Bo Chang, Alexandros Karatzoglou, Yuyan Wang, Can Xu, Ed H. Chi, Minmin Chen
Venue: Conference WWW 2023
Year: 2023

Sequential recommender models are essential components of modern industrial recommender systems. These models learn to predict the next items a user is likely to interact with based on his/her interaction history on the platform. Most sequential recommenders however lack a higher-level understanding of user intents, which often drive user behaviors online. Intent modeling is thus critical for understanding users and optimizing long-term user experience. We propose a probabilistic modeling approach and formulate user intent as latent variables, which are inferred based on user behavior signals using variational autoencoders (VAE). The recommendation policy is then adjusted accordingly given the inferred user intent. We demonstrate the effectiveness of the latent user intent modeling via offline analyses as well as live experiments on a large-scale industrial recommendation platform.
Notellm-2: Multimodal large representation models for recommendation
Authors: Chao Zhang, Haoxin Zhang, Shiwei Wu, Di Wu, Tong Xu, Xiangyu Zhao, Yan Gao, Yao Hu, Enhong Chen
Venue: Conference KDD 2025
Year: 2025

Large Language Models (LLMs) have demonstrated exceptional proficiency in text understanding and embedding tasks. However, their potential in multimodal representation, particularly for item-to-item (I2I) recommendations, remains underexplored. While leveraging existing Multimodal Large Language Models (MLLMs) for such tasks is promising, challenges arise due to their delayed release compared to corresponding LLMs and the inefficiency in representation tasks. To address these issues, we propose an end-to-end fine-tuning method that customizes the integration of any existing LLMs and vision encoders for efficient multimodal representation. Preliminary experiments revealed that fine-tuned LLMs often neglect image content. To counteract this, we propose NoteLLM-2, a novel framework that enhances visual information. Specifically, we propose two approaches: first, a prompt-based method that segregates visual and textual content, employing a multimodal In-Context Learning strategy to balance focus across modalities; second, a late fusion technique that directly integrates visual information into the final representations. Extensive experiments, both online and offline, demonstrate the effectiveness of our approach. Code is available at https://github.com/Applied-Machine-Learning-Lab/NoteLLM.
Fedkc: Federated knowledge composition for multilingual natural language understanding
Authors: Haoyu Wang, Handong Zhao, Yaqing Wang, Tong Yu, Jiuxiang Gu, Jing Gao
Venue: Conference WWW 2022
Year: 2022

Multilingual natural language understanding, which aims to comprehend multilingual documents, is an important task. Existing efforts have been focusing on the analysis of centrally stored text data, but in real practice, multilingual data is usually distributed. Federated learning is a promising paradigm to solve this problem, which trains local models with decentralized data on local clients and aggregates local models on the central server to achieve a good global model. However, existing federated learning methods assume that data are independent and identically distributed (IID), and cannot handle multilingual data, that are usually non-IID with severely skewed distributions: First, multilingual data is stored on local client devices such that there are only monolingual or bilingual data stored on each client. This makes it difficult for local models to know the information of documents in other languages. Second, the distribution over different languages could be skewed. High resource language data is much more abundant than low resource language data. The model trained on such skewed data may focus more on high resource languages but fail to consider the key information of low resource languages. To solve the aforementioned challenges of multilingual federated NLU, we propose a plug-and-play knowledge composition (KC) module, called FedKC, which exchanges knowledge among clients without sharing raw data. Specifically, we propose an effective way to calculate a consistency loss defined based on the shared knowledge across clients, which enables models trained on different clients achieve similar predictions on similar data. Leveraging this consistency loss, joint training is thus conducted on distributed data respecting the privacy constraints. We also analyze the potential risk of FedKC and provide theoretical bound to show that it is difficult to recover data from the corrupted data. We conduct extensive experiments on three public multilingual datasets for three typical NLU tasks, including paraphrase identification, question answering matching, and news classification. The experiment results show that the proposed FedKC can outperform state-of-the-art baselines on the three datasets significantly.
Content-based collaborative generation for recommender systems
Authors: Yidan Wang, Zhaochun Ren, Weiwei Sun, Jiyuan Yang, Zhixiang Liang, Xin Chen, Ruobing Xie, Su Yan, Xu Zhang, Pengjie Ren, Zhumin Chen, Xin Xin
Venue: Conference CIKM 2024
Year: 2024

Generative models have emerged as a promising utility to enhance recommender systems. It is essential to model both item content and user-item collaborative interactions in a unified generative framework for better recommendation. Although some existing large language model (LLM)-based methods contribute to fusing content information and collaborative signals, they fundamentally rely on textual language generation, which is not fully aligned with the recommendation task. How to integrate content knowledge and collaborative interaction signals in a generative framework tailored for item recommendation is still an open research challenge. In this paper, we propose co ntent-based col la borative generation for rec ommender systems, namely ColaRec. ColaRec is a sequence-to-sequence framework which is tailored for directly generating the recommended item identifier. Precisely, the input sequence comprises data pertaining to the user's interacted items, and the output sequence represents the generative identifier (GID) for the suggested item. To model collaborative signals, the GIDs are constructed from a pretrained collaborative filtering model, and the user is represented as the content aggregation of interacted items. To this end, ColaRec captures both collaborative signals and content information in a unified framework. Then an item indexing task is proposed to conduct the alignment between the content-based semantic space and the interaction-based collaborative space. Besides, a contrastive loss is further introduced to ensure that items with similar collaborative GIDs have similar content representations. To verify the effectiveness of ColaRec, we conduct experiments on four benchmark datasets. Empirical results demonstrate the superior performance of ColaRec.
Advertising Keyword Suggestion Based on Concept Hierarchy
Authors: Yifan Chen, Gui-Rong Xue, Yong Yu
Venue: Conference WSDM 2008
Year: 2008

The increasing growth of the World Wide Web constantly enlarges the revenue generated by search engine advertising. Advertisers bid on keywords associated with their products to display their ads on the search result pages. Keyword suggestion methods are proposed to fill the gap between the keywords chosen by advertisers and the popular queries, through finding new relevant keywords according to some statistical information (for example, the keyword co-occurrence). However, there is little effort taking semantic information, such as concept hierarchy, into account. In this paper, we propose a novel keyword suggestion method that fully exploits the semantic knowledge among concept hierarchy. Given a keyword, we first match it with some relevant concepts. Then the relevant concepts are used with their hierarchy to fertilize the meanings of the keywords. Finally new keywords are suggested according to the concept information rather than the statistical co-occurrence of the keyword itself. Experimental results show that our proposed method can successfully provide suggestion that meets the accuracy and coverage requirements
One-step reach: Llm-based keyword generation for sponsored search advertising
Authors: Yang Wang, Zheyi Sha, Kunhai Lin, Chaobing Feng, Kunhong Zhu, Lipeng Wang, Xuewu Jiao, Fei Huang, Chao Ye, Dengwu He, Zhi Guo, Shuanglong Li, Lin Liu
Venue: Conference WWW 2024
Year: 2024

Query keyword matching plays a crucial role in sponsored search advertising by retrieving semantically related keywords of the user query to target relevant advertisements. Conventional technical solutions adopt the retrieve-judge-then-rank retrieval framework structured in cascade funnels. However, it has limitations in accurately depicting the semantic relevance between the query and keyword, and the cumulative funnel losses result in unsatisfactory precision and recall. To address the above issues, this paper proposes a Large Language Model (LLM)-based keyword generation method (LKG) to reach related keywords from the search query in one step. LKG models the query keyword matching as an end-to-end keyword generation task based on the LLM through multi-match prompt tuning. Moreover, it employs the feedback tuning and the prefix tree-based constrained beam search to improve the generation quality and efficiency. Extensive offline experiments and online A/B testing demonstrate the effectiveness and superiority of LKG which is fully deployed in the Baidu sponsored search system bringing significant improvements.
Llmcloudhunter: Harnessing llms for automated extraction of detection rules from cloud-based cti
Authors: Yuval Schwartz, Lavi Ben-Shimol, Dudu Mimran, Yuval Elovici, Asaf Shabtai
Venue: Conference WWW 2025
Year: 2025

As the number and sophistication of cyber attacks have increased, threat hunting has become a critical aspect of active security, enabling proactive detection and mitigation of threats before they cause significant harm. Open-source cyber threat intelligence (OSCTI) is a valuable resource for threat hunters, however, it often comes in unstructured formats that require further manual analysis. Previous studies aimed at automating OSCTI analysis are limited since (1) they failed to provide actionable outputs, (2) they did not take advantage of images present in OSCTI sources, and (3) they focused on on-premises environments, overlooking the growing importance of cloud environments. To address these gaps, we propose LLMCloudHunter, a novel framework that leverages large language models (LLMs) to automatically generate generic-signature detection rule candidates from textual and visual OSCTI data. We evaluated the quality of the rules generated by the proposed framework using 20 annotated real-world cloud threat reports. The results show that our framework achieved a precision of 83% and recall of 99% for the task of accurately extracting API calls made by the threat actor and a precision of 99% with a recall of 97% for IoCs. Additionally, 99.18% of the generated detection rule candidates were successfully compiled and converted into Splunk queries.
Ideabench: Benchmarking large language models for research idea generation
Authors: Sikun Guo, Amir Hassan Shariatmadari, Guangzhi Xiong, Albert Huang, Myles Kim, Corey M. Williams, Stefan Bekiranov, Aidong Zhang
Venue: Conference KDD 2025
Year: 2025

Large Language Models (LLMs) have revolutionized interactions between human and artificial intelligence (AI) systems, demonstrating state-of-the-art performance across various domains, including scientific discovery and hypothesis generation. However, the absence of a comprehensive and systematic evaluation framework for LLM-driven research idea generation hinders a rigorous understanding of their strengths and limitations. To address this gap, we propose IdeaBench, a benchmark system that provides a structured dataset and evaluation framework for standardizing the assessment of research idea generation by LLMs. Our dataset comprises titles and abstracts from 2,374 influential papers across eight research domains, along with their 29,408 referenced works, creating a context-rich environment that mirrors human researchers' ideation processes. By profiling LLMs as domain-specific researchers and grounding them in similar contextual constraints, we directly leverage the models' knowledge learned from the pre-training stage to generate new research ideas. To systematically evaluate LLMs' research ideation capability and approximate human assessment, we propose a reference-based metric that aligns with human judgment to quantify idea quality with the assistance of LLMs. Through this evaluation, we find that while LLMs excel at generating novel ideas, they may struggle with generating feasible ideas. IdeaBench serves as a critical resource for benchmarking and comparing LLMs, ultimately advancing research on AI's role in automating scientific discovery.
A field guide to automatic evaluation of llm-generated summaries
Authors: Tempest A. van Schaik, Brittany Pugh
Venue: Conference SIGIR 2024
Year: 2024

Large Language models (LLMs) are rapidly being adopted for tasks such as text summarization, in a wide range of industries. This has driven the need for scalable, automatic, reliable, and cost-effective methods to evaluate the quality of LLM-generated text. What is meant by evaluating an LLM is not yet well defined and there are widely different expectations about what kind of information evaluation will produce. Evaluation methods that were developed for traditional Natural Language Processing (NLP) tasks (before the rise of LLMs) remain applicable but are not sufficient for capturing high-level semantic qualities of summaries. Emerging evaluation methods that use LLMs to evaluate LLM-output, appear to be powerful but lacking in reliability. New elements of LLM generated text that were not an element of previous NLP tasks, such as the artifacts of hallucination, need to be considered. We outline the different types of LLM evaluation currently used in the literature but focus on offline, system-level evaluation of the text generated by LLMs. Evaluating LLM-generated summaries is a complex and fast-evolving area, and we propose strategies for applying evaluation methods to avoid common pitfalls. Despite having promising strategies for evaluating LLM summaries, we highlight some open challenges that remain.
Ciral: A test collection for clir evaluations in african languages
Authors: Mofetoluwa Adeyemi, Akintunde Oladipo, Xinyu Zhang, David Alfonso-Hermelo, Mehdi Rezagholizadeh, Boxing Chen, Abdul-Hakeem Omotayo, Idris Abdulmumin, Naome A. Etori, Toyib Babatunde Musa, Samuel Fanijo, Oluwabusayo Olufunke Awoyomi, Saheed Abdullahi Salahudeen, Labaran Adamu Mohammed, Daud Olamide Abolade, Falalu Ibrahim Lawan, Maryam Sabo Abubakar, Ruqayya Nasir Iro, Amina Abubakar Imam, Shafie Abdi Mohamed, Hanad Mohamud Mohamed, Tunde Oluwaseyi Ajayi, Jimmy Lin
Venue: Conference SIGIR 2024
Year: 2024

Cross-lingual information retrieval (CLIR) continues to be an actively studied topic in information retrieval (IR), and there have been consistent efforts in curating test collections to support its research. However, there is a lack of high-quality human-annotated CLIR resources for African languages: the few existing collections are mostly curated synthetically or from sources with limited corpora for these languages. We present CIRAL, a test collection for cross-lingual retrieval with English queries and passages in four African languages: Hausa, Somali, Swahili, and Yoruba. CIRAL's corpora are obtained from Indigenous African websites and consist of a total of over 2.5 million passages. We gathered over 1,600 queries and 30k high-quality binary relevance judgments annotated by native speakers of the languages. Additional pools were also obtained at CIRAL's shared task, which was hosted at the Forum for Information Retrieval Evaluation 2023 to encourage community participation in CLIR for African languages. We describe the design and curation process of our test collection and provide reproducible baselines that demonstrate CIRAL's utility in evaluating the effectiveness of systems. CIRAL is available at https://github.com/ciralproject/ciral.
Query expansion for mixed-script information retrieval
Authors: Parth Gupta, Kalika Bali, Rafael E. Banchs, Monojit Choudhury, Paolo Rosso
Venue: Conference SIGIR 2014
Year: 2014

For many languages that use non-Roman based indigenous scripts (e.g., Arabic, Greek and Indic languages) one can often find a large amount of user generated transliterated content on the Web in the Roman script. Such content creates a monolingual or multi-lingual space with more than one script which we refer to as the Mixed-Script space. IR in the mixed-script space is challenging because queries written in either the native or the Roman script need to be matched to the documents written in both the scripts. Moreover, transliterated content features extensive spelling variations. In this paper, we formally introduce the concept of Mixed-Script IR, and through analysis of the query logs of Bing search engine, estimate the prevalence and thereby establish the importance of this problem. We also give a principled solution to handle the mixed-script term matching and spelling variation where the terms across the scripts are modelled jointly in a deep-learning architecture and can be compared in a low-dimensional abstract space. We present an extensive empirical analysis of the proposed method along with the evaluation results in an ad-hoc retrieval setting of mixed-script IR where the proposed method achieves significantly better results (12% increase in MRR and 29% increase in MAP) compared to other state-of-the-art baselines.
User-llm: Efficient LLM contextualization with user embeddings
Authors: Lin Ning, Luyang Liu, Jiaxing Wu, Neo Wu, Devora Berlowitz, Sushant Prakash, Bradley Green, Shawn O&apos, Banion, Jun Xie
Venue: Conference WWW 2025
Year: 2025

Large language models (LLMs) hold immense potential for personalized AI, but effectively incorporating user history for personalized responses remains challenging. Existing methods often convert user timelines into lengthy text descriptions, leading to high computational cost and potential loss of nuanced information. Inspired by the successful integration of LLMs with other modalities, such as images, we introduce USER-LLM, a novel framework that treats user timelines as a distinct modality and leverages user embeddings for efficient LLM contextualization. User embeddings, generated by a pretrained user encoder, capture latent user behaviors and interests from diverse interaction data. By integrating these embeddings with LLMs through cross-attention, USER-LLM enables LLMs to dynamically adapt their responses to individual user history. Our evaluation on three diverse datasets (MovieLens, Amazon Review, and Google Local Review) demonstrates that User-LLM achieves substantial computation reduction (up to 78.1X) compared to text-prompt-based methods, without sacrificing performance. Importantly, User-LLM maintains or even improves performance on tasks requiring deep user understanding, particularly with long user histories, highlighting its effectiveness in efficiently capturing and leveraging user information for personalized responses.
Finreport: Explainable stock earnings forecasting via news factor analyzing model
Authors: Xiangyu Li, Xinjie Shen, Yawen Zeng, Xiaofen Xing, Jin Xu
Venue: Conference WWW 2024
Year: 2024

The task of stock earnings forecasting has received considerable attention due to the demand investors in real-world scenarios. However, compared with financial institutions, it is not easy for ordinary investors to mine factors and analyze news. On the other hand, although large language models in the financial field can serve users in the form of dialogue robots, it still requires users to have financial knowledge to ask reasonable questions. To serve the user experience, we aim to build an automatic system, FinReport, for ordinary investors to collect information, analyze it, and generate reports after summarizing. Specifically, our FinReport is based on financial news announcements and a multi-factor model to ensure the professionalism of the report. The FinReport consists of three modules: news factorization module, return forecasting module, risk assessment module. The news factorization module involves understanding news information and combining it with stock factors, the return forecasting module aim to analysis the impact of news on market sentiment, and the risk assessment module is adopted to control investment risk. Extensive experiments on real-world datasets have well verified the effectiveness and explainability of our proposed FinReport. Our codes and datasets are available at https://github.com/frinkleko/FinReport.
Graphwiz: An instruction-following language model for graph computational problems
Authors: Nuo Chen 0001, Yuhan Li 0001, Jianheng Tang, Jia Li 0009
Venue: Conference KDD 2024
Year: 2024

Large language models (LLMs) have achieved impressive success across various domains, but their capability in understanding and resolving complex graph problems is less explored. To bridge this gap, we introduce GraphInstruct, a novel instruction-tuning dataset aimed at enabling language models to tackle a broad spectrum of graph problems through explicit reasoning paths. Utilizing GraphInstruct, we build GraphWiz, an open-source language model capable of solving various graph computational problems while generating clear reasoning processes. To further enhance the model's performance and reliability, we integrate the Direct Preference Optimization (DPO) framework within the graph problem-solving context. The improved model, GraphWiz-DPO, achieves an average accuracy of 65% across nine tasks with different complexity levels, surpassing GPT-4 which has an average accuracy of 43.8%. Our study also investigates the relationship between training data volume and model performance, emphasizing the risk of overfitting as data volume increases. Additionally, we explore the transferability of the proposed model across different tasks and datasets, demonstrating its robust zero-shot generalization capability. GraphWiz offers a new blueprint and valuable insights for developing LLMs specialized in graph reasoning and problem-solving.