Reviewerly/exHarmony huggingface dataset:
Recommendation Systems in the Era of LLMs

Recommendation Systems (RecSys) in e-commerce play a pivotal role in enhancing the user experience by providing personalized recommendations of products specific to the user preferences. The power of RecSys lies in its ability to anticipate user needs through signals like historical purchases, tastes, preferences, similar user types, etc. These factors drive customer satisfaction, user engagement, revenue growth, etc. Since the advent of transformer-based embeddings, RecSys has been primarily driven by BERT-based architectures. However, in the era of Large Language Models (LLMs), interest in applying LLMs in the RecSys domain has gained significant attention. LLMs trained on massive amounts of data have the potential to enhance various aspects of RecSys through techniques like fine-tuning, prompt tuning, etc. However, these techniques are not without challenges for deployment in real-world applications. This talk discusses the potential use of LLMs in recommendation systems and challenges in real-world applications such as training efficiency, inference, and bias.
Towards development of an effective AI-based system for relevant comment generation

The comments are essential to understand the functionality of the source code. The comments are crucial to enhancing code readability, maintainability, and collaboration among developers. However, writing elaborate comments for codes is sometimes time-consuming and the comments written for codes do not clearly bring out the meaning of the latter. Furthermore, given the availability of AI-based text generators in the form of ChatGPT, developers are using them to automatically generate comments for their programs to save time and effort. However, the quality of the comments generated by these AI-based systems and the ability of these systems to generate comments for complex programs is not well-studied in the literature. In this research work, we wish to explore the efficacy of AI-based systems in generating comments and compare the quality of such synthetically generated comments with human-written comments.
Student Behavior Detection in the classroom based on Human-Object Interaction model

Student behavior in the classroom is an important topic in educational analysis, and it is important to better understand and measure these behaviors. The smart classroom of the future will enhance the learning experience and support seamless communication between students and teachers through real-time learning and machine learning. The use of Object detection algorithm to recognize student behavior in the classroom can improve the speed of work, accuracy of analysis and application scale, and there are some studies that will promote the development of education with science and technology. At the same time, with the development of recognition algorithms, the use of neural networks in object detection has become a trend.
Overview of the FIRE 2023 Track: Artificial Intelligence on Social Media (AISoMe)

The COVID-19 pandemic showed the importance of vaccination at a large scale. However, quite often people expressed different concerns they had towards vaccines which made them hesitant to take them. Some people were concerned about the potential side-effects of vaccines, while some believed that the vaccines were not necessary due to the disease being mild. These concerns were frequently shared on social media sites such as Twitter. The FIRE 2023 AISoMe track focused on identifying these specific concern(s) that people have towards vaccines from tweets, as a 12-class multi-label classification task.
Relative Relation Representation Aware Link Prediction in Heterogeneous Information Network

Most real-world networks are heterogeneous in nature, with various types of nodes and edges. Relationships (edges) between different types of entities (nodes) have different semantics; thus, relationship prediction in heterogeneous information networks (HINs) aims to predict the existence of a relation between entities that can be of different types. Existing link prediction methods on HINs are mainly created by resembling the propagation mechanism of node representations, ignoring the diversity of various edge types. In this work, we present an end-to-end link prediction concept for HINs using relation representation learning. By leveraging the diversity of multiple types of relation, we introduce a method that uses node representation as well as relative relation representation to predict the links.
Towards Semantic Role Labeling of Legal Documents for Improving NLP/IR Tasks in the Law Domain

Abstract The exponential growth of the number of legal documents online in digital format has posed unforeseen challenges for legal practitioners. Legal documents are often long, and complex, and handling them efficiently needs sophisticated technologies. A pivotal characteristic of a Common Law System is that it accords great importance to prior cases. The lack of familiarity with the law is another serious problem for the common man. In the scope of my doctoral thesis, my primary aim is to develop assistive tools and techniques for use by legal practitioners as well as by the common man. I plan to extract important events or fine-grained information and leverage these to expedite tasks such as finding pertinent prior case documents, summarising legal text, and so on. We strive to create methods that can assist the average person with legal questions, as well as increase public awareness of the law.
CIRAL at FIRE 2023: Cross-Lingual Information Retrieval for African Languages

This paper provides a short overview of the CIRAL track at the Forum for Information Retrieval Evaluation (FIRE) 2023. CIRAL focused on cross-lingual information retrieval (CLIR) between English and four African languages which include Hausa, Somali, Swahili, and Yoruba. In a bid to promote CLIR research for African languages and curate a test collection for the languages, community evaluations were carried out via pooling. We briefly discuss details of the task, dataset, relevance assessment and results from the track in this paper.
CoLI@FIRE2023: Findings of Word-level Language Identification in Code-mixed Tulu Text

Word-level Language Identification (LI) task determines the language of each word in a given code-mixed sentence, where a sentence is made up of words belonging to more than one language at word/sub-word level. This task is explored to a greater extent in high-resource languages like Spanish, French, and German in a code-mixed context, whereas it is very less explored in a few under-resourced languages and not yet addressed in a few other languages. In view of this, "CoLI-Tunglish: Word-level Language Identification in Code-mixed Tulu Texts" shared task at Forum for Information Retrieval Evaluation (FIRE) 2023 invites researchers to develop learning models for Word-level LI in Code-mixed Tulu Texts. CoLI-Tunglish dataset consists of mixing of three languages (Tulu, Kannada, and English) at word/sub-word level with the objective of assigning one of seven predefined labels: Tulu, Kannada, English, Mixed (a combination of Tulu, Kannada, and/or English languages), Name, Location, and Other, to each word in a given sentence. This paper describes the overview of the methodology and results obtained by five distinct teams who submitted 10 different runs out of 14 registered teams. Among all the models submitted by the participants, the top-performing model obtained a macro F1 score of 0.81. The outcomes achieved by the participating teams indicate a promising direction for tackling word-level LI challenges in code-mixed Tulu text. These results offer valuable insights and potential solutions, opening the new avenues of research for advancements in linguistic technologies for code-mixed Tulu text.
L3Cube-MahaSocialNER: A Social Media based Marathi Named Entity Recognition Dataset and BERT models

This work introduces the L3Cube-MahaSocialNER dataset, the first and largest social media dataset specifically designed for Named Entity Recognition (NER) in the Marathi language. The dataset comprises 18,000 manually labeled sentences covering eight entity classes, addressing challenges posed by social media data, including non-standard language and informal idioms. Deep learning models, including CNN, LSTM, BiLSTM, and Transformer models, are evaluated on the individual dataset with IOB and non-IOB notations. The results demonstrate the effectiveness of these models in accurately recognizing named entities in Marathi informal text. The L3Cube-MahaSocialNER dataset offers user-centric information extraction and supports real-time applications, providing a valuable resource for public opinion analysis, news, and marketing on social media platforms. We also show that the zero-shot results of the regular NER model are poor on the social NER test set thus highlighting the need for more social NER datasets. The datasets and models are publicly available at https://github.com/l3cube-pune/MarathiNLP
Unleashing the Power of Large Language Models: A Hands-On Tutorial

LLMs have opened up possibilities for advancing the state-of-the-art in natural language processing (NLP). In this tutorial, we present the audience with an introduction to LLMs and the associated challenges. The tutorial is structured in the following manner. First, we provide a brief preface that outlines the fundamental principles of NLP, following which, we explore the area of distributional representation learning for NLP. Then, we delve into the essential component of transformer-based pretrained language models. We then follow this up with the concept of prompt learning or in-context learning (ICL) and discuss how it is emerging as a popular methodology replacing the conventional supervised learning workflow comprised of pretraining and fine-tuning. We outline the research challenges in ICL, which usually involves finding the correct set of examples and contexts for the purpose of guiding the LLM decoder towards effective predictions. Afterwards, a hands-on coding and demonstration session will be carried out to impart practical knowledge about LLMs and ICL to the tutorial participants.
Overview of the HASOC Subtracks at FIRE 2023: Detection of Hate Spans and Conversational Hate-Speech

The proliferation of hateful content on Social Media has drawn a lot of attention lately, which has prompted practitioners to develop systems that can recognize this kind of content automatically. The evaluation requires reliable benchmark data and evaluation techniques. This paper reports on two sub-tracks related to Hate Speech detection. The task ICHCL provided a dataset in code-mixed Hindi, which includes the conversational context of a tweet. In many cases, including the context is necessary for detecting the hatefulness of the tweet. The best submission reached an F1 measure of 0.8. The task HateNorm (Identification of Tokens Contributing to Explicit Hate in English by Span Detection) introduced a span annotated dataset and required systems to find a sequence of words relevant to the hatefulness of a post. The highest scoring system obtained a macro-F1 of 0.581.
A Comparative Analysis of Retrievability and PageRank Measures

The accessibility of documents within a collection holds a pivotal role in Information Retrieval, signifying the ease of locating specific content in a collection of documents. This accessibility can be achieved via two distinct avenues. The first is through some retrieval model using a keyword or other feature-based search, and the other is where a document can be navigated using links associated with them, if available. Metrics such as PageRank, Hub, and Authority illuminate the pathways through which documents can be discovered within the network of content while the concept of Retrievability is used to quantify the ease with which a document can be found by a retrieval model. In this paper, we compare these two perspectives, PageRank and retrievability, as they quantify the importance and discoverability of content in a corpus. Through empirical experimentation on benchmark datasets, we demonstrate a subtle similarity between retrievability and PageRank particularly distinguishable for larger datasets.
Indian Language Summarization at FIRE 2023

This abstract provides a short overview of the second edition of the shared task on Indian Language Summarization (ILSUM) organized at the 15th Forum for Information Retrieval Evaluation (FIRE 2023). A more detailed discussion of the approaches used by the participating teams is available in the track overview paper. ILSUM 2023 consisted of two sub-tasks. The first sub-task continued the summarization task from last year. Besides extending the datasets for Hindi, Gujarati and Indian English we also introduced Bengali this year. The second sub-task focused on detecting misinformation in machine-generated summaries. While the task saw an enthusiastic response in registrations, with 25 teams requesting the dataset, only 6 teams finally submitted the runs and 4 of them submitted the working notes.
CricGPT: A GPT-aided Question-Answering system for Cricket

Cricket is a popular sport with avid followers across the globe. This paper presents a cricket Question-Answering (QA) system, viz. CricGPT, that leverages the publicly available ESPNCricinfo API data, OpenAI language model, SQLite database, and EJS templates to provide users with a comprehensive and interactive platform for accessing and analyzing cricket statistics on players. The proposed system comprises two distinct functionalities: Field Search and Free Text Search. The Field Search page allows users to explore field-wise statistics through dropdown menus; while the Free Text Search page enables users to input free-text queries and receive accurate responses using the OpenAI language model. For the latter, the system dynamically constructs SQLite queries using OpenAI based on user selections and executes them on the ESPNCricinfo API dataset stored in the SQLite database. The retrieved data is then presented to the user using EJS templates, ensuring a visually appealing and structured representation of cricket statistics. This research contributes to the field of cricket analytics by offering an intuitive and efficient means of accessing and analyzing cricket data using GPT. To our knowledge, no prior work has deployed GPTs for a QA system in cricket. On a set comprising simple and complex free-text questions, the proposed system attains an accuracy of around 85%. Especially on complex questions, CricGPT handsomely outperforms state-of-the-art search systems like ChatGPT, Google and Bing.
Recent Advancements in Unbiased Learning to Rank

Since its inception, the field of unbiased learning to rank (ULTR) has remained very active and has seen several impactful advancements in recent years. This tutorial provides both an introduction to the core concepts of the field and an overview of recent advancements in its foundations along with several applications of its methods.
Efficiency of Large Language Models to scale up Ground Truth: Overview of the IRSE Track at Forum for Information Retrieval 2023

The Software Engineering Information Retrieval (IRSE) track aims to devise solutions for the automated evaluation of code comments within a machine learning framework, with labels generated by both humans and large language models. Within this track, there is a binary classification task: discerning comments as either useful or not useful. The dataset includes 9,048 pairs of code comments and surrounding code snippets drawn from open-source C-based projects on GitHub and an additional dataset generated by teams employing large language models. In total, 17 teams representing various universities and software companies have contributed 56 experiments. These experiments were assessed through quantitative metrics, primarily the F1-Score, and qualitative evaluations based on the features developed, the supervised learning models employed, and their respective hyperparameters. It is worth noting that labels generated by large language models introduce bias into the prediction model but lead to less over-fitted results.
Abstractive Summarization of Scientific Documents: Models and Evaluation Techniques

Text summarization refers to the procedure of condensing the key ideas present in a single document or a group of related documents. The fundamental advantage of text summarization is that it saves the reader’s time by extracting the most crucial information from lengthy documents. Textual information abounds in every study topic, with fresh research articles published on a daily basis. A scientific publication is typically preceded by an abstract that summarizes the paper. In recent times, there has been a growing emphasis on research highlights, which serve as a supplementary summary alongside the traditional abstract, distilling the key findings of the research paper. It helps the reader to quickly understand the primary contributions of the paper. Abstractive summaries, which more closely resemble human-written summaries, are preferred over extractive ones. Our primary objective is to generate highlights from a research paper using abstractive approaches. Automatic generation of multi-document summary is our second goal. Measuring the quality of the generated summary is a vital aspect of summarization systems, and, therefore, our research also focuses on metrics and methods to assess the quality of a summary. In this paper, we outline our proposed plan of research on models and evaluation techniques for scientific text summarization, emphasizing the difficulties involved and the need to confront and solve them.
Overview of MTIL Track at FIRE 2023: Machine Translation for Indian Languages

The objective of the MTIL track in FIRE 2023 was to encourage the development of Indian Language to Indian Language (IL-IL) Neural Machine Translation models. The languages covered in the track included Hindi, Gujarati, Kannada, Odia, Punjabi, Urdu, Telugu, Kashmiri, and Sindhi. The track consists of two tasks: (i) a General Translation Task and (ii) a Domain specific Translation Task with Governance and Healthcare being the chosen domains. For the listed languages, we proposed 12 diverse language directions for the general domain translation task and 8 each for healthcare and governance domains. Participants were encouraged to submit models for one or more language pairs. We witnessed the creation of 34 distinct models spanning various language pairs and domains. Model assessments were conducted using five evaluation metrics: BLEU, CHRF, CHRF++, TER, and COMET. The submitted model outputs were ranked based on the CHRF score.
Harnessing the Power of Prompt-based Techniques for Generating School-Level Questions using Large Language Models

Designing high-quality educational questions is a challenging and time-consuming task. In this work, we propose a novel approach that utilizes prompt-based techniques to generate descriptive and reasoning-based questions. However, current question-answering (QA) datasets are inadequate for conducting our experiments on prompt-based question generation (QG) in an educational setting. Therefore, we curate a new QG dataset called EduProbe for school-level subjects, by leveraging the rich content of NCERT textbooks. We carefully annotate this dataset as quadruples of 1) Context: a segment upon which the question is formed; 2) Long Prompt: a long textual cue for the question (i.e., a longer sequence of words or phrases, covering the main theme of the context); 3) Short Prompt: a short textual cue for the question (i.e., a condensed representation of the key information or focus of the context); 4) Question: a deep question that aligns with the context and is coherent with the prompts. We investigate several prompt-based QG methods by fine-tuning pre-trained transformer-based large language models (LLMs), namely PEGASUS, T5, MBART, and BART. Moreover, we explore the performance of two general-purpose pre-trained LLMs such as Text-Davinci-003 and GPT-3.5-Turbo without any further training. By performing automatic evaluation, we show that T5 (with long prompt) outperforms all other models, but still falls short of the human baseline. Under human evaluation criteria, Text-Davinci-003 usually shows better results than other models under various prompt settings. Even in the case of human evaluation criteria, QG models mostly fall short of the human baseline. Our code and dataset are available at: https://github.com/my625/PromptQG
A Personal Knowledge Graph for Researchers

A researcher’s work entails various responsibilities. Managing all the information related to a researcher’s activities can be a demanding task that takes up a considerable amount of time and energy of the researcher. Knowledge graphs (KGs) are effective ways of organizing related information as knowledge. But instead of generalised KGs, personal knowledge graphs (PKGs) are more suitable to store the knowledge that is relevant to an individual only. This doctoral work focuses on assisting researchers by developing a PKG specifically for them and a virtual personal assistant powered by the PKG. The goal of this work is to look into the construction, maintenance and evaluation of a personal research knowledge graph.
Acceptance of the use of e-learning systems during the COVID -19 pandemic: A case study of the librarians of Tehran Province Public Libraries

Acceptance of the use of e-learning systems during the COVID -19 pandemic: A case study of the librarians of Tehran Province Public Libraries
Investigating the factors affecting the willingness to continue using Instagram pages of public libraries in West Azerbaijan province during the COVID-19 pandemic

Investigating the factors affecting the willingness to continue using Instagram pages of public libraries in West Azerbaijan province during the COVID-19 pandemic
Illustrating the Effectiveness of Reading Plan with Family and Book Reading Circles on the Two Components of Reading Motivation and Literacy Rate of Students and Providing Solutions to Improve the Quality of This Plan

Illustrating the Effectiveness of Reading Plan with Family and Book Reading Circles on the Two Components of Reading Motivation and Literacy Rate of Students and Providing Solutions to Improve the Quality of This Plan
Leveraging Computer Vision to Establish a Correlation Between Eye-Gaze Estimation and Saccades

The purpose of this project was to develop a model that can detect pupils on a subject’s face and draw the graph of both pupils to determine if a subject has saccades. Current methods cannot accurately detect contours when the face is presented at an angle. To resolve this issue, subjects recorded a video while performing the Head Impulse Test (HIT). Next, a computer vision library, Opencv, extracted frames from the recorded video and detected the facial key point for the nose. The Multi-task Cascaded Convolutional Neural Network (MTCNN) extracted the face from the frame and generated contours for the eyes using a segmentation library. The largest contours on the mask were divided into two parts and their extreme bounding box points were identified using dilation, erosion, and blur. Our model rendered a video of both contours applied to the patient performing the HIT test. Also, the model generated two graphs, comparing each eye’s gaze with the pose estimation. There were two main outputs: the graph may resemble a y = -x line if a patient does not have saccades or the graph may be more distorted if the patient has saccades due to a sudden shift in eye-gaze movement. This study demonstrates a precise method of eye-gaze estimation and the detection of saccades. Further experiments will help to validate the notion that saccades are connected to neurological disorders. Future experiments may include more data during the training process and to quantitatively determine the accuracy of our model.