OpenRaiser/Intern-Atlas huggingface dataset:
A Multimodal Transformer for Live Streaming Highlight Prediction
Venue: arXiv cs.MM 2024
Year: 2024

Recently, live streaming platforms have gained immense popularity. Traditional video highlight detection mainly focuses on visual features and utilizes both past and future content for prediction. However, live streaming requires models to infer without future frames and process complex multimodal interactions, including images, audio and text comments. To address these issues, we propose a multimodal transformer that incorporates historical look-back windows. We introduce a novel Modality Temporal Alignment Module to handle the temporal shift of cross-modal signals. Additionally, using existing datasets with limited manual annotations is insufficient for live streaming whose topics are constantly updated and changed. Therefore, we propose a novel Border-aware Pairwise Loss to learn from a large-scale dataset and utilize user implicit feedback as a weak supervision signal. Extensive experiments show our model outperforms various strong baselines on both real-world scenarios and public datasets. And we will release our dataset and code to better assess this topic.
An investigation into the scientific landscape of the conversational and generative artificial intelligence, and human-chatbot interaction in education and research
Venue: arXiv cs.CY 2024
Year: 2024

Artificial intelligence (AI) as a disruptive technology is not new. However, its recent evolution, engineered by technological transformation, big data analytics, and quantum computing, produces conversational and generative AI (CGAI/GenAI) and human-like chatbots that disrupt conventional operations and methods in different fields. This study investigates the scientific landscape of CGAI and human-chatbot interaction/collaboration and evaluates use cases, benefits, challenges, and policy implications for multidisciplinary education and allied industry operations. The publications trend showed that just 4% (n=75) occurred during 2006-2018, while 2019-2023 experienced astronomical growth (n=1763 or 96%). The prominent use cases of CGAI (e.g., ChatGPT) for teaching, learning, and research activities occurred in computer science [multidisciplinary and AI] (32%), medical/healthcare (17%), engineering (7%), and business fields (6%). The intellectual structure shows strong collaboration among eminent multidisciplinary sources in business, Information Systems, and other areas. The thematic structure of SLP highlights prominent CGAI use cases, including improved user experience in human-computer interaction, computer programs/code generation, and systems creation. Widespread CGAI usefulness for teachers, researchers, and learners includes syllabi/course content generation, testing aids, and academic writing. The concerns about abuse and misuse (plagiarism, academic integrity, privacy violations) and issues about misinformation, danger of self-diagnoses, and patient privacy in medical/healthcare applications are prominent. Formulating strategies and policies to address potential CGAI challenges in teaching/learning and practice are priorities. Developing discipline-based automatic detection of GenAI contents to check abuse is proposed.
TreeSeg: Hierarchical Topic Segmentation of Large Transcripts
Venue: arXiv cs.CL 2024
Year: 2024

From organizing recorded videos and meetings into chapters, to breaking down large inputs in order to fit them into the context window of commoditized Large Language Models (LLMs), topic segmentation of large transcripts emerges as a task of increasing significance. Still, accurate segmentation presents many challenges, including (a) the noisy nature of the Automatic Speech Recognition (ASR) software typically used to obtain the transcripts, (b) the lack of diverse labeled data and (c) the difficulty in pin-pointing the ground-truth number of segments. In this work we present TreeSeg, an approach that combines off-the-shelf embedding models with divisive clustering, to generate hierarchical, structured segmentations of transcripts in the form of binary trees. Our approach is robust to noise and can handle large transcripts efficiently. We evaluate TreeSeg on the ICSI and AMI corpora, demonstrating that it outperforms all baselines. Finally, we introduce TinyRec, a small-scale corpus of manually annotated transcripts, obtained from self-recorded video sessions.
Towards a Harms Taxonomy of AI Likeness Generation
Venue: arXiv cs.CY 2024
Year: 2024

Generative artificial intelligence models, when trained on a sufficient number of a person's images, can replicate their identifying features in a photorealistic manner. We refer to this process as 'likeness generation'. Likeness-featuring synthetic outputs often present a person's likeness without their control or consent, and may lead to harmful consequences. This paper explores philosophical and policy issues surrounding generated likeness. It begins by offering a conceptual framework for understanding likeness generation by examining the novel capabilities introduced by generative systems. The paper then establishes a definition of likeness by tracing its historical development in legal literature. Building on this foundation, we present a taxonomy of harms associated with generated likeness, derived from a comprehensive meta-analysis of relevant literature. This taxonomy categorises harms into seven distinct groups, unified by shared characteristics. Utilising this taxonomy, we raise various considerations that need to be addressed for the deployment of appropriate mitigations. Given the multitude of stakeholders involved in both the creation and distribution of likeness, we introduce concepts such as indexical sufficiency, a distinction between generation and distribution, and harms as having a context-specific nature. This work aims to serve industry, policymakers, and future academic researchers in their efforts to address the societal challenges posed by likeness generation.
GameDevDojo -- An Educational Game for Teaching Game Development Concepts
Venue: arXiv cs.HC 2024
Year: 2024

Computer Science (CS) has experienced significant growth and diversification in recent decades. However, there is a lack of diversity in CS learning approaches. Traditional teaching methods and hands-on learning dominate this field, with limited use of playful and interactive learning methods such as educational games. This gap is particularly evident in game development as a subfield of CS. To address this problem, we present a game-based learning approach to teach foundational concepts for game development. The paper aims to expand the educational landscape within CSE, offering a unique and engaging platform for learners to explore the intricacies of game creation by integrating gamified learning strategies. In this paper, we investigate the user's learning experience and motivation, and the differences between traditional learning and game-based learning methods for teaching game development concepts. The study involves 57 participants in an AB test to assess learners' motivation, user experience, and learning outcomes. The results indicate a significantly increased learning outcome for the game-based learning approach, as well as higher motivation in learning game development concepts.
Multistep Brent Oil Price Forecasting with a Multi-Aspect Meta-heuristic Optimization and Ensemble Deep Learning Model
Venue: arXiv cs.LG 2024
Year: 2024

Accurate crude oil price forecasting is crucial for various economic activities, including energy trading, risk management, and investment planning. Although deep learning models have emerged as powerful tools for crude oil price forecasting, achieving accurate forecasts remains challenging. Deep learning models' performance is heavily influenced by hyperparameters tuning, and they are expected to perform differently under various circumstances. Furthermore, price volatility is also sensitive to external factors such as world events. To address these limitations, we propose a hybrid approach that integrates metaheuristic optimization with an ensemble of five widely used neural network architectures for time series forecasting. Unlike existing methods that apply metaheuristics to optimise hyperparameters within the neural network architecture, we exploit the GWO metaheuristic optimiser at four levels: feature selection, data preparation, model training, and forecast blending. The proposed approach has been evaluated for forecasting three-ahead days using real-world Brent crude oil price data, and the obtained results demonstrate that the proposed approach improves the forecasting performance measured using various benchmarks, achieving 0.000127 of MSE.
LLMs-in-the-loop Part-1: Expert Small AI Models for Bio-Medical Text Translation
Venue: arXiv cs.CL 2024
Year: 2024

Machine translation is indispensable in healthcare for enabling the global dissemination of medical knowledge across languages. However, complex medical terminology poses unique challenges to achieving adequate translation quality and accuracy. This study introduces a novel "LLMs-in-the-loop" approach to develop supervised neural machine translation models optimized specifically for medical texts. While large language models (LLMs) have demonstrated powerful capabilities, this research shows that small, specialized models trained on high-quality in-domain (mostly synthetic) data can outperform even vastly larger LLMs. Custom parallel corpora in six languages were compiled from scientific articles, synthetically generated clinical documents, and medical texts. Our LLMs-in-the-loop methodology employs synthetic data generation, rigorous evaluation, and agent orchestration to enhance performance. We developed small medical translation models using the MarianMT base model. We introduce a new medical translation test dataset to standardize evaluation in this domain. Assessed using BLEU, METEOR, ROUGE, and BERT scores on this test set, our MarianMT-based models outperform Google Translate, DeepL, and GPT-4-Turbo. Results demonstrate that our LLMs-in-the-loop approach, combined with fine-tuning high-quality, domain-specific data, enables specialized models to outperform general-purpose and some larger systems. This research, part of a broader series on expert small models, paves the way for future healthcare-related AI developments, including deidentification and bio-medical entity extraction models. Our study underscores the potential of tailored neural translation models and the LLMs-in-the-loop methodology to advance the field through improved data generation, evaluation, agent, and modeling techniques.
Molecular Topological Profile (MOLTOP) -- Simple and Strong Baseline for Molecular Graph Classification
Venue: arXiv cs.LG 2024
Year: 2024

We revisit the effectiveness of topological descriptors for molecular graph classification and design a simple, yet strong baseline. We demonstrate that a simple approach to feature engineering - employing histogram aggregation of edge descriptors and one-hot encoding for atomic numbers and bond types - when combined with a Random Forest classifier, can establish a strong baseline for Graph Neural Networks (GNNs). The novel algorithm, Molecular Topological Profile (MOLTOP), integrates Edge Betweenness Centrality, Adjusted Rand Index and SCAN Structural Similarity score. This approach proves to be remarkably competitive when compared to modern GNNs, while also being simple, fast, low-variance and hyperparameter-free. Our approach is rigorously tested on MoleculeNet datasets using fair evaluation protocol provided by Open Graph Benchmark. We additionally show out-of-domain generation capabilities on peptide classification task from Long Range Graph Benchmark. The evaluations across eleven benchmark datasets reveal MOLTOP's strong discriminative capabilities, surpassing the $1$-WL test and even $3$-WL test for some classes of graphs. Our conclusion is that descriptor-based baselines, such as the one we propose, are still crucial for accurately assessing advancements in the GNN domain.
Neural Passage Quality Estimation for Static Pruning
Venue: arXiv cs.IR 2024
Year: 2024

Neural networks -- especially those that use large, pre-trained language models -- have improved search engines in various ways. Most prominently, they can estimate the relevance of a passage or document to a user's query. In this work, we depart from this direction by exploring whether neural networks can effectively predict which of a document's passages are unlikely to be relevant to any query submitted to the search engine. We refer to this query-agnostic estimation of passage relevance as a passage's quality. We find that our novel methods for estimating passage quality allow passage corpora to be pruned considerably while maintaining statistically equivalent effectiveness; our best methods can consistently prune >25% of passages in a corpora, across various retrieval pipelines. Such substantial pruning reduces the operating costs of neural search engines in terms of computing resources, power usage, and carbon footprint -- both when processing queries (thanks to a smaller index size) and when indexing (lightweight models can prune low-quality passages prior to the costly dense or learned sparse encoding step). This work sets the stage for developing more advanced neural "learning-what-to-index" methods.
A UAV-assisted Wireless Localization Challenge on AERPAW
Venue: arXiv cs.NI 2024
Year: 2024

As wireless researchers are tasked to enable wireless communication as infrastructure in more dynamic aerial settings, there is a growing need for large-scale experimental platforms that provide realistic, reproducible, and reliable experimental validation. To bridge the research-to-implementation gap, the Aerial Experimentation and Research Platform for Advanced Wireless (AERPAW) offers open-source tools, reference experiments, and hardware to facilitate and evaluate the development of wireless research in controlled digital twin environments and live testbed flights. The inaugural AERPAW Challenge, "Find a Rover," was issued to spark collaborative efforts and test the platform's capabilities. The task involved localizing a narrowband wireless signal, with teams given ten minutes to find the "rover" within a twenty-acre area. By engaging in this exercise, researchers can validate the platform's value as a tool for innovation in wireless communications research within aerial robotics. This paper recounts the methods and experiences of the top three teams in automating and rapidly locating a wireless signal by automating and controlling an aerial drone in a realistic testbed scenario.
L2AI: lightweight three-factor authentication and authorization in IOMT blockchain-based environment
Venue: arXiv cs.CR 2024
Year: 2024

Medical Internet of Things (IoMT) is the next frontier in the digital revolution and is utilized in healthcare. In this context, IoT enables individuals to remotely manage their essential activities with minimal interaction. However, the limitations of network resources and the challenges of establishing a secure channel, as well as sharing and collecting sensitive information through an insecure public channel, pose security challenges for the medical IoT. This paper presents a lightweight multi-factor authentication and anonymous user authentication scheme to access real-time data in a blockchain-based environment. The scheme utilizes an insecure channel called L2AI. L2AI ensures security and efficiency while enhancing user anonymity through the use of pseudo-identity and dynamic indexing. The proposed method supports highly scalable systems with an efficient user registration process, allowing authenticated users to access both existing and newly added system entities without additional processes. Although the scheme is primarily designed for large systems, such as health infrastructure, it is also suitable for resource-constrained devices. The scheme relies on one-way cryptographic hashing functions and bitwise XOR operations. Additionally, a fuzzy mining algorithm is employed on the user side to verify the user's biometric information. L2AI adopts the "Real-Or-Random (ROR)" model for security proof and employs BAN logic for proof of authenticity. Formal security verification is conducted using the "Automatic Validation of Internet Security Protocols and Programs" (Proverif) tool, complemented by informal security analysis demonstrating the proper functionality of L2AI.
Dynamic Task Control Method of a Flexible Manipulator Using a Deep Recurrent Neural Network
Venue: arXiv cs.RO 2024
Year: 2024

The flexible body has advantages over the rigid body in terms of environmental contact thanks to its underactuation. On the other hand, when applying conventional control methods to realize dynamic tasks with the flexible body, there are two difficulties: accurate modeling of the flexible body and the derivation of intermediate postures to achieve the tasks. Learning-based methods are considered to be more effective than accurate modeling, but they require explicit intermediate postures. To solve these two difficulties at the same time, we developed a real-time task control method with a deep recurrent neural network named Dynamic Task Execution Network (DTXNET), which acquires the relationship among the control command, robot state including image information, and task state. Once the network is trained, only the target event and its timing are needed to realize a given task. To demonstrate the effectiveness of our method, we applied it to the task of Wadaiko (traditional Japanese drum) drumming as an example, and verified the best configuration of DTXNET.
Generalization of the Fano and Non-Fano Index Coding Instances
Venue: arXiv cs.IT 2024
Year: 2024

Matroid theory is fundamentally connected with index coding and network coding problems. In fact, the reliance of linear index coding and network coding rates on the characteristic of a field has been demonstrated by using the two well-known matroid instances, namely the Fano and non-Fano matroids. This established the insufficiency of linear coding, one of the fundamental theorems in both index coding and network coding. While the Fano matroid is linearly representable only over fields with characteristic two, the non-Fano instance is linearly representable only over fields with odd characteristic. For fields with arbitrary characteristic $p$, the Fano and non-Fano matroids were extended to new classes of matroid instances whose linear representations are dependent on fields with characteristic $p$. However, these matroids have not been well appreciated nor cited in the fields of network coding and index coding. In this paper, we first reintroduce these matroids in a more structured way. Then, we provide a completely independent alternative proof with the main advantage of using only matrix manipulation rather than complex concepts in number theory and matroid theory. In this new proof, it is shown that while the class $p$-Fano matroid instances are linearly representable only over fields with characteristic $p$, the class $p$-non-Fano instances are representable over fields with any characteristic other than characteristic $p$. Finally, following the properties of the class $p$-Fano and $p$-non-Fano matroid instances, we characterize two new classes of index coding instances, respectively, referred to as the class $p$-Fano and $p$-non-Fano index coding, each with a size of $p^2 + 4p + 3$.
Delay Tradeoff and Adaptive Finite Blocklength Framework for URLLC
Venue: arXiv cs.IT 2024
Year: 2024

With various time-sensitive tasks to be served, ultra-reliable and low-latency communications (URLLC) has become one of the most important scenarios for the fifth generation (5G) wireless communications. The end-to-end delay from the sub-millisecond-level to the second-level is first put forward for a wide range of delay-sensitive tasks in the future sixth generation (6G) communication networks, which imposes a strict requirement on satisfying various real-time services and applications with different stringent quality of service (QoS) demands. Thus, we need to find out new delay reduction framework to satisfy the more stringent delay requirements. In this article, a state-of-the-art overview of end-to-end delay composition and delay analysis combined with access protocols are elaborated. We reveal the tradeoff relationship among transmission delay, queuing delay, and retransmission times with the change of blocklength in the finite blocklength (FBL) regime. Based on the tradeoff and combining the grant-free (GF) random access (RA) scheme, we propose the adaptive blocklength framework and investigate several effective algorithms for efficiently reducing the over-the-air delay. Numerical results show that our proposed framework and schemes can significantly reduce the over-the-air delay for URLLC.
Explaining Deep Neural Networks by Leveraging Intrinsic Methods
Venue: arXiv cs.LG 2024
Year: 2024

Despite their impact on the society, deep neural networks are often regarded as black-box models due to their intricate structures and the absence of explanations for their decisions. This opacity poses a significant challenge to AI systems wider adoption and trustworthiness. This thesis addresses this issue by contributing to the field of eXplainable AI, focusing on enhancing the interpretability of deep neural networks. The core contributions lie in introducing novel techniques aimed at making these networks more interpretable by leveraging an analysis of their inner workings. Specifically, the contributions are threefold. Firstly, the thesis introduces designs for self-explanatory deep neural networks, such as the integration of external memory for interpretability purposes and the usage of prototype and constraint-based layers across several domains. Secondly, this research delves into novel investigations on neurons within trained deep neural networks, shedding light on overlooked phenomena related to their activation values. Lastly, the thesis conducts an analysis of the application of explanatory techniques in the field of visual analytics, exploring the maturity of their adoption and the potential of these systems to convey explanations to users effectively.
COKE: Causal Discovery with Chronological Order and Expert Knowledge in High Proportion of Missing Manufacturing Data
Venue: arXiv cs.LG 2024
Year: 2024

Understanding causal relationships between machines is crucial for fault diagnosis and optimization in manufacturing processes. Real-world datasets frequently exhibit up to 90% missing data and high dimensionality from hundreds of sensors. These datasets also include domain-specific expert knowledge and chronological order information, reflecting the recording order across different machines, which is pivotal for discerning causal relationships within the manufacturing data. However, previous methods for handling missing data in scenarios akin to real-world conditions have not been able to effectively utilize expert knowledge. Conversely, prior methods that can incorporate expert knowledge struggle with datasets that exhibit missing values. Therefore, we propose COKE to construct causal graphs in manufacturing datasets by leveraging expert knowledge and chronological order among sensors without imputing missing data. Utilizing the characteristics of the recipe, we maximize the use of samples with missing values, derive embeddings from intersections with an initial graph that incorporates expert knowledge and chronological order, and create a sensor ordering graph. The graph-generating process has been optimized by an actor-critic architecture to obtain a final graph that has a maximum reward. Experimental evaluations in diverse settings of sensor quantities and missing proportions demonstrate that our approach compared with the benchmark methods shows an average improvement of 39.9% in the F1-score. Moreover, the F1-score improvement can reach 62.6% when considering the configuration similar to real-world datasets, and 85.0% in real-world semiconductor datasets. The source code is available at https://github.com/OuTingYun/COKE.