OpenRaiser/Intern-Atlas huggingface dataset:
VisioBlend: Sketch and Stroke-Guided Denoising Diffusion Probabilistic Model for Realistic Image Generation
Venue: arXiv cs.CV 2024
Year: 2024

Generating images from hand-drawings is a crucial and fundamental task in content creation. The translation is challenging due to the infinite possibilities and the diverse expectations of users. However, traditional methods are often limited by the availability of training data. Therefore, VisioBlend, a unified framework supporting three-dimensional control over image synthesis from sketches and strokes based on diffusion models, is proposed. It enables users to decide the level of faithfulness to the input strokes and sketches. VisioBlend achieves state-of-the-art performance in terms of realism and flexibility, enabling various applications in image synthesis from sketches and strokes. It solves the problem of data availability by synthesizing new data points from hand-drawn sketches and strokes, enriching the dataset and enabling more robust and diverse image synthesis. This work showcases the power of diffusion models in image creation, offering a user-friendly and versatile approach for turning artistic visions into reality.
Fréchet Distance in Subquadratic Time
Venue: arXiv cs.CG 2024
Year: 2024

Let $m$ and $n$ be the numbers of vertices of two polygonal curves in $\mathbb{R}^d$ for any fixed $d$ such that $m \leq n$. Since it was known in 1995 how to compute the Fréchet distance of these two curves in $O(mn\log (mn))$ time, it has been an open problem whether the running time can be reduced to $o(n^2)$ when $m = Ω(n)$. In the mean time, several well-known quadratic time barriers in computational geometry have been overcome: 3SUM, some 3SUM-hard problems, and the computation of some distances between two polygonal curves, including the discrete Fréchet distance, the dynamic time warping distance, and the geometric edit distance. It is curious that the quadratic time barrier for Fréchet distance still stands. We present an algorithm to compute the Fréchet distance in $O(mn(\log\log n)^{2+μ}\log n/\log^{1+μ} m)$ expected time for some constant $μ\in (0,1)$. It is the first algorithm that returns the Fréchet distance in $o(mn)$ time when $m = Ω(n^{\varepsilon})$ for any fixed $\varepsilon \in (0,1]$.
An efficient pipeline to compute patient-specific cerebral aneurysm wall tension
Venue: arXiv cs.CE 2024
Year: 2024

Cerebral aneurysm rupture, leading to subarachnoid hemorrhage with a high mortality rate, disproportionately affects younger populations, resulting in a significant loss of productive life years. A significant proportion of these deaths is due to aneurysmal re-bleeding within the first three days following the initial bleed, prior to treatment. While early aneurysm treatment is recommended, there is no consensus on the ideal timing, and emergency treatment offers only an incremental benefit at a significant cost. Although various multivariable prediction models have been proposed to provide personalized risk assessments, no validated patient-specific predictor is available to rationalize emergency treatment. Furthermore, no model has yet incorporated emerging computational biomechanics-based biomarkers such as wall tension. In this paper, we proposed and validated an efficient semi-automatic pipeline to compute patient-specific cerebral aneurysm wall tension as a potential biomarker for the likelihood of re-bleeding. Our pipeline uses the patient's computed tomography angiography (CTA) image obtained at the time of subarachnoid hemorrhage diagnosis to create a patient-specific biomechanical model of the cerebral aneurysm using the finite element method. A distinctive feature of our approach is the straightforward model creation and wall tension computation using shell finite elements, without requiring patient-specific material properties or aneurysm wall thickness. Our non-invasive, patient-specific method for cerebral aneurysm wall tension can potentially provide individualized risk prediction and enhance clinical decision-making.
RIS-assisted Coverage Enhancement in mmWave Integrated Sensing and Communication Networks
Venue: arXiv cs.IT 2024
Year: 2024

Integrated sensing and communication (ISAC) has emerged as a promising technology to facilitate high-rate communications and super-resolution sensing, particularly operating in the millimeter wave (mmWave) band. However, the vulnerability of mmWave signals to blockages severely impairs ISAC capabilities and coverage. To tackle this, an efficient and low-cost solution is to deploy distributed reconfigurable intelligent surfaces (RISs) to construct virtual links between the base stations (BSs) and users in a controllable fashion. In this paper, we investigate the generalized RIS-assisted mmWave ISAC networks considering the blockage effect, and examine the beneficial impact of RISs on the coverage rate utilizing stochastic geometry. Specifically, taking into account the coupling effect of ISAC dual functions within the same network topology, we derive the conditional coverage probability of ISAC performance for two association cases, based on the proposed beam pattern model and user association policies. Then, the marginal coverage rate is calculated by combining these two cases through the distance-dependent thinning method. Simulation results verify the accuracy of derived theoretical formulations and provide valuable guidelines for the practical network deployment. Specifically, our results indicate the superiority of the RIS deployment with the density of 40 km${}^{-2}$ BSs, and that the joint coverage rate of ISAC performance exhibits potential growth from $67.1\%$ to $92.2\%$ with the deployment of RISs.
Federated Knowledge Transfer Fine-tuning Large Server Model with Resource-Constrained IoT Clients
Venue: arXiv cs.LG 2024
Year: 2024

The training of large models, involving fine-tuning, faces the scarcity of high-quality data. Compared to the solutions based on centralized data centers, updating large models in the Internet of Things (IoT) faces challenges in coordinating knowledge from distributed clients by using their private and heterogeneous data. To tackle such a challenge, we propose KOALA (Federated Knowledge Transfer Fine-tuning Large Server Model with Resource-Constrained IoT Clients) to impel the training of large models in IoT. Since the resources obtained by IoT clients are limited and restricted, it is infeasible to locally execute large models and also update them in a privacy-preserving manner. Therefore, we leverage federated learning and knowledge distillation to update large models through collaboration with their small models, which can run locally at IoT clients to process their private data separately and enable large-small model knowledge transfer through iterative learning between the server and clients. Moreover, to support clients with similar or different computing capacities, KOALA is designed with two kinds of large-small model joint learning modes, namely to be homogeneous or heterogeneous. Experimental results demonstrate that compared to the conventional approach, our method can not only achieve similar training performance but also significantly reduce the need for local storage and computing power resources.
Understanding and Addressing Gender Bias in Expert Finding Task
Venue: arXiv cs.IR 2024
Year: 2024

The Expert Finding (EF) task is critical in community Question&Answer (CQ&A) platforms, significantly enhancing user engagement by improving answer quality and reducing response times. However, biases, especially gender biases, have been identified in these platforms. This study investigates gender bias in state-of-the-art EF models and explores methods to mitigate it. Utilizing a comprehensive dataset from StackOverflow, the largest community in the StackExchange network, we conduct extensive experiments to analyze how EF models' candidate identification processes influence gender representation. Our findings reveal that models relying on reputation metrics and activity levels disproportionately favor male users, who are more active on the platform. This bias results in the underrepresentation of female experts in the ranking process. We propose adjustments to EF models that incorporate a more balanced preprocessing strategy and leverage content-based and social network-based information, with the aim to provide a fairer representation of genders among identified experts. Our analysis shows that integrating these methods can significantly enhance gender balance without compromising model accuracy. To the best of our knowledge, this study is the first to focus on detecting and mitigating gender bias in EF methods.
The Switch, the Ladder, and the Matrix: Models for Classifying AI Systems
Venue: arXiv cs.CY 2024
Year: 2024

Organisations that design and deploy artificial intelligence (AI) systems increasingly commit themselves to high-level, ethical principles. However, there still exists a gap between principles and practices in AI ethics. One major obstacle organisations face when attempting to operationalise AI Ethics is the lack of a well-defined material scope. Put differently, the question to which systems and processes AI ethics principles ought to apply remains unanswered. Of course, there exists no universally accepted definition of AI, and different systems pose different ethical challenges. Nevertheless, pragmatic problem-solving demands that things should be sorted so that their grouping will promote successful actions for some specific end. In this article, we review and compare previous attempts to classify AI systems for the purpose of implementing AI governance in practice. We find that attempts to classify AI systems found in previous literature use one of three mental model. The Switch, i.e., a binary approach according to which systems either are or are not considered AI systems depending on their characteristics. The Ladder, i.e., a risk-based approach that classifies systems according to the ethical risks they pose. And the Matrix, i.e., a multi-dimensional classification of systems that take various aspects into account, such as context, data input, and decision-model. Each of these models for classifying AI systems comes with its own set of strengths and weaknesses. By conceptualising different ways of classifying AI systems into simple mental models, we hope to provide organisations that design, deploy, or regulate AI systems with the conceptual tools needed to operationalise AI governance in practice.
Image-Conditional Diffusion Transformer for Underwater Image Enhancement
Venue: arXiv cs.CV 2024
Year: 2024

Underwater image enhancement (UIE) has attracted much attention owing to its importance for underwater operation and marine engineering. Motivated by the recent advance in generative models, we propose a novel UIE method based on image-conditional diffusion transformer (ICDT). Our method takes the degraded underwater image as the conditional input and converts it into latent space where ICDT is applied. ICDT replaces the conventional U-Net backbone in a denoising diffusion probabilistic model (DDPM) with a transformer, and thus inherits favorable properties such as scalability from transformers. Furthermore, we train ICDT with a hybrid loss function involving variances to achieve better log-likelihoods, which meanwhile significantly accelerates the sampling process. We experimentally assess the scalability of ICDTs and compare with prior works in UIE on the Underwater ImageNet dataset. Besides good scaling properties, our largest model, ICDT-XL/2, outperforms all comparison methods, achieving state-of-the-art (SOTA) quality of image enhancement.
CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens
Venue: arXiv cs.SD 2024
Year: 2024

Recent years have witnessed a trend that large language model (LLM) based text-to-speech (TTS) emerges into the mainstream due to their high naturalness and zero-shot capacity. In this paradigm, speech signals are discretized into token sequences, which are modeled by an LLM with text as prompts and reconstructed by a token-based vocoder to waveforms. Obviously, speech tokens play a critical role in LLM-based TTS models. Current speech tokens are learned in an unsupervised manner, which lacks explicit semantic information and alignment to the text. In this paper, we propose to represent speech with supervised semantic tokens, which are derived from a multilingual speech recognition model by inserting vector quantization into the encoder. Based on the tokens, we further propose a scalable zero-shot TTS synthesizer, CosyVoice, which consists of an LLM for text-to-token generation and a conditional flow matching model for token-to-speech synthesis. Experimental results show that supervised semantic tokens significantly outperform existing unsupervised tokens in terms of content consistency and speaker similarity for zero-shot voice cloning. Moreover, we find that utilizing large-scale data further improves the synthesis performance, indicating the scalable capacity of CosyVoice. To the best of our knowledge, this is the first attempt to involve supervised speech tokens into TTS models.
FM-OSD: Foundation Model-Enabled One-Shot Detection of Anatomical Landmarks
Venue: arXiv cs.CV 2024
Year: 2024

One-shot detection of anatomical landmarks is gaining significant attention for its efficiency in using minimal labeled data to produce promising results. However, the success of current methods heavily relies on the employment of extensive unlabeled data to pre-train an effective feature extractor, which limits their applicability in scenarios where a substantial amount of unlabeled data is unavailable. In this paper, we propose the first foundation model-enabled one-shot landmark detection (FM-OSD) framework for accurate landmark detection in medical images by utilizing solely a single template image without any additional unlabeled data. Specifically, we use the frozen image encoder of visual foundation models as the feature extractor, and introduce dual-branch global and local feature decoders to increase the resolution of extracted features in a coarse to fine manner. The introduced feature decoders are efficiently trained with a distance-aware similarity learning loss to incorporate domain knowledge from the single template image. Moreover, a novel bidirectional matching strategy is developed to improve both robustness and accuracy of landmark detection in the case of scattered similarity map obtained by foundation models. We validate our method on two public anatomical landmark detection datasets. By using solely a single template image, our method demonstrates significant superiority over strong state-of-the-art one-shot landmark detection methods.
Multimodal Language Models for Domain-Specific Procedural Video Summarization
Venue: arXiv cs.CV 2024
Year: 2024

Videos serve as a powerful medium to convey ideas, tell stories, and provide detailed instructions, especially through long-format tutorials. Such tutorials are valuable for learning new skills at one's own pace, yet they can be overwhelming due to their length and dense content. Viewers often seek specific information, like precise measurements or step-by-step execution details, making it essential to extract and summarize key segments efficiently. An intelligent, time-sensitive video assistant capable of summarizing and detecting highlights in long videos is highly sought after. Recent advancements in Multimodal Large Language Models offer promising solutions to develop such an assistant. Our research explores the use of multimodal models to enhance video summarization and step-by-step instruction generation within specific domains. These models need to understand temporal events and relationships among actions across video frames. Our approach focuses on fine-tuning TimeChat to improve its performance in specific domains: cooking and medical procedures. By training the model on domain-specific datasets like Tasty for cooking and MedVidQA for medical procedures, we aim to enhance its ability to generate concise, accurate summaries of instructional videos. We curate and restructure these datasets to create high-quality video-centric instruction data. Our findings indicate that when finetuned on domain-specific procedural data, TimeChat can significantly improve the extraction and summarization of key instructional steps in long-format videos. This research demonstrates the potential of specialized multimodal models to assist with practical tasks by providing personalized, step-by-step guidance tailored to the unique aspects of each domain.
Towards Bridging the Cross-modal Semantic Gap for Multi-modal Recommendation
Venue: arXiv cs.IR 2024
Year: 2024

Multi-modal recommendation greatly enhances the performance of recommender systems by modeling the auxiliary information from multi-modality contents. Most existing multi-modal recommendation models primarily exploit multimedia information propagation processes to enrich item representations and directly utilize modal-specific embedding vectors independently obtained from upstream pre-trained models. However, this might be inappropriate since the abundant task-specific semantics remain unexplored, and the cross-modality semantic gap hinders the recommendation performance. Inspired by the recent progress of the cross-modal alignment model CLIP, in this paper, we propose a novel \textbf{CLIP} \textbf{E}nhanced \textbf{R}ecommender (\textbf{CLIPER}) framework to bridge the semantic gap between modalities and extract fine-grained multi-view semantic information. Specifically, we introduce a multi-view modality-alignment approach for representation extraction and measure the semantic similarity between modalities. Furthermore, we integrate the multi-view multimedia representations into downstream recommendation models. Extensive experiments conducted on three public datasets demonstrate the consistent superiority of our model over state-of-the-art multi-modal recommendation models.
A Manifesto for a Pro-Actively Responsible AI in Education
Venue: arXiv cs.CY 2024
Year: 2024

This paper examines the historical foundations, current practices, and emerging challenges for Artificial Intelligence in Education (AIED) within broader AI practices. It highlights AIED's unique and rich potential for contributing to the current AI policy and practices, especially in the context of responsible AI. It also discusses the key gaps in the AIED field, which need to be addressed by the community to elevate the field from a cottage industry to the level where it will deservedly be seen as key to advancin AI research and practical applications. The paper offers a five-point manifesto aimed to revitalise AIED' contributions to education and broader AI community, suggesting enhanced interdisciplinary collaboration, a broadened understanding of AI's impact on human functioning, and commitment to setting agendas for human-centred educational innovations.This approach positions AIED to significantly influence educational technologies to achieve genuine positive impact across diverse societal segments.
BiRoDiff: Diffusion policies for bipedal robot locomotion on unseen terrains
Venue: arXiv cs.RO 2024
Year: 2024

Locomotion on unknown terrains is essential for bipedal robots to handle novel real-world challenges, thus expanding their utility in disaster response and exploration. In this work, we introduce a lightweight framework that learns a single walking controller that yields locomotion on multiple terrains. We have designed a real-time robot controller based on diffusion models, which not only captures multiple behaviours with different velocities in a single policy but also generalizes well for unseen terrains. Our controller learns with offline data, which is better than online learning in aspects like scalability, simplicity in training scheme etc. We have designed and implemented a diffusion model-based policy controller in simulation on our custom-made Bipedal Robot model named Stoch BiRo. We have demonstrated its generalization capability and high frequency control step generation relative to typical generative models, which require huge onboarding compute.
AI in Manufacturing: Market Analysis and Opportunities
Venue: arXiv cs.AI 2024
Year: 2024

In this paper, we explore the transformative impact of Artificial Intelligence (AI) in the manufacturing sector, highlighting its potential to revolutionize industry practices and enhance operational efficiency. We delve into various applications of AI in manufacturing, with a particular emphasis on human-machine interfaces (HMI) and AI-powered milling machines, showcasing how these technologies contribute to more intuitive operations and precision in production processes. Through rigorous market analysis, the paper presents insightful data on AI adoption rates among German manufacturers, comparing these figures with global trends and exploring the specific uses of AI in production, maintenance, customer service, and more. In addition, the paper examines the emerging field of Generative AI and the potential applications of large language models in manufacturing processes. The findings indicate a significant increase in AI adoption from 6% in 2020 to 13.3% in 2023 among German companies, with a projection of substantial economic impact by 2030. The study also addresses the challenges faced by companies, such as data quality and integration hurdles, providing a balanced view of the opportunities and obstacles in AI implementation.
Enhancing Computer Programming Education with LLMs: A Study on Effective Prompt Engineering for Python Code Generation
Venue: arXiv cs.AI 2024
Year: 2024

Large language models (LLMs) and prompt engineering hold significant potential for advancing computer programming education through personalized instruction. This paper explores this potential by investigating three critical research questions: the systematic categorization of prompt engineering strategies tailored to diverse educational needs, the empowerment of LLMs to solve complex problems beyond their inherent capabilities, and the establishment of a robust framework for evaluating and implementing these strategies. Our methodology involves categorizing programming questions based on educational requirements, applying various prompt engineering strategies, and assessing the effectiveness of LLM-generated responses. Experiments with GPT-4, GPT-4o, Llama3-8b, and Mixtral-8x7b models on datasets such as LeetCode and USACO reveal that GPT-4o consistently outperforms others, particularly with the "multi-step" prompt strategy. The results show that tailored prompt strategies significantly enhance LLM performance, with specific strategies recommended for foundational learning, competition preparation, and advanced problem-solving. This study underscores the crucial role of prompt engineering in maximizing the educational benefits of LLMs. By systematically categorizing and testing these strategies, we provide a comprehensive framework for both educators and students to optimize LLM-based learning experiences. Future research should focus on refining these strategies and addressing current LLM limitations to further enhance educational outcomes in computer programming instruction.
Towards Perceived Security, Perceived Privacy, and the Universal Design of E-Payment Applications
Venue: arXiv cs.HC 2024
Year: 2024

With the growth of digital monetary transactions and cashless payments, encouraged by the COVID-19 pandemic, use of e-payment applications is on the rise. It is thus imperative to understand and evaluate the current posture of e-payment applications from three major user-facing angles: security, privacy, and usability. To this, we created a high-fidelity prototype of an e-payment application that encompassed features that we wanted to test with users. We then conducted a pilot study where we recruited 12 participants who tested our prototype. We find that both security and privacy are important for users of e-payment applications. Additionally, some participants perceive the strength of security and privacy based on the usability of the application. We provide recommendations such as universal design of e-payment applications.