OpenRaiser/Intern-Atlas huggingface dataset:
Power Aware Container Placement in Cloud Computing with Affinity and Cubic Power Model
Venue: arXiv cs.DC 2024
Year: 2024

Modern data centres are increasingly adopting containers to enhance power and performance efficiency. These data centres consist of multiple heterogeneous machines, each equipped with varying amounts of resources such as CPU, I/O, memory, and network bandwidth. Data centers rent their resources to applications, which demand different amounts of resources and execute on machines for extended durations if the machines provide the demanded resources to the applications. Certain applications run efficiently on specific machines, referred to as system affinity between applications and machines. In contrast, others are incompatible with specific machines, referred to as anti-affinity between applications and machines. We consider that there are multiple applications, and data centers need to execute as many applications as possible. Data centers incur electricity based on CPU usage due to the execution of applications, with the cost being proportional to the cube of the total CPU usage. It is a challenging problem to place applications on the machines they have an affinity for while keeping the electricity cost in check. Our work addresses the placement problem of matching applications to machines to minimize overall electricity costs while maximizing the number of affinity pairs of machines and applications. We propose three solution approaches: (a) Power-Aware Placement (PAP): applications are placed on machines where power usage is minimized, (b) Affinity-Aware Placement (AAP): applications are placed on machines where affinity is maximized, (c) Combined Power-Affinity Placement (CPAAP): this approach integrates the benefits of both PAP and AAP. Our proposed approach improves the affinity satisfaction ratio by up to 4% while reducing the total system cost by up to 26% and improving the affinity payoff ratio by up to 37% compared to state-of-the-art approaches for real-life datasets.
A Weakly Supervised and Globally Explainable Learning Framework for Brain Tumor Segmentation
Venue: arXiv cs.CV 2024
Year: 2024

Machine-based brain tumor segmentation can help doctors make better diagnoses. However, the complex structure of brain tumors and expensive pixel-level annotations present challenges for automatic tumor segmentation. In this paper, we propose a counterfactual generation framework that not only achieves exceptional brain tumor segmentation performance without the need for pixel-level annotations, but also provides explainability. Our framework effectively separates class-related features from class-unrelated features of the samples, and generate new samples that preserve identity features while altering class attributes by embedding different class-related features. We perform topological data analysis on the extracted class-related features and obtain a globally explainable manifold, and for each abnormal sample to be segmented, a meaningful normal sample could be effectively generated with the guidance of the rule-based paths designed within the manifold for comparison for identifying the tumor regions. We evaluate our proposed method on two datasets, which demonstrates superior performance of brain tumor segmentation. The code is available at https://github.com/xrt11/tumor-segmentation.
CLIP4Sketch: Enhancing Sketch to Mugshot Matching through Dataset Augmentation using Diffusion Models
Venue: arXiv cs.CV 2024
Year: 2024

Forensic sketch-to-mugshot matching is a challenging task in face recognition, primarily hindered by the scarcity of annotated forensic sketches and the modality gap between sketches and photographs. To address this, we propose CLIP4Sketch, a novel approach that leverages diffusion models to generate a large and diverse set of sketch images, which helps in enhancing the performance of face recognition systems in sketch-to-mugshot matching. Our method utilizes Denoising Diffusion Probabilistic Models (DDPMs) to generate sketches with explicit control over identity and style. We combine CLIP and Adaface embeddings of a reference mugshot, along with textual descriptions of style, as the conditions to the diffusion model. We demonstrate the efficacy of our approach by generating a comprehensive dataset of sketches corresponding to mugshots and training a face recognition model on our synthetic data. Our results show significant improvements in sketch-to-mugshot matching accuracy over training on an existing, limited amount of real face sketch data, validating the potential of diffusion models in enhancing the performance of face recognition systems across modalities. We also compare our dataset with datasets generated using GAN-based methods to show its superiority.
Jacta: A Versatile Planner for Learning Dexterous and Whole-body Manipulation
Venue: arXiv cs.RO 2024
Year: 2024

Robotic manipulation is challenging due to discontinuous dynamics, as well as high-dimensional state and action spaces. Data-driven approaches that succeed in manipulation tasks require large amounts of data and expert demonstrations, typically from humans. Existing planners are restricted to specific systems and often depend on specialized algorithms for using demonstrations. Therefore, we introduce a flexible motion planner tailored to dexterous and whole-body manipulation tasks. Our planner creates readily usable demonstrations for reinforcement learning algorithms, eliminating the need for additional training pipeline complexities. With this approach, we can efficiently learn policies for complex manipulation tasks, where traditional reinforcement learning alone only makes little progress. Furthermore, we demonstrate that learned policies are transferable to real robotic systems for solving complex dexterous manipulation tasks. Project website: https://jacta-manipulation.github.io/
Designing the virtual CAT: A digital tool for algorithmic thinking assessment in compulsory education
Venue: arXiv cs.HC 2024
Year: 2024

Algorithmic thinking (AT) is a critical skill in today's digital society, and it is indispensable not only in computer science-related fields but also in everyday problem-solving. As a foundational component of digital education and literacy, fostering AT skills is increasingly relevant for all students and should become a standard part of compulsory education. However, successfully integrating AT into formal education requires effective teaching strategies and robust and scalable assessment procedures. In this paper, we present the design and development process of the virtual Cross Array Task (CAT), a digital adaptation of an unplugged assessment activity aimed at evaluating algorithmic skills in Swiss compulsory education. The development process followed iterative design cycles, incorporating expert evaluations to refine the tool's usability, accessibility and functionality. A participatory design study played a dual role in shaping the platform. First, it gathered valuable insights from end users, including students and teachers, to ensure the tool's relevance and practicality in classroom settings. Second, it facilitated the collection and preliminary analysis of data related to students' AT skills, providing an initial evaluation of the tool's assessment capabilities across various developmental stages. This was achieved through a pilot study involving a diverse group of students aged 4 to 12, spanning preschool to lower secondary school levels. The resulting instrument features multilingual support and includes both gesture-based and visual block-based programming interfaces, making it accessible to a broad range of learners. Findings from the pilot study demonstrate the platform's usability and accessibility, as well as its suitability for assessing AT skills, with preliminary results showing its ability to cater to diverse age groups and educational contexts.
Reconsidering Degeneration of Token Embeddings with Definitions for Encoder-based Pre-trained Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Learning token embeddings based on token co-occurrence statistics has proven effective for both pre-training and fine-tuning in natural language processing. However, recent studies have pointed out that the distribution of learned embeddings degenerates into anisotropy (i.e., non-uniform distribution), and even pre-trained language models (PLMs) suffer from a loss of semantics-related information in embeddings for low-frequency tokens. This study first analyzes the fine-tuning dynamics of encoder-based PLMs and demonstrates their robustness against degeneration. On the basis of this analysis, we propose DefinitionEMB, a method that utilizes definitions to re-construct isotropically distributed and semantics-related token embeddings for encoder-based PLMs while maintaining original robustness during fine-tuning. Our experiments demonstrate the effectiveness of leveraging definitions from Wiktionary to re-construct such embeddings for two encoder-based PLMs: RoBERTa-base and BART-large. Furthermore, the re-constructed embeddings for low-frequency tokens improve the performance of these models across various GLUE and four text summarization datasets.
Prompt Refinement or Fine-tuning? Best Practices for using LLMs in Computational Social Science Tasks
Venue: arXiv cs.CY 2024
Year: 2024

Large Language Models are expressive tools that enable complex tasks of text understanding within Computational Social Science. Their versatility, while beneficial, poses a barrier for establishing standardized best practices within the field. To bring clarity on the values of different strategies, we present an overview of the performance of modern LLM-based classification methods on a benchmark of 23 social knowledge tasks. Our results point to three best practices: select models with larger vocabulary and pre-training corpora; avoid simple zero-shot in favor of AI-enhanced prompting; fine-tune on task-specific data, and consider more complex forms instruction-tuning on multiple datasets only when only training data is more abundant.
Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulation
Venue: arXiv cs.RO 2024
Year: 2024

Humans possess a remarkable talent for flexibly alternating to different senses when interacting with the environment. Picture a chef skillfully gauging the timing of ingredient additions and controlling the heat according to the colors, sounds, and aromas, seamlessly navigating through every stage of the complex cooking process. This ability is founded upon a thorough comprehension of task stages, as achieving the sub-goal within each stage can necessitate the utilization of different senses. In order to endow robots with similar ability, we incorporate the task stages divided by sub-goals into the imitation learning process to accordingly guide dynamic multi-sensory fusion. We propose MS-Bot, a stage-guided dynamic multi-sensory fusion method with coarse-to-fine stage understanding, which dynamically adjusts the priority of modalities based on the fine-grained state within the predicted current stage. We train a robot system equipped with visual, auditory, and tactile sensors to accomplish challenging robotic manipulation tasks: pouring and peg insertion with keyway. Experimental results indicate that our approach enables more effective and explainable dynamic fusion, aligning more closely with the human fusion process than existing methods.
NOLO: Navigate Only Look Once
Venue: arXiv cs.CV 2024
Year: 2024

The in-context learning ability of Transformer models has brought new possibilities to visual navigation. In this paper, we focus on the video navigation setting, where an in-context navigation policy needs to be learned purely from videos in an offline manner, without access to the actual environment. For this setting, we propose Navigate Only Look Once (NOLO), a method for learning a navigation policy that possesses the in-context ability and adapts to new scenes by taking corresponding context videos as input without finetuning or re-training. To enable learning from videos, we first propose a pseudo action labeling procedure using optical flow to recover the action label from egocentric videos. Then, offline reinforcement learning is applied to learn the navigation policy. Through extensive experiments on different scenes both in simulation and the real world, we show that our algorithm outperforms baselines by a large margin, which demonstrates the in-context learning ability of the learned policy. For videos and more information, visit https://sites.google.com/view/nol0.
Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs
Venue: arXiv cs.CL 2024
Year: 2024

Humans spontaneously use increasingly efficient language as interactions progress, by adapting and forming ad-hoc conventions. This phenomenon has been studied extensively using reference games, showing properties of human language that go beyond relaying intents. It remains unexplored whether multimodal large language models (MLLMs) similarly increase communication efficiency during interactions, and what mechanisms they may adopt for this purpose. We introduce ICCA, an automated framework to evaluate such conversational adaptation as an in-context behavior in MLLMs. We evaluate several state-of-the-art MLLMs, and observe that while they may understand the increasingly efficient language of their interlocutor, they do not spontaneously make their own language more efficient over time. This latter ability can only be elicited in some models (e.g., GPT-4) with heavy-handed prompting. This shows that this property of linguistic interaction does not arise from current training regimes, even though it is a common hallmark of human language. ICCA is available at https://github.com/lil-lab/ICCA.
No Size Fits All: The Perils and Pitfalls of Leveraging LLMs Vary with Company Size
Venue: arXiv cs.CY 2024
Year: 2024

Large language models (LLMs) are playing a pivotal role in deploying strategic use cases across a range of organizations, from large pan-continental companies to emerging startups. The issues and challenges involved in the successful utilization of LLMs can vary significantly depending on the size of the organization. It is important to study and discuss these pertinent issues of LLM adaptation with a focus on the scale of the industrial concerns and brainstorm possible solutions and prospective directions. Such a study has not been prominently featured in the current research literature. In this study, we adopt a threefold strategy: first, we conduct a case study with industry practitioners to formulate the key research questions; second, we examine existing industrial publications to address these questions; and finally, we provide a practical guide for industries to utilize LLMs more efficiently. We release the GitHub\footnote{\url{https://github.com/vinayakcse/IndustrialLLMsPapers}} repository with the most recent papers in the field.
AI Act for the Working Programmer
Venue: arXiv cs.CY 2024
Year: 2024

The European AI Act is a new, legally binding instrument that will enforce certain requirements on the development and use of AI technology potentially affecting people in Europe. It can be expected that the stipulations of the Act, in turn, are going to affect the work of many software engineers, software testers, data engineers, and other professionals across the IT sector in Europe and beyond. The 113 articles, 180 recitals, and 13 annexes that make up the Act cover 144 pages. This paper aims at providing an aid for navigating the Act from the perspective of some professional in the software domain, termed "the working programmer", who feels the need to know about the stipulations of the Act.
A systematic review and analysis of the viability of virtual reality (VR) in construction work and education
Venue: arXiv cs.CY 2024
Year: 2024

This systematic review explores the viability of virtual reality (VR) technologies for enhancing learning outcomes and operational efficiency within the construction industry. This study evaluates the current integration of VR in construction education and practice. Employing the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, this review analyzed 36 peer-reviewed journal articles from databases such as the Web of Science, ERIC, and Scopus. The methodology focused on identifying, appraising, and synthesizing all relevant studies to assess the effectiveness of VR applications in construction-related fields. This review highlights that VR significantly enhances learning by providing immersive interactive simulations that improve the understanding of every complex construction process, such as structural elements or tunnel-boring machine operations. This review contributes by systematically compiling and evaluating evidence on using VR in construction, which has seen a limited comprehensive analysis. It provides practical examples of how VR can revolutionize education and work.
Amman City, Jordan: Toward a Sustainable City from the Ground Up
Venue: arXiv cs.CY 2024
Year: 2024

The idea of smart cities (SCs) has gained substantial attention in recent years. The SC paradigm aims to improve citizens' quality of life and protect the city's environment. As we enter the age of next-generation SCs, it is important to explore all relevant aspects of the SC paradigm. In recent years, the advancement of Information and Communication Technologies (ICT) has produced a trend of supporting daily objects with smartness, targeting to make human life easier and more comfortable. The paradigm of SCs appears as a response to the purpose of building the city of the future with advanced features. SCs still face many challenges in their implementation, but increasingly more studies regarding SCs are implemented. Nowadays, different cities are employing SC features to enhance services or the residents quality of life. This work provides readers with useful and important information about Amman Smart City.
Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization
Venue: arXiv cs.SD 2024
Year: 2024

In the digital age, the emergence of deepfakes and synthetic media presents a significant threat to societal and political integrity. Deepfakes based on multi-modal manipulation, such as audio-visual, are more realistic and pose a greater threat. Current multi-modal deepfake detectors are often based on the attention-based fusion of heterogeneous data streams from multiple modalities. However, the heterogeneous nature of the data (such as audio and visual signals) creates a distributional modality gap and poses a significant challenge in effective fusion and hence multi-modal deepfake detection. In this paper, we propose a novel multi-modal attention framework based on recurrent neural networks (RNNs) that leverages contextual information for audio-visual deepfake detection. The proposed approach applies attention to multi-modal multi-sequence representations and learns the contributing features among them for deepfake detection and localization. Thorough experimental validations on audio-visual deepfake datasets, namely FakeAVCeleb, AV-Deepfake1M, TVIL, and LAV-DF datasets, demonstrate the efficacy of our approach. Cross-comparison with the published studies demonstrates superior performance of our approach with an improved accuracy and precision by 3.47% and 2.05% in deepfake detection and localization, respectively. Thus, obtaining state-of-the-art performance. To facilitate reproducibility, the code and the datasets information is available at https://github.com/vcbsl/audiovisual-deepfake/.
Exploring the Frontiers of Energy Efficiency using Power Management at System Scale
Venue: arXiv cs.DC 2024
Year: 2024

In the face of surging power demands for exascale HPC systems, this work tackles the critical challenge of understanding the impact of software-driven power management techniques like Dynamic Voltage and Frequency Scaling (DVFS) and Power Capping. These techniques have been actively developed over the past few decades. By combining insights from GPU benchmarking to understand application power profiles, we present a telemetry data-driven approach for deriving energy savings projections. This approach has been demonstrably applied to the Frontier supercomputer at scale. Our findings based on three months of telemetry data indicate that, for certain resource-constrained jobs, significant energy savings (up to 8.5%) can be achieved without compromising performance. This translates to a substantial cost reduction, equivalent to 1438 MWh of energy saved. The key contribution of this work lies in the methodology for establishing an upper limit for these best-case scenarios and its successful application. This work sheds light on potential energy savings and empowers HPC professionals to optimize the power-performance trade-off within constrained power budgets, not only for the exascale era but also beyond.
Counterfactual Explanations for Medical Image Classification and Regression using Diffusion Autoencoder
Venue: arXiv cs.CV 2024
Year: 2024

Counterfactual explanations (CEs) aim to enhance the interpretability of machine learning models by illustrating how alterations in input features would affect the resulting predictions. Common CE approaches require an additional model and are typically constrained to binary counterfactuals. In contrast, we propose a novel method that operates directly on the latent space of a generative model, specifically a Diffusion Autoencoder (DAE). This approach offers inherent interpretability by enabling the generation of CEs and the continuous visualization of the model's internal representation across decision boundaries. Our method leverages the DAE's ability to encode images into a semantically rich latent space in an unsupervised manner, eliminating the need for labeled data or separate feature extraction models. We show that these latent representations are helpful for medical condition classification and the ordinal regression of severity pathologies, such as vertebral compression fractures (VCF) and diabetic retinopathy (DR). Beyond binary CEs, our method supports the visualization of ordinal CEs using a linear model, providing deeper insights into the model's decision-making process and enhancing interpretability. Experiments across various medical imaging datasets demonstrate the method's advantages in interpretability and versatility. The linear manifold of the DAE's latent space allows for meaningful interpolation and manipulation, making it a powerful tool for exploring medical image properties. Our code is available at https://doi.org/10.5281/zenodo.13859266.
MMPKUBase: A Comprehensive and High-quality Chinese Multi-modal Knowledge Graph
Venue: arXiv cs.CL 2024
Year: 2024

Multi-modal knowledge graphs have emerged as a powerful approach for information representation, combining data from different modalities such as text, images, and videos. While several such graphs have been constructed and have played important roles in applications like visual question answering and recommendation systems, challenges persist in their development. These include the scarcity of high-quality Chinese knowledge graphs and limited domain coverage in existing multi-modal knowledge graphs. This paper introduces MMPKUBase, a robust and extensive Chinese multi-modal knowledge graph that covers diverse domains, including birds, mammals, ferns, and more, comprising over 50,000 entities and over 1 million filtered images. To ensure data quality, we employ Prototypical Contrastive Learning and the Isolation Forest algorithm to refine the image data. Additionally, we have developed a user-friendly platform to facilitate image attribute exploration.