OpenRaiser/Intern-Atlas huggingface dataset:
Estimating Environmental Cost Throughout Model's Adaptive Life Cycle
Venue: arXiv cs.CY 2024
Year: 2024

With the rapid increase in the research, development, and application of neural networks in the current era, there is a proportional increase in the energy needed to train and use models. Crucially, this is accompanied by the increase in carbon emissions into the environment. A sustainable and socially beneficial approach to reducing the carbon footprint and rising energy demands associated with the modern age of AI/deep learning is the adaptive and continuous reuse of models with regard to changes in the environment of model deployment or variations/changes in the input data. In this paper, we propose PreIndex, a predictive index to estimate the environmental and compute resources associated with model retraining to distributional shifts in data. PreIndex can be used to estimate environmental costs such as carbon emissions and energy usage when retraining from current data distribution to new data distribution. It also correlates with and can be used to estimate other resource indicators associated with deep learning, such as epochs, gradient norm, and magnitude of model parameter change. PreIndex requires only one forward pass of the data, following which it provides a single concise value to estimate resources associated with retraining to the new distribution shifted data. We show that PreIndex can be reliably used across various datasets, model architectures, different types, and intensities of distribution shifts. Thus, PreIndex enables users to make informed decisions for retraining to different distribution shifts and determine the most cost-effective and sustainable option, allowing for the reuse of a model with a much smaller footprint in the environment. The code for this work is available here: https://github.com/JEKimLab/AIES2024PreIndex
15 Years of Algorithmic Fairness -- Scoping Review of Interdisciplinary Developments in the Field
Venue: arXiv cs.CY 2024
Year: 2024

This paper presents a scoping review of algorithmic fairness research over the past fifteen years, utilising a dataset sourced from Web of Science, HEIN Online, FAccT and AIES proceedings. All articles come from the computer science and legal field and focus on AI algorithms with potential discriminatory effects on population groups. Each article is annotated based on their discussed technology, demographic focus, application domain and geographical context. Our analysis reveals a growing trend towards specificity in addressed domains, approaches, and demographics, though a substantial portion of contributions remains generic. Specialised discussions often concentrate on gender- or race-based discrimination in classification tasks. Regarding the geographical context of research, the focus is overwhelming on North America and Europe (Global North Countries), with limited representation from other regions. This raises concerns about overlooking other types of AI applications, their adverse effects on different types of population groups, and the cultural considerations necessary for addressing these problems. With the help of some highlighted works, we advocate why a wider range of topics must be discussed and why domain-, technological, diverse geographical and demographic-specific approaches are needed. This paper also explores the interdisciplinary nature of algorithmic fairness research in law and computer science to gain insight into how researchers from these fields approach the topic independently or in collaboration. By examining this, we can better understand the unique contributions that both disciplines can bring.
Future and AI-Ready Data Strategies: Response to DOC RFI on AI and Open Government Data Assets
Venue: arXiv cs.CY 2024
Year: 2024

The following is a response to the US Department of Commerce's Request for Information (RFI) regarding AI and Open Government Data Assets. First, we commend the Department for its initiative in seeking public insights on the organization and sharing of data. To facilitate scientific discovery and advance AI development, it is crucial for all data producers, including the Department of Commerce and other governmental entities, to prioritize the quality of their data corpora. Ensuring data is accessible, scalable, and secure is essential for harnessing its full potential. In our response, we outline best practices and key considerations for AI and the Department of Commerce's Open Government Data Assets.
Solid-State Oxide-Ion Synaptic Transistor for Neuromorphic Computing
Venue: arXiv cs.ET 2024
Year: 2024

Neuromorphic hardware facilitates rapid and energy-efficient training and operation of neural network models for artificial intelligence. However, existing analog in-memory computing devices, like memristors, continue to face significant challenges that impede their commercialization. These challenges include high variability due to their stochastic nature. Microfabricated electrochemical synapses offer a promising approach by functioning as an analog programmable resistor based on deterministic ion-insertion mechanisms. Here, we developed an all-solid-state oxide-ion synaptic transistor employing $\text{Bi}_2\text{V}_{0.9}\text{Cu}_{0.1}\text{O}_{5.35}$ as a superior oxide-ion conductor electrolyte and $\text{La}_\text{0.5}\text{Sr}_\text{0.5}\text{F}\text{O}_\text{3-$δ$}$ as a variable resistance channel able to efficiently operate at temperatures compatible with conventional electronics. Our transistor exhibits essential synaptic behaviors such as long- and short-term potentiation, paired-pulse facilitation, and post-tetanic potentiation, mimicking fundamental properties of biological neural networks. Key criteria for efficient neuromorphic computing are satisfied, including excellent linear and symmetric synaptic plasticity, low energy consumption per programming pulse, and high endurance with minimal cycle-to-cycle variation. Integrated into an artificial neural network (ANN) simulation for handwritten digit recognition, the presented synaptic transistor achieved a 96% accuracy on the MNIST dataset, illustrating the effective implementation of our device in ANNs. These findings demonstrate the potential of oxide-ion based synaptic transistors for effective implementation in analog neuromorphic computing based on iontronics.
From Program Logics to Language Logics
Venue: arXiv cs.PL 2024
Year: 2024

Program logics are a powerful formal method in the context of program verification. Can we develop a counterpart of program logics in the context of language verification? This paper proposes language logics, which allow for statements of the form $\{P\}\ \mathcal{X}\ \{Q\}$ where $\mathcal{X}$, the subject of analysis, can be a language component such as a piece of grammar, a typing rule, a reduction rule or other parts of a language definition. To demonstrate our approach, we develop $\mathbb{L}$, a language logic that can be used to analyze language definitions on various aspects of language design. We illustrate $\mathbb{L}$ to the analysis of some selected aspects of a programming language. We have also implemented an automated prover for $\mathbb{L}$, and we confirm that the tool repeats these analyses. Ultimately, $\mathbb{L}$ cannot verify languages. Nonetheless, we believe that this paper provides a strong first step towards adopting the methods of program logics for the analysis of languages.
Using LLMs to Establish Implicit User Sentiment of Software Desirability
Venue: arXiv cs.CL 2024
Year: 2024

This study explores the use of LLMs for providing quantitative zero-shot sentiment analysis of implicit software desirability, addressing a critical challenge in product evaluation where traditional review scores, though convenient, fail to capture the richness of qualitative user feedback. Innovations include establishing a method that 1) works with qualitative user experience data without the need for explicit review scores, 2) focuses on implicit user satisfaction, and 3) provides scaled numerical sentiment analysis, offering a more nuanced understanding of user sentiment, instead of simply classifying sentiment as positive, neutral, or negative. Data is collected using the Microsoft Product Desirability Toolkit (PDT), a well-known qualitative user experience analysis tool. For initial exploration, the PDT metric was given to users of two software systems. PDT data was fed through several LLMs (Claude Sonnet 3 and 3.5, GPT4, and GPT4o) and through a leading transfer learning technique, Twitter-Roberta-Base-Sentiment, and Vader, a leading sentiment analysis tool. Each system was asked to evaluate the data in two ways, by looking at the sentiment expressed in the PDT word/explanation pairs; and by looking at the sentiment expressed by the users in their grouped selection of five words and explanations, as a whole. Each LLM provided a sentiment score, its confidence (low, medium, high) in the score, and an explanation of the score. All LLMs tested were able to statistically detect user sentiment from the users' grouped data, whereas TRBS and Vader were not. The confidence and explanation of confidence provided by the LLMs assisted in understanding user sentiment. This study adds deeper understanding of evaluating user experiences, toward the goal of creating a universal tool that quantifies implicit sentiment.
Autonomous Integration of Bench-Top Wet Lab Equipment
Venue: arXiv cs.RO 2024
Year: 2024

Laboratory automation is an expensive and complicated endeavor with limited inflexible options for small-scale labs. We develop a prototype system for tending to a bench-top centrifuge using computer vision methods for color detection and circular Hough Transforms to detect and localize centrifuge buckets. Initial results show that the prototype is capable of automating the usage of regular bench-top lab equipment.
MAC Address De-Randomization Using Multi-Channel Sniffers and Two-Stage Clustering
Venue: arXiv cs.NI 2024
Year: 2024

MAC randomization is a widely used technique implemented on most modern smartphones to protect user's privacy against tracking based on Probe Request frames capture. However, there exist weaknesses in such a methodology which may still expose distinctive information, allowing to track the device generating the Probe Requests. Such techniques, known as MAC de-randomization algorithms, generally exploit Information Elements (IEs) contained in the Probe Requests and use clustering methodologies to group together frames belonging to the same device. While effective on heterogeneous device types, such techniques are not able to differentiate among devices of identical type and running the same Operating System (OS). In this paper, we propose a MAC de-randomization technique able to overcome such a weakness. First, we propose a new dataset of Probe Requests captured from devices sharing the same characteristics. Secondly, we observe that the time-frequency pattern of Probe Request emission is unique among devices and can therefore be used as a discriminative feature. We embed such a feature in a two-stage clustering methodology and show through experiments its effectiveness compared to state-of-the-art techniques based solely on IEs fingerprinting. The original dataset used in this work is made publicly available for reproducible research.
Soil Sample Search in Partially Observable Environments
Venue: arXiv cs.RO 2024
Year: 2024

To work in unknown outdoor environments, autonomous sampling machines need the ability to target samples despite limited visibility and robotic arm reach distance. We design a heuristic guided search method to speed up the search process and more efficiently localize the approximate center of soil regions. Through simulation experiments, we assess the effectiveness of the proposed algorithm and discover superior performance in terms of speed, distance traveled, and success rate compared to naive baselines.
Deep Learning Meets OBIA: Tasks, Challenges, Strategies, and Perspectives
Venue: arXiv cs.CV 2024
Year: 2024

Deep learning has gained significant attention in remote sensing, especially in pixel- or patch-level applications. Despite initial attempts to integrate deep learning into object-based image analysis (OBIA), its full potential remains largely unexplored. In this article, as OBIA usage becomes more widespread, we conducted a comprehensive review and expansion of its task subdomains, with or without the integration of deep learning. Furthermore, we have identified and summarized five prevailing strategies to address the challenge of deep learning's limitations in directly processing unstructured object data within OBIA, and this review also recommends some important future research directions. Our goal with these endeavors is to inspire more exploration in this fascinating yet overlooked area and facilitate the integration of deep learning into OBIA processing workflows.
Three-dimensional Morphological Reconstruction of Millimeter-Scale Soft Continuum Robots based on Dual-Stereo-Vision
Venue: arXiv cs.RO 2024
Year: 2024

Continuum robots can be miniaturized to just a few millimeters in diameter. Among these, notched tubular continuum robots (NTCR) show great potential in many delicate applications. Existing works in robotic modeling focus on kinematics and dynamics but still face challenges in reproducing the robot's morphology -- a significant factor that can expand the research landscape of continuum robots, especially for those with asymmetric continuum structures. This paper proposes a dual stereo vision-based method for the three-dimensional morphological reconstruction of millimeter-scale NTCRs. The method employs two oppositely located stationary binocular cameras to capture the point cloud of the NTCR, then utilizes predefined geometry as a reference for the KD tree method to relocate the capture point clouds, resulting in a morphologically correct NTCR despite the low-quality raw point cloud collection. The method has been proved feasible for an NTCR with a 3.5 mm diameter, capturing 14 out of 16 notch features, with the measurements generally centered around the standard of 1.5 mm, demonstrating the capability of revealing morphological details. Our proposed method paves the way for 3D morphological reconstruction of millimeter-scale soft robots for further self-modeling study.
Managing Human-Centric Software Defects: Insights from GitHub and Practitioners' Perspectives
Venue: arXiv cs.SE 2024
Year: 2024

Context: Human-centric defects (HCDs) are nuanced and subjective defects that often occur due to end-user perceptions or differences, such as their genders, ages, cultures, languages, disabilities, socioeconomic status, and educational backgrounds. Development teams have a limited understanding of these issues, which leads to the neglect of these defects. Defect reporting tools do not adequately handle the capture and fixing of HCDs. Objective: This research aims to understand the current defect reporting process and tools for managing defects. Our study aims to capture process flaws and create a preliminary defect categorisation and practices of a defect-reporting tool that can improve the reporting and fixing of HCDs in software engineering. Method: We first manually classified 1,100 open-source issues from the GitHub defect reporting tool to identify human-centric defects and to understand the categories of such reported defects. We then interviewed software engineering practitioners to elicit feedback on our findings from the GitHub defects analysis and gauge their knowledge and experience of the defect-reporting process and tools for managing human-centric defects. Results: We identified 176 HCDs from 1,100 open-source issues across six domains: IT-Healthcare, IT-Web, IT-Spatial, IT-Manufacturing, IT-Finance, and IT-Gaming. Additionally, we interviewed 15 software practitioners to identify shortcomings in the current defect reporting process and determine practices for addressing these weaknesses. Conclusion: HCDs present in open-source repositories are fairly technical, and due to the lack of awareness and improper defect reports, they present a major challenge to software practitioners. However, the management of HCDs can be enhanced by implementing the practices for an ideal defect reporting tool developed as part of this study.
Degrade to Function: Towards Eco-friendly Morphing Devices that Function Through Programmed Sequential Degradation
Venue: arXiv cs.HC 2024
Year: 2024

While it seems counterintuitive to think of degradation within an operating device as beneficial, one may argue that when rationally designed, the controlled breakdown of materials can be harnessed for specific functions. To apply this principle to the design of morphing devices, we introduce the concept of Degrade to Function (DtF). This concept aims to create eco-friendly and self-contained morphing devices that operate through a sequence of environmentally-triggered degradations. We explore its design considerations and implementation techniques by identifying environmental conditions and degradation types that can be exploited, evaluating potential materials capable of controlled degradation, suggesting designs for structures that can leverage degradation to achieve various transformations and functions, and developing sequential control approaches that integrate degradation triggers. To demonstrate the viability and versatility of this design strategy, we showcase several application examples across a range of environmental conditions.
SAT3D: Image-driven Semantic Attribute Transfer in 3D
Venue: arXiv cs.CV 2024
Year: 2024

GAN-based image editing task aims at manipulating image attributes in the latent space of generative models. Most of the previous 2D and 3D-aware approaches mainly focus on editing attributes in images with ambiguous semantics or regions from a reference image, which fail to achieve photographic semantic attribute transfer, such as the beard from a photo of a man. In this paper, we propose an image-driven Semantic Attribute Transfer method in 3D (SAT3D) by editing semantic attributes from a reference image. For the proposed method, the exploration is conducted in the style space of a pre-trained 3D-aware StyleGAN-based generator by learning the correlations between semantic attributes and style code channels. For guidance, we associate each attribute with a set of phrase-based descriptor groups, and develop a Quantitative Measurement Module (QMM) to quantitatively describe the attribute characteristics in images based on descriptor groups, which leverages the image-text comprehension capability of CLIP. During the training process, the QMM is incorporated into attribute losses to calculate attribute similarity between images, guiding target semantic transferring and irrelevant semantics preserving. We present our 3D-aware attribute transfer results across multiple domains and also conduct comparisons with classical 2D image editing methods, demonstrating the effectiveness and customizability of our SAT3D.
SynopGround: A Large-Scale Dataset for Multi-Paragraph Video Grounding from TV Dramas and Synopses
Venue: arXiv cs.CV 2024
Year: 2024

Video grounding is a fundamental problem in multimodal content understanding, aiming to localize specific natural language queries in an untrimmed video. However, current video grounding datasets merely focus on simple events and are either limited to shorter videos or brief sentences, which hinders the model from evolving toward stronger multimodal understanding capabilities. To address these limitations, we present a large-scale video grounding dataset named SynopGround, in which more than 2800 hours of videos are sourced from popular TV dramas and are paired with accurately localized human-written synopses. Each paragraph in the synopsis serves as a language query and is manually annotated with precise temporal boundaries in the long video. These paragraph queries are tightly correlated to each other and contain a wealth of abstract expressions summarizing video storylines and specific descriptions portraying event details, which enables the model to learn multimodal perception on more intricate concepts over longer context dependencies. Based on the dataset, we further introduce a more complex setting of video grounding dubbed Multi-Paragraph Video Grounding (MPVG), which takes as input multiple paragraphs and a long video for grounding each paragraph query to its temporal interval. In addition, we propose a novel Local-Global Multimodal Reasoner (LGMR) to explicitly model the local-global structures of long-term multimodal inputs for MPVG. Our method provides an effective baseline solution to the multi-paragraph video grounding problem. Extensive experiments verify the proposed model's effectiveness as well as its superiority in long-term multi-paragraph video grounding over prior state-of-the-arts. Dataset and code are publicly available. Project page: https://synopground.github.io/.
Prototyping of a multirotor UAV for precision landing under rotor failures
Venue: arXiv cs.RO 2024
Year: 2024

This work presents a prototype of a multirotor aerial vehicle capable of precision landing, even under the effects of rotor failures. The manuscript presents the fault-tolerant techniques and mechanical designs to achieve a fault-tolerant multirotor, and a vision-based navigation system required to achieve a precision landing. Preliminary experimental results will be shown, to validate on one hand the fault-tolerant control vehicle and, on the other hand, the autonomous landing algorithm. Also, a prototype of the fault-tolerant UAV is presented, capable of precise autonomous landing, which will be used in future experiments.
HIVE: HIerarchical Volume Encoding for Neural Implicit Surface Reconstruction
Venue: arXiv cs.CV 2024
Year: 2024

Neural implicit surface reconstruction has become a new trend in reconstructing a detailed 3D shape from images. In previous methods, however, the 3D scene is only encoded by the MLPs which do not have an explicit 3D structure. To better represent 3D shapes, we introduce a volume encoding to explicitly encode the spatial information. We further design hierarchical volumes to encode the scene structures in multiple scales. The high-resolution volumes capture the high-frequency geometry details since spatially varying features could be learned from different 3D points, while the low-resolution volumes enforce the spatial consistency to keep the shape smooth since adjacent locations possess the same low-resolution feature. In addition, we adopt a sparse structure to reduce the memory consumption at high-resolution volumes, and two regularization terms to enhance results smoothness. This hierarchical volume encoding could be appended to any implicit surface reconstruction method as a plug-and-play module, and can generate a smooth and clean reconstruction with more details. Superior performance is demonstrated in DTU, EPFL, and BlendedMVS datasets with significant improvement on the standard metrics.