OpenRaiser/Intern-Atlas huggingface dataset:
GELDA: A generative language annotation framework to reveal visual biases in datasets
Venue: arXiv cs.CV 2023
Year: 2023

Bias analysis is a crucial step in the process of creating fair datasets for training and evaluating computer vision models. The bottleneck in dataset analysis is annotation, which typically requires: (1) specifying a list of attributes relevant to the dataset domain, and (2) classifying each image-attribute pair. While the second step has made rapid progress in automation, the first has remained human-centered, requiring an experimenter to compile lists of in-domain attributes. However, an experimenter may have limited foresight leading to annotation "blind spots," which in turn can lead to flawed downstream dataset analyses. To combat this, we propose GELDA, a nearly automatic framework that leverages large generative language models (LLMs) to propose and label various attributes for a domain. GELDA takes a user-defined domain caption (e.g., "a photo of a bird," "a photo of a living room") and uses an LLM to hierarchically generate attributes. In addition, GELDA uses the LLM to decide which of a set of vision-language models (VLMs) to use to classify each attribute in images. Results on real datasets show that GELDA can generate accurate and diverse visual attribute suggestions, and uncover biases such as confounding between class labels and background features. Results on synthetic datasets demonstrate that GELDA can be used to evaluate the biases of text-to-image diffusion models and generative adversarial networks. Overall, we show that while GELDA is not accurate enough to replace human annotators, it can serve as a complementary tool to help humans analyze datasets in a cheap, low-effort, and flexible manner.
Zooming Out on Zooming In: Advancing Super-Resolution for Remote Sensing
Venue: arXiv cs.CV 2023
Year: 2023

Super-Resolution for remote sensing has the potential for huge impact on planet monitoring by producing accurate and realistic high resolution imagery on a frequent basis and a global scale. Despite a lot of attention, several inconsistencies and challenges have prevented it from being deployed in practice. These include the lack of effective metrics, fragmented and relatively small-scale datasets for training, insufficient comparisons across a suite of methods, and unclear evidence for the use of super-resolution outputs for machine consumption. This work presents a new metric for super-resolution, CLIPScore, that corresponds far better with human judgments than previous metrics on an extensive study. We use CLIPScore to evaluate four standard methods on a new large-scale dataset, S2-NAIP, and three existing benchmark datasets, and find that generative adversarial networks easily outperform more traditional L2 loss-based models and are more semantically accurate than modern diffusion models. We also find that using CLIPScore as an auxiliary loss can speed up the training of GANs by 18x and lead to improved outputs, resulting in an effective model in diverse geographies across the world which we will release publicly. The dataset, pre-trained model weights, and code are available at https://github.com/allenai/satlas-super-resolution/.
Leveraging a Randomized Key Matrix to Enhance the Security of Symmetric Substitution Ciphers
Venue: arXiv cs.CR 2023
Year: 2023

An innovative strategy to enhance the security of symmetric substitution ciphers is presented, through the implementation of a randomized key matrix suitable for various file formats, including but not limited to binary and text files. Despite their historical relevance, symmetric substitution ciphers have been limited by vulnerabilities to cryptanalytic methods like frequency analysis and known plaintext attacks. The aim of our research is to mitigate these vulnerabilities by employing a polyalphabetic substitution strategy that incorporates a distinct randomized key matrix. This matrix plays a pivotal role in generating a unique random key, comprising characters, encompassing both uppercase and lowercase letters, numeric, and special characters, to derive the corresponding ciphertext. The effectiveness of the proposed methodology in enhancing the security of conventional substitution methods for file encryption and decryption is supported by comprehensive testing and analysis, which encompass computational speed, frequency analysis, keyspace examination, Kasiski test, entropy analysis, and the utilization of a large language model.
Data-Driven Shape Sensing in Continuum Manipulators via Sliding Resistive Flex Sensors
Venue: arXiv cs.RO 2023
Year: 2023

We introduce a novel shape-sensing method using Resistive Flex Sensors (RFS) embedded in cable-driven Continuum Dexterous Manipulators (CDMs). The RFS is predominantly sensitive to deformation rather than direct forces, making it a distinctive tool for shape sensing. The RFS unit we designed is a considerably less expensive and robust alternative, offering comparable accuracy and real-time performance to existing shape sensing methods used for the CDMs proposed for minimally-invasive surgery. Our design allows the RFS to move along and inside the CDM conforming to its curvature, offering the ability to capture resistance metrics from various bending positions without the need for elaborate sensor setups. The RFS unit is calibrated using an overhead camera and a ResNet machine learning framework. Experiments using a 3D printed prototype of the CDM achieved an average shape estimation error of 0.968 mm with a standard error of 0.275 mm. The response time of the model was approximately 1.16 ms, making real-time shape sensing feasible. While this preliminary study successfully showed the feasibility of our approach for C-shape CDM deformations with non-constant curvatures, we are currently extending the results to show the feasibility for adapting to more complex CDM configurations such as S-shape created in obstructed environments or in presence of the external forces.
A-Scan2BIM: Assistive Scan to Building Information Modeling
Venue: arXiv cs.CV 2023
Year: 2023

This paper proposes an assistive system for architects that converts a large-scale point cloud into a standardized digital representation of a building for Building Information Modeling (BIM) applications. The process is known as Scan-to-BIM, which requires many hours of manual work even for a single building floor by a professional architect. Given its challenging nature, the paper focuses on helping architects on the Scan-to-BIM process, instead of replacing them. Concretely, we propose an assistive Scan-to-BIM system that takes the raw sensor data and edit history (including the current BIM model), then auto-regressively predicts a sequence of model editing operations as APIs of a professional BIM software (i.e., Autodesk Revit). The paper also presents the first building-scale Scan2BIM dataset that contains a sequence of model editing operations as the APIs of Autodesk Revit. The dataset contains 89 hours of Scan2BIM modeling processes by professional architects over 16 scenes, spanning over 35,000 m^2. We report our system's reconstruction quality with standard metrics, and we introduce a novel metric that measures how natural the order of reconstructed operations is. A simple modification to the reconstruction module helps improve performance, and our method is far superior to two other baselines in the order metric. We will release data, code, and models at a-scan2bim.github.io.
Odor Intensity Shift Keying (OISK) and Channel Capacity of Odor-based Molecular Communications in Internet of Everything
Venue: arXiv cs.ET 2023
Year: 2023

Molecular communication is a new, active area of research that has created a paradigm shift in the way a communication system is perceived. An artificial molecular communication network is created using biological molecules for encoding, transmitting and decoding the symbols to convey information. In addition to typical biological molecules, we are also exploring other classes of molecules that possess unique distinctive features which can be potentially exploited for establishing reliable communications. Odor molecules are one such class of molecules which possess several distinctive features such as Intensity, Headonic tone which provides a basis to convey the information in an olfactory communication system. In our work, we investigate the ICT (information and communication theory) perspective of the olfactory communications by evaluating the channel capacity of an odor molecular communication (OMC) system with the help of a novel modulation scheme viz. odor intensity shift keying (OISK), where information is being conveyed from the intensity level of an odor. Furthermore, we also analyse the effects of critical parameters like temperature and noise on the achievable channel capacity to provide an insight about the resilience of the proposed OMC system towards any such anomaly faced by it.
Whole-body Dynamic Collision Avoidance with Time-varying Control Barrier Functions
Venue: arXiv cs.RO 2023
Year: 2023

Recently, there has been increasing attention in robot research towards the whole-body collision avoidance. In this paper, we propose a safety-critical controller that utilizes time-varying control barrier functions (time varying CBFs) constructed by Robo-centric Euclidean Signed Distance Field (RC-ESDF) to achieve dynamic collision avoidance. The RC-ESDF is constructed in the robot body frame and solely relies on the robot's shape, eliminating the need for real-time updates to save computational resources. Additionally, we design two control Lyapunov functions (CLFs) to ensure that the robot can reach its destination. To enable real-time application, our safety-critical controller which incorporates CLFs and CBFs as constraints is formulated as a quadratic program (QP) optimization problem. We conducted numerical simulations on two different dynamics of an L-shaped robot to verify the effectiveness of our proposed approach.
Beyond Entropy: Style Transfer Guided Single Image Continual Test-Time Adaptation
Venue: arXiv cs.CV 2023
Year: 2023

Continual test-time adaptation (cTTA) methods are designed to facilitate the continual adaptation of models to dynamically changing real-world environments where computational resources are limited. Due to this inherent limitation, existing approaches fail to simultaneously achieve accuracy and efficiency. In detail, when using a single image, the instability caused by batch normalization layers and entropy loss significantly destabilizes many existing methods in real-world cTTA scenarios. To overcome these challenges, we present BESTTA, a novel single image continual test-time adaptation method guided by style transfer, which enables stable and efficient adaptation to the target environment by transferring the style of the input image to the source style. To implement the proposed method, we devise BeIN, a simple yet powerful normalization method, along with the style-guided losses. We demonstrate that BESTTA effectively adapts to the continually changing target environment, leveraging only a single image on both semantic segmentation and image classification tasks. Remarkably, despite training only two parameters in a BeIN layer consuming the least memory, BESTTA outperforms existing state-of-the-art methods in terms of performance.
Categorical Traffic Transformer: Interpretable and Diverse Behavior Prediction with Tokenized Latent
Venue: arXiv cs.LG 2023
Year: 2023

Adept traffic models are critical to both planning and closed-loop simulation for autonomous vehicles (AV), and key design objectives include accuracy, diverse multimodal behaviors, interpretability, and downstream compatibility. Recently, with the advent of large language models (LLMs), an additional desirable feature for traffic models is LLM compatibility. We present Categorical Traffic Transformer (CTT), a traffic model that outputs both continuous trajectory predictions and tokenized categorical predictions (lane modes, homotopies, etc.). The most outstanding feature of CTT is its fully interpretable latent space, which enables direct supervision of the latent variable from the ground truth during training and avoids mode collapse completely. As a result, CTT can generate diverse behaviors conditioned on different latent modes with semantic meanings while beating SOTA on prediction accuracy. In addition, CTT's ability to input and output tokens enables integration with LLMs for common-sense reasoning and zero-shot generalization.
Sharing Experience Around Component Compositions
Venue: arXiv cs.HC 2023
Year: 2023

Society currently lives in a world of tailorable systems in which end-users are able to transform their working environment while achieving their tasks, day to day and over the time. Tailorability is most of the time achieved through dynamic component integration thanks to a huge number of components available over the Internet. In this context, the main problem for users is not anymore the integration of new components, but how to find the most interesting set of components that will fulfill their needs. Facing this issue, the authors' assumption is that it would be helpful for users to take benefit of the experience of other users and our work aims at enhancing current software ecosystems to support this sharing of experience. The authors have applied this approach in the context of software development while considering Eclipse as one of the most advanced and used software ecosystem. The authors then offer ShareXP, an Eclipse feature that allows members of a group to share their expertise, this expertise being embodied in the ``compositions'' each of them has built. ShareXP was already presented in (Bourguin et al., 2012). The current paper is an extension where the authors deeper show that ShareXP is only a first step in their global approach trying to enhance not only the Eclipse ecosystem, but software ecosystems in general.
On Exact Inversion of DPM-Solvers
Venue: arXiv cs.CV 2023
Year: 2023

Diffusion probabilistic models (DPMs) are a key component in modern generative models. DPM-solvers have achieved reduced latency and enhanced quality significantly, but have posed challenges to find the exact inverse (i.e., finding the initial noise from the given image). Here we investigate the exact inversions for DPM-solvers and propose algorithms to perform them when samples are generated by the first-order as well as higher-order DPM-solvers. For each explicit denoising step in DPM-solvers, we formulated the inversions using implicit methods such as gradient descent or forward step method to ensure the robustness to large classifier-free guidance unlike the prior approach using fixed-point iteration. Experimental results demonstrated that our proposed exact inversion methods significantly reduced the error of both image and noise reconstructions, greatly enhanced the ability to distinguish invisible watermarks and well prevented unintended background changes consistently during image editing. Project page: \url{https://smhongok.github.io/inv-dpm.html}.
RainAI -- Precipitation Nowcasting from Satellite Data
Venue: arXiv cs.CV 2023
Year: 2023

This paper presents a solution to the Weather4Cast 2023 competition, where the goal is to forecast high-resolution precipitation with an 8-hour lead time using lower-resolution satellite radiance images. We propose a simple, yet effective method for spatiotemporal feature learning using a 2D U-Net model, that outperforms the official 3D U-Net baseline in both performance and efficiency. We place emphasis on refining the dataset, through importance sampling and dataset preparation, and show that such techniques have a significant impact on performance. We further study an alternative cross-entropy loss function that improves performance over the standard mean squared error loss, while also enabling models to produce probabilistic outputs. Additional techniques are explored regarding the generation of predictions at different lead times, specifically through Conditioning Lead Time. Lastly, to generate high-resolution forecasts, we evaluate standard and learned upsampling methods. The code and trained parameters are available at https://github.com/rafapablos/w4c23-rainai.
OpenMORE: an open-source tool for sampling-based path replanning in ROS
Venue: arXiv cs.RO 2023
Year: 2023

With the spread of robots in unstructured, dynamic environments, the topic of path replanning has gained importance in the robotics community. Although the number of replanning strategies has significantly increased, there is a lack of agreed-upon libraries and tools, making the use, development, and benchmarking of new algorithms arduous. This paper introduces OpenMORE, a new open-source ROS-based C++ library for sampling-based path replanning algorithms. The library builds a framework that allows for continuous replanning and collision checking of the traversed path during the execution of the robot trajectory. Users can solve replanning tasks exploiting the already available algorithms and can easily integrate new ones, leveraging the library to manage the entire execution.
Exploring the Temperature-Dependent Phase Transition in Modern Hopfield Networks
Venue: arXiv cs.LG 2023
Year: 2023

The recent discovery of a connection between Transformers and Modern Hopfield Networks (MHNs) has reignited the study of neural networks from a physical energy-based perspective. This paper focuses on the pivotal effect of the inverse temperature hyperparameter $β$ on the distribution of energy minima of the MHN. To achieve this, the distribution of energy minima is tracked in a simplified MHN in which equidistant normalised patterns are stored. This network demonstrates a phase transition at a critical temperature $β_{\text{c}}$, from a single global attractor towards highly pattern specific minima as $β$ is increased. Importantly, the dynamics are not solely governed by the hyperparameter $β$ but are instead determined by an effective inverse temperature $β_{\text{eff}}$ which also depends on the distribution and size of the stored patterns. Recognizing the role of hyperparameters in the MHN could, in the future, aid researchers in the domain of Transformers to optimise their initial choices, potentially reducing the necessity for time and energy expensive hyperparameter fine-tuning.
How Much Is Hidden in the NAS Benchmarks? Few-Shot Adaptation of a NAS Predictor
Venue: arXiv cs.LG 2023
Year: 2023

Neural architecture search has proven to be a powerful approach to designing and refining neural networks, often boosting their performance and efficiency over manually-designed variations, but comes with computational overhead. While there has been a considerable amount of research focused on lowering the cost of NAS for mainstream tasks, such as image classification, a lot of those improvements stem from the fact that those tasks are well-studied in the broader context. Consequently, applicability of NAS to emerging and under-represented domains is still associated with a relatively high cost and/or uncertainty about the achievable gains. To address this issue, we turn our focus towards the recent growth of publicly available NAS benchmarks in an attempt to extract general NAS knowledge, transferable across different tasks and search spaces. We borrow from the rich field of meta-learning for few-shot adaptation and carefully study applicability of those methods to NAS, with a special focus on the relationship between task-level correlation (domain shift) and predictor transferability; which we deem critical for improving NAS on diverse tasks. In our experiments, we use 6 NAS benchmarks in conjunction, spanning in total 16 NAS settings -- our meta-learning approach not only shows superior (or matching) performance in the cross-validation experiments but also successful extrapolation to a new search space and tasks.
ESPiM: Eye-Strain Probation Model, An Eye-Tracking Analysis Measure for Digital Displays
Venue: arXiv cs.HC 2023
Year: 2023

Eye-strain is a common issue among computer users due to the prolonged periods they spend working in front of digital displays. This can lead to vision problems, such as irritation and tiredness of the eyes and headaches. We propose the Eye-Strain Probation Model (ESPiM), a computational model based on eye-tracking data that measures eye-strain on digital displays based on the spatial properties of the user interface and display area for a required period of time. As well as measuring eye-strain, ESPiM can be applied to compare (a) different user interface designs, (b) different display devices, and (c) different interaction techniques. Two user studies were conducted to evaluate the effectiveness of ESPiM. The first was conducted in the form of an in-person study with an infrared eye-tracking sensor with 32 participants. The second was conducted in the form of an online study with a video-based eye-tracking technique via webcams on users' computers with 13 participants. Our analysis showed significantly different eye-strain patterns based on the video gameplay frequency of participants. Further, we found distinctive patterns among users on a regular 9-to-5 routine versus those with more flexible work hours in terms of (a) error rates and (b) reported eye-strain symptoms.
CLS-CAD: Synthesizing CAD Assemblies in Fusion 360
Venue: arXiv cs.RO 2023
Year: 2023

The CAD design process includes a number of repetitive steps when creating assemblies. This issue is compounded when engineering whole product lines or design families, as steps like inserting parts common to all variations, such as fasteners and product-integral base parts, get repeated numerous times. This makes creating designs time-, and as a result, cost-intensive. While many CAD software packages have APIs, the effort of creating use-case specific plugins to automate creation of assemblies usually outweighs the benefit. We developed a plugin for the CAD software package "Fusion 360" which tackles this issue. The plugin adds several graphical interfaces to Fusion 360 that allow parts to be annotated with types, subtype hierarchies to be managed, and requests to synthesize assembly programs for assemblies to be posed. The plugin is use-case agnostic and is able to generate arbitrary open kinematic chain structures. We envision engineers working with CAD software being able to make designed parts reusable and automate the generation of different design alternatives as well as whole product lines.
End-to-End Retrieval with Learned Dense and Sparse Representations Using Lucene
Venue: arXiv cs.IR 2023
Year: 2023

The bi-encoder architecture provides a framework for understanding machine-learned retrieval models based on dense and sparse vector representations. Although these representations capture parametric realizations of the same underlying conceptual framework, their respective implementations of top-$k$ similarity search require the coordination of different software components (e.g., inverted indexes, HNSW indexes, and toolkits for neural inference), often knitted together in complex architectures. In this work, we ask the following question: What's the simplest design, in terms of requiring the fewest changes to existing infrastructure, that can support end-to-end retrieval with modern dense and sparse representations? The answer appears to be that Lucene is sufficient, as we demonstrate in Anserini, a toolkit for reproducible information retrieval research. That is, effective retrieval with modern single-vector neural models can be efficiently performed directly in Java on the CPU. We examine the implications of this design for information retrieval researchers pushing the state of the art as well as for software engineers building production search systems.