OpenRaiser/Intern-Atlas huggingface dataset:
A Hybrid Approach To Real-Time Multi-Object Tracking
Venue: arXiv cs.CV 2023
Year: 2023

Multi-Object Tracking, also known as Multi-Target Tracking, is a significant area of computer vision that has many uses in a variety of settings. The development of deep learning, which has encouraged researchers to propose more and more work in this direction, has significantly impacted the scientific advancement around the study of tracking as well as many other domains related to computer vision. In fact, all of the solutions that are currently state-of-the-art in the literature and in the tracking industry, are built on top of deep learning methodologies that produce exceptionally good results. Deep learning is enabled thanks to the ever more powerful technology researchers can use to handle the significant computational resources demanded by these models. However, when real-time is a main requirement, developing a tracking system without being constrained by expensive hardware support with enormous computational resources is necessary to widen tracking applications in real-world contexts. To this end, a compromise is to combine powerful deep strategies with more traditional approaches to favor considerably lower processing solutions at the cost of less accurate tracking results even though suitable for real-time domains. Indeed, the present work goes in that direction, proposing a hybrid strategy for real-time multi-target tracking that combines effectively a classical optical flow algorithm with a deep learning architecture, targeted to a human-crowd tracking system exhibiting a desirable trade-off between performance in tracking precision and computational costs. The developed architecture was experimented with different settings, and yielded a MOTA of 0.608 out of the compared state-of-the-art 0.549 results, and about half the running time when introducing the optical flow phase, achieving almost the same performance in terms of accuracy.
A Probabilistic Approach to Self-Supervised Learning using Cyclical Stochastic Gradient MCMC
Venue: arXiv cs.LG 2023
Year: 2023

In this paper we present a practical Bayesian self-supervised learning method with Cyclical Stochastic Gradient Hamiltonian Monte Carlo (cSGHMC). Within this framework, we place a prior over the parameters of a self-supervised learning model and use cSGHMC to approximate the high dimensional and multimodal posterior distribution over the embeddings. By exploring an expressive posterior over the embeddings, Bayesian self-supervised learning produces interpretable and diverse representations. Marginalizing over these representations yields a significant gain in performance, calibration and out-of-distribution detection on a variety of downstream classification tasks. We provide experimental results on multiple classification tasks on four challenging datasets. Moreover, we demonstrate the effectiveness of the proposed method in out-of-distribution detection using the SVHN and CIFAR-10 datasets.
Evaluating the Robustness of Test Selection Methods for Deep Neural Networks
Venue: arXiv cs.LG 2023
Year: 2023

Testing deep learning-based systems is crucial but challenging due to the required time and labor for labeling collected raw data. To alleviate the labeling effort, multiple test selection methods have been proposed where only a subset of test data needs to be labeled while satisfying testing requirements. However, we observe that such methods with reported promising results are only evaluated under simple scenarios, e.g., testing on original test data. This brings a question to us: are they always reliable? In this paper, we explore when and to what extent test selection methods fail for testing. Specifically, first, we identify potential pitfalls of 11 selection methods from top-tier venues based on their construction. Second, we conduct a study on five datasets with two model architectures per dataset to empirically confirm the existence of these pitfalls. Furthermore, we demonstrate how pitfalls can break the reliability of these methods. Concretely, methods for fault detection suffer from test data that are: 1) correctly classified but uncertain, or 2) misclassified but confident. Remarkably, the test relative coverage achieved by such methods drops by up to 86.85%. On the other hand, methods for performance estimation are sensitive to the choice of intermediate-layer output. The effectiveness of such methods can be even worse than random selection when using an inappropriate layer.
An enhanced motion planning approach by integrating driving heterogeneity and long-term trajectory prediction for automated driving systems
Venue: arXiv cs.RO 2023
Year: 2023

Navigating automated driving systems (ADSs) through complex driving environments is difficult. Predicting the driving behavior of surrounding human-driven vehicles (HDVs) is a critical component of an ADS. This paper proposes an enhanced motion-planning approach for an ADS in a highway-merging scenario. The proposed enhanced approach utilizes the results of two aspects: the driving behavior and long-term trajectory of surrounding HDVs, which are coupled using a hierarchical model that is used for the motion planning of an ADS to improve driving safety.
What Is the Difference Between a Mountain and a Molehill? Quantifying Semantic Labeling of Visual Features in Line Charts
Venue: arXiv cs.HC 2023
Year: 2023

Relevant language describing visual features in charts can be useful for authoring captions and summaries about the charts to help with readers' takeaways. To better understand the interplay between concepts that describe visual features and the semantic relationships among those concepts (e.g., 'sharp increase' vs. 'gradual rise'), we conducted a crowdsourced study to collect labels and visual feature pairs for univariate line charts. Using this crowdsourced dataset of labeled visual signatures, this paper proposes a novel method for labeling visual chart features based on combining feature-word distributions with the visual features and the data domain of the charts. These feature-word-topic models identify word associations with similar yet subtle differences in semantics, such as 'flat,' 'plateau,' and 'stagnant,' and descriptors of the visual features, such as 'sharp increase,' 'slow climb,' and 'peak.' Our feature-word-topic model is computed using both a quantified semantics approach and a signal processing-inspired least-errors shape-similarity approach. We finally demonstrate the application of this dataset for annotating charts and generating textual data summaries.
Computational Long Exposure Mobile Photography
Venue: arXiv cs.CV 2023
Year: 2023

Long exposure photography produces stunning imagery, representing moving elements in a scene with motion-blur. It is generally employed in two modalities, producing either a foreground or a background blur effect. Foreground blur images are traditionally captured on a tripod-mounted camera and portray blurred moving foreground elements, such as silky water or light trails, over a perfectly sharp background landscape. Background blur images, also called panning photography, are captured while the camera is tracking a moving subject, to produce an image of a sharp subject over a background blurred by relative motion. Both techniques are notoriously challenging and require additional equipment and advanced skills. In this paper, we describe a computational burst photography system that operates in a hand-held smartphone camera app, and achieves these effects fully automatically, at the tap of the shutter button. Our approach first detects and segments the salient subject. We track the scene motion over multiple frames and align the images in order to preserve desired sharpness and to produce aesthetically pleasing motion streaks. We capture an under-exposed burst and select the subset of input frames that will produce blur trails of controlled length, regardless of scene or camera motion velocity. We predict inter-frame motion and synthesize motion-blur to fill the temporal gaps between the input frames. Finally, we composite the blurred image with the sharp regular exposure to protect the sharpness of faces or areas of the scene that are barely moving, and produce a final high resolution and high dynamic range (HDR) photograph. Our system democratizes a capability previously reserved to professionals, and makes this creative style accessible to most casual photographers. More information and supplementary material can be found on our project webpage: https://motion-mode.github.io/
Achieving state-of-the-art performance in the Medical Out-of-Distribution (MOOD) challenge using plausible synthetic anomalies
Venue: arXiv cs.CV 2023
Year: 2023

The detection and localization of anomalies is one important medical image analysis task. Most commonly, Computer Vision anomaly detection approaches rely on manual annotations that are both time consuming and expensive to obtain. Unsupervised anomaly detection, or Out-of-Distribution detection, aims at identifying anomalous samples relying only on unannotated samples considered normal. In this study we present a new unsupervised anomaly detection method. Our method builds upon the self-supervised strategy consisting on training a segmentation network to identify local synthetic anomalies. Our contributions improve the synthetic anomaly generation process, making synthetic anomalies more heterogeneous and challenging by 1) using complex random shapes and 2) smoothing the edges of synthetic anomalies so networks cannot rely on the high gradient between image and synthetic anomalies. In our implementation we adopted standard practices in 3D medical image segmentation, including 3D U-Net architecture, patch-wise training and model ensembling. Our method was evaluated using a validation set with different types of synthetic anomalies. Our experiments show that our method improved substantially the baseline method performance. Additionally, we evaluated our method by participating in the Medical Out-of-Distribution (MOOD) Challenge held at MICCAI in 2022 and achieved first position in both sample-wise and pixel-wise tasks. Our experiments and results in the latest MOOD challenge show that our simple yet effective approach can substantially improve the performance of Out-of-Distribution detection techniques which rely on synthetic anomalies.
An Effective Data Creation Pipeline to Generate High-quality Financial Instruction Data for Large Language Model
Venue: arXiv cs.CL 2023
Year: 2023

At the beginning era of large language model, it is quite critical to generate a high-quality financial dataset to fine-tune a large language model for financial related tasks. Thus, this paper presents a carefully designed data creation pipeline for this purpose. Particularly, we initiate a dialogue between an AI investor and financial expert using ChatGPT and incorporate the feedback of human financial experts, leading to the refinement of the dataset. This pipeline yielded a robust instruction tuning dataset comprised of 103k multi-turn chats. Extensive experiments have been conducted on this dataset to evaluate the model's performance by adopting an external GPT-4 as the judge. The promising experimental results verify that our approach led to significant advancements in generating accurate, relevant, and financial-style responses from AI models, and thus providing a powerful tool for applications within the financial sector.
FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysis
Venue: arXiv cs.CL 2023
Year: 2023

In this paper, we propose FinVis-GPT, a novel multimodal large language model (LLM) specifically designed for financial chart analysis. By leveraging the power of LLMs and incorporating instruction tuning and multimodal capabilities, FinVis-GPT is capable of interpreting financial charts and providing valuable analysis. To train FinVis-GPT, a financial task oriented dataset was generated for pre-training alignment and instruction tuning, comprising various types of financial charts and their corresponding descriptions. We evaluate the model performance via several case studies due to the time limit, and the promising results demonstrated that FinVis-GPT is superior in various financial chart related tasks, including generating descriptions, answering questions and predicting future market trends, surpassing existing state-of-the-art multimodal LLMs. The proposed FinVis-GPT serves as a pioneering effort in utilizing multimodal LLMs in the finance domain and our generated dataset will be release for public use in the near future to speedup related research.
Novel Physics-Based Machine-Learning Models for Indoor Air Quality Approximations
Venue: arXiv cs.LG 2023
Year: 2023

Cost-effective sensors are capable of real-time capturing a variety of air quality-related modalities from different pollutant concentrations to indoor/outdoor humidity and temperature. Machine learning (ML) models are capable of performing air-quality "ahead-of-time" approximations. Undoubtedly, accurate indoor air quality approximation significantly helps provide a healthy indoor environment, optimize associated energy consumption, and offer human comfort. However, it is crucial to design an ML architecture to capture the domain knowledge, so-called problem physics. In this study, we propose six novel physics-based ML models for accurate indoor pollutant concentration approximations. The proposed models include an adroit combination of state-space concepts in physics, Gated Recurrent Units, and Decomposition techniques. The proposed models were illustrated using data collected from five offices in a commercial building in California. The proposed models are shown to be less complex, computationally more efficient, and more accurate than similar state-of-the-art transformer-based models. The superiority of the proposed models is due to their relatively light architecture (computational efficiency) and, more importantly, their ability to capture the underlying highly nonlinear patterns embedded in the often contaminated sensor-collected indoor air quality temporal data.
Decentralized Translator of Trust: Supporting Heterogeneous TEE for Critical Infrastructure Protection
Venue: arXiv cs.CR 2023
Year: 2023

Trusted execution environment (TEE) technology has found many applications in mitigating various security risks in an efficient manner, which is attractive for critical infrastructure protection. First, the natural of critical infrastructure requires it to be well protected from various cyber attacks. Second, performance is usually important for critical infrastructure and it cannot afford an expensive protection mechanism. While a large number of TEE-based critical infrastructure protection systems have been proposed to address various security challenges (e.g., secure sensing and reliable control), most existing works ignore one important feature, i.e., devices comprised the critical infrastructure may be equipped with multiple incompatible TEE technologies and belongs to different owners. This feature makes it hard for these devices to establish mutual trust and form a unified TEE environment. To address these challenges and fully unleash the potential of TEE technology for critical infrastructure protection, we propose DHTee, a decentralized coordination mechanism. DHTee uses blockchain technology to support key TEE functions in a heterogeneous TEE environment, especially the attestation service. A Device equipped with one TEE can interact securely with the blockchain to verify whether another potential collaborating device claiming to have a different TEE meets the security requirements. DHTee is also flexible and can support new TEE schemes without affecting devices using existing TEEs that have been supported by the system.
Investigating Reinforcement Learning for Communication Strategies in a Task-Initiative Setting
Venue: arXiv cs.CL 2023
Year: 2023

Many conversational domains require the system to present nuanced information to users. Such systems must follow up what they say to address clarification questions and repair misunderstandings. In this work, we explore this interactive strategy in a referential communication task. Using simulation, we analyze the communication trade-offs between initial presentation and subsequent followup as a function of user clarification strategy, and compare the performance of several baseline strategies to policies derived by reinforcement learning. We find surprising advantages to coherence-based representations of dialogue strategy, which bring minimal data requirements, explainable choices, and strong audit capabilities, but incur little loss in predicted outcomes across a wide range of user models.
Erase and Repair: An Efficient Box-Free Removal Attack on High-Capacity Deep Hiding
Venue: arXiv cs.CR 2023
Year: 2023

Deep hiding, embedding images with others using deep neural networks, has demonstrated impressive efficacy in increasing the message capacity and robustness of secret sharing. In this paper, we challenge the robustness of existing deep hiding schemes by preventing the recovery of secret images, building on our in-depth study of state-of-the-art deep hiding schemes and their vulnerabilities. Leveraging our analysis, we first propose a simple box-free removal attack on deep hiding that does not require any prior knowledge of the deep hiding schemes. To improve the removal performance on the deep hiding schemes that may be enhanced by adversarial training, we further design a more powerful removal attack, efficient box-free removal attack (EBRA), which employs image inpainting techniques to remove secret images from container images. In addition, to ensure the effectiveness of our attack and preserve the fidelity of the processed container images, we design an erasing phase based on the locality of deep hiding to remove secret information and then make full use of the visual information of container images to repair the erased visual content. Extensive evaluations show our method can completely remove secret images from container images with negligible impact on the quality of container images.
PPI-NET: End-to-End Parametric Primitive Inference
Venue: arXiv cs.CV 2023
Year: 2023

In engineering applications, line, circle, arc, and point are collectively referred to as primitives, and they play a crucial role in path planning, simulation analysis, and manufacturing. When designing CAD models, engineers typically start by sketching the model's orthographic view on paper or a whiteboard and then translate the design intent into a CAD program. Although this design method is powerful, it often involves challenging and repetitive tasks, requiring engineers to perform numerous similar operations in each design. To address this conversion process, we propose an efficient and accurate end-to-end method that avoids the inefficiency and error accumulation issues associated with using auto-regressive models to infer parametric primitives from hand-drawn sketch images. Since our model samples match the representation format of standard CAD software, they can be imported into CAD software for solving, editing, and applied to downstream design tasks.
VisAlign: Dataset for Measuring the Degree of Alignment between AI and Humans in Visual Perception
Venue: arXiv cs.CV 2023
Year: 2023

AI alignment refers to models acting towards human-intended goals, preferences, or ethical principles. Given that most large-scale deep learning models act as black boxes and cannot be manually controlled, analyzing the similarity between models and humans can be a proxy measure for ensuring AI safety. In this paper, we focus on the models' visual perception alignment with humans, further referred to as AI-human visual alignment. Specifically, we propose a new dataset for measuring AI-human visual alignment in terms of image classification, a fundamental task in machine perception. In order to evaluate AI-human visual alignment, a dataset should encompass samples with various scenarios that may arise in the real world and have gold human perception labels. Our dataset consists of three groups of samples, namely Must-Act (i.e., Must-Classify), Must-Abstain, and Uncertain, based on the quantity and clarity of visual information in an image and further divided into eight categories. All samples have a gold human perception label; even Uncertain (severely blurry) sample labels were obtained via crowd-sourcing. The validity of our dataset is verified by sampling theory, statistical theories related to survey design, and experts in the related fields. Using our dataset, we analyze the visual alignment and reliability of five popular visual perception models and seven abstention methods. Our code and data is available at https://github.com/jiyounglee-0523/VisAlign.
Multi-robot Path Planning with Rapidly-exploring Random Disjointed-Trees
Venue: arXiv cs.RO 2023
Year: 2023

Multi-robot path planning is a computational process involving finding paths for each robot from its start to the goal while ensuring collision-free operation. It is widely used in robots and autonomous driving. However, the computational time of multi-robot path planning algorithms is enormous, resulting in low efficiency in practical applications. To address this problem, this article proposes a novel multi-robot path planning algorithm (Multi-Agent Rapidly-exploring Random Disjointed-Trees*, MA-RRdT*) based on multi-tree random sampling. The proposed algorithm is based on a single-robot path planning algorithm (Rapidly-exploring Random disjointed-Trees*, RRdT*). The novel MA-RRdT* algorithm has the advantages of fast speed, high space exploration efficiency, and suitability for complex maps. Comparative experiments are completed to evaluate the effectiveness of MA-RRdT*. The final experimental results validate the superior performance of the MA-RRdT* algorithm in terms of time cost and space exploration efficiency.
VCTP: A Verifiable Credential-based Trust Propagation Protocol for Personal Issuers in Self-Sovereign Identity Platforms
Venue: arXiv cs.CR 2023
Year: 2023

Self Sovereign Identity (SSI) is an emerging identity system that facilitates secure credential issuance and verification without placing trust in any centralised authority. To bypass central trust, most SSI implementations place blockchain as a trusted mediator by placing credential transactions on-chain. Yet, existing SSI platforms face trust issues as all credential issuers in SSI are not supported with adequate trust. Current SSI solutions provide trust support to the officiated issuers (e.g., government agencies), who must follow a precise process to assess their credentials. However, there is no structured trust support for individuals of SSI who may attempt to issue a credential (e.g., letter of consent) in the context of business processes. Therefore, some risk-averse verifiers in the system may not accept the credentials from individual issuers to avoid carrying the cost of mishaps from potentially inadmissible credentials without reliance on a trusted agency. This paper proposes a trust propagation protocol that supports individual users to be trusted as verifiable issuers in the SSI platform by establishing a trust propagation credential template in the blockchain. Our approach utilises (i) the sanitizable signature scheme to propagate the required trust to an individual issuer, (ii) a voting mechanism to minimises the possibility of collusion. Our implementation demonstrates that the solution is both practical and performs well under varying system loads.