OpenRaiser/Intern-Atlas huggingface dataset:
Mitigating Metric Bias in Minimum Bayes Risk Decoding
Venue: arXiv cs.CL 2024
Year: 2024

While Minimum Bayes Risk (MBR) decoding using metrics such as COMET or MetricX has outperformed traditional decoding methods such as greedy or beam search, it introduces a challenge we refer to as metric bias. As MBR decoding aims to produce translations that score highly according to a specific utility metric, this very process makes it impossible to use the same metric for both decoding and evaluation, as improvements might simply be due to reward hacking rather than reflecting real quality improvements. In this work we find that compared to human ratings, neural metrics not only overestimate the quality of MBR decoding when the same metric is used as the utility metric, but they also overestimate the quality of MBR/QE decoding with other neural utility metrics as well. We also show that the metric bias issue can be mitigated by using an ensemble of utility metrics during MBR decoding: human evaluations show that MBR decoding using an ensemble of utility metrics outperforms a single utility metric.
VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigation
Venue: arXiv cs.RO 2024
Year: 2024

With the recent rise of Large Language Models (LLMs), Vision-Language Models (VLMs), and other general foundation models, there is growing potential for multimodal, multi-task embodied agents that can operate in diverse environments given only natural language as input. One such application area is indoor navigation using natural language instructions. However, despite recent progress, this problem remains challenging due to the spatial reasoning and semantic understanding required, particularly in arbitrary scenes that may contain many objects belonging to fine-grained classes. To address this challenge, we curate the largest real-world dataset for Vision and Language-guided Action in 3D Scenes (VLA-3D), consisting of over 11.5K scanned 3D indoor rooms from existing datasets, 23.5M heuristically generated semantic relations between objects, and 9.7M synthetically generated referential statements. Our dataset consists of processed 3D point clouds, semantic object and room annotations, scene graphs, navigable free space annotations, and referential language statements that specifically focus on view-independent spatial relations for disambiguating objects. The goal of these features is to aid the downstream task of navigation, especially on real-world systems where some level of robustness must be guaranteed in an open world of changing scenes and imperfect language. We benchmark our dataset with current state-of-the-art models to obtain a performance baseline. All code to generate and visualize the dataset is publicly released, see https://github.com/HaochenZ11/VLA-3D. With the release of this dataset, we hope to provide a resource for progress in semantic 3D scene understanding that is robust to changes and one which will aid the development of interactive indoor navigation systems.
Semantic Navigation for AI-assisted Ideation
Venue: arXiv cs.HC 2024
Year: 2024

We present a novel AI-based ideation assistant and evaluate it in a user study with a group of innovators. The key contribution of our work is twofold: we propose a method of idea exploration in a constrained domain by means of LLM-supported semantic navigation of problem and solution spaces, and employ novel automated data input filtering to improve generations. We found that semantic exploration is preferred to the traditional prompt-output interactions, measured both in explicit survey rankings, and in terms of innovation assistant engagement, where 2.1x more generations were performed using semantic exploration. We also show that filtering input data with metrics such as relevancy, coherence and human alignment leads to improved generations in the same metrics as well as enhanced quality of experience among innovators.
AMNCutter: Affinity-Attention-Guided Multi-View Normalized Cutter for Unsupervised Surgical Instrument Segmentation
Venue: arXiv cs.CV 2024
Year: 2024

Surgical instrument segmentation (SIS) is pivotal for robotic-assisted minimally invasive surgery, assisting surgeons by identifying surgical instruments in endoscopic video frames. Recent unsupervised surgical instrument segmentation (USIS) methods primarily rely on pseudo-labels derived from low-level features such as color and optical flow, but these methods show limited effectiveness and generalizability in complex and unseen endoscopic scenarios. In this work, we propose a label-free unsupervised model featuring a novel module named Multi-View Normalized Cutter (m-NCutter). Different from previous USIS works, our model is trained using a graph-cutting loss function that leverages patch affinities for supervision, eliminating the need for pseudo-labels. The framework adaptively determines which affinities from which levels should be prioritized. Therefore, the low- and high-level features and their affinities are effectively integrated to train a label-free unsupervised model, showing superior effectiveness and generalization ability. We conduct comprehensive experiments across multiple SIS datasets to validate our approach's state-of-the-art (SOTA) performance, robustness, and exceptional potential as a pre-trained model. Our code is released at https://github.com/MingyuShengSMY/AMNCutter.
The Essence of the Essence from the Web:The Metasearch Engine
Venue: arXiv cs.IR 2024
Year: 2024

The exponential growth of information source on the web and in turn continuing technological progress of searching the information by using tools like Search Engines gives rise to many problems for the user to know which tool is best for their query and which tool is not. At this time Metasearch Engine comes into play by reducing the user burden by dispatching queries to multiple search engines in parallel and refining the results of these search engines to give the best out of best by doing superior job on their side. These engines do not own a database of Web pages rather they send search terms to the databases maintained by the search engine companies, get back results from all the search engines queried and then compile the results to be presented to the user. In this paper, we describe the working of a typical metasearch engine and then present a comparative study of traditional search engines and metasearch engines on the basis of different parameters and show how metasearch engines are better than the other search engines.
These Maps Are Made by Propagation: Adapting Deep Stereo Networks to Road Scenarios with Decisive Disparity Diffusion
Venue: arXiv cs.CV 2024
Year: 2024

Stereo matching has emerged as a cost-effective solution for road surface 3D reconstruction, garnering significant attention towards improving both computational efficiency and accuracy. This article introduces decisive disparity diffusion (D3Stereo), marking the first exploration of dense deep feature matching that adapts pre-trained deep convolutional neural networks (DCNNs) to previously unseen road scenarios. A pyramid of cost volumes is initially created using various levels of learned representations. Subsequently, a novel recursive bilateral filtering algorithm is employed to aggregate these costs. A key innovation of D3Stereo lies in its alternating decisive disparity diffusion strategy, wherein intra-scale diffusion is employed to complete sparse disparity images, while inter-scale inheritance provides valuable prior information for higher resolutions. Extensive experiments conducted on our created UDTIRI-Stereo and Stereo-Road datasets underscore the effectiveness of D3Stereo strategy in adapting pre-trained DCNNs and its superior performance compared to all other explicit programming-based algorithms designed specifically for road surface 3D reconstruction. Additional experiments conducted on the Middlebury dataset with backbone DCNNs pre-trained on the ImageNet database further validate the versatility of D3Stereo strategy in tackling general stereo matching problems.
Optimal prefix-suffix queries with applications
Venue: arXiv cs.DS 2024
Year: 2024

We revisit the classic border tree data structure [Gu, Farach, Beigel, SODA 1994] that answers the following prefix-suffix queries on a string $T$ of length $n$ over an integer alphabet $Σ=[0,σ)$: for any $i,j \in [0,n)$ return all occurrences of $T$ in $T[0\mathinner{.\,.} i]T[j\mathinner{.\,.} n-1]$. The border tree of $T$ can be constructed in $\mathcal{O}(n)$ time and answers prefix-suffix queries in $\mathcal{O}(\log n + \textsf{Occ})$ time, where $\textsf{Occ}$ is the number of occurrences of $T$ in $T[0\mathinner{.\,.} i]T[j\mathinner{.\,.} n-1]$. Our contribution here is the following. We present a completely different and remarkably simple data structure that can be constructed in the optimal $\mathcal{O}(n/\log_σn)$ time and supports queries in the optimal $\mathcal{O}(1)$ time. Our result is based on a new structural lemma that lets us encode the output of any query in constant time and space. We also show a new direct application of our result in pattern matching on node-labeled graphs.
The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation
Venue: arXiv cs.LG 2024
Year: 2024

Speculative decoding aims to speed up autoregressive generation of a language model by verifying in parallel the tokens generated by a smaller draft model.In this work, we explore the effectiveness of learning-free, negligible-cost draft strategies, namely $N$-grams obtained from the model weights and the context. While the predicted next token of the base model is rarely the top prediction of these simple strategies, we observe that it is often within their top-$k$ predictions for small $k$. Based on this, we show that combinations of simple strategies can achieve significant inference speedups over different tasks. The overall performance is comparable to more complex methods, yet does not require expensive preprocessing or modification of the base model, and allows for seamless `plug-and-play' integration into pipelines.
Harmformer: Harmonic Networks Meet Transformers for Continuous Roto-Translation Equivariance
Venue: arXiv cs.CV 2024
Year: 2024

CNNs exhibit inherent equivariance to image translation, leading to efficient parameter and data usage, faster learning, and improved robustness. The concept of translation equivariant networks has been successfully extended to rotation transformation using group convolution for discrete rotation groups and harmonic functions for the continuous rotation group encompassing $360^\circ$. We explore the compatibility of the SA mechanism with full rotation equivariance, in contrast to previous studies that focused on discrete rotation. We introduce the Harmformer, a harmonic transformer with a convolutional stem that achieves equivariance for both translation and continuous rotation. Accompanied by an end-to-end equivariance proof, the Harmformer not only outperforms previous equivariant transformers, but also demonstrates inherent stability under any continuous rotation, even without seeing rotated samples during training.
On the Decomposition of Differential Game
Venue: arXiv cs.GT 2024
Year: 2024

To understand the complexity of the dynamic of learning in differential games, we decompose the game into components where the dynamic is well understood. One of the possible tools is Helmholtz's theorem, which can decompose a vector field into a potential and a harmonic component. This has been shown to be effective in finite and normal-form games. However, applying Helmholtz's theorem by connecting it with the Hodge theorem on $\mathbb{R}^n$ (which is the strategy space of differential game) is non-trivial due to the non-compactness of $\mathbb{R}^n$. Bridging the dynamic-strategic disconnect through Hodge/Helmoltz's theorem in differential games is then left as an open problem \cite{letcher2019differentiable}. In this work, we provide two decompositions of differential games to answer this question: the first as an exact scalar potential part, a near vector potential part, and a non-strategic part; the second as a near scalar potential part, an exact vector potential part, and a non-strategic part. We show that scalar potential games coincide with potential games proposed by \cite{monderer1996potential}, where the gradient descent dynamic can successfully find the Nash equilibrium. For the vector potential game, we show that the individual gradient field is divergence-free, in which case the gradient descent dynamic may either be divergent or recurrent.
An Enhancement of Haar Cascade Algorithm Applied to Face Recognition for Gate Pass Security
Venue: arXiv cs.CV 2024
Year: 2024

This study is focused on enhancing the Haar Cascade Algorithm to decrease the false positive and false negative rate in face matching and face detection to increase the accuracy rate even under challenging conditions. The face recognition library was implemented with Haar Cascade Algorithm in which the 128-dimensional vectors representing the unique features of a face are encoded. A subprocess was applied where the grayscale image from Haar Cascade was converted to RGB to improve the face encoding. Logical process and face filtering are also used to decrease non-face detection. The Enhanced Haar Cascade Algorithm produced a 98.39% accuracy rate (21.39% increase), 63.59% precision rate, 98.30% recall rate, and 72.23% in F1 Score. In comparison, the Haar Cascade Algorithm achieved a 46.70% to 77.00% accuracy rate, 44.15% precision rate, 98.61% recall rate, and 47.01% in F1 Score. Both algorithms used the Confusion Matrix Test with 301,950 comparisons using the same dataset of 550 images. The 98.39% accuracy rate shows a significant decrease in false positive and false negative rates in facial recognition. Face matching and face detection are more accurate in images with complex backgrounds, lighting variations, and occlusions, or even those with similar attributes.
Accelerating DNA Read Mapping with Digital Processing-in-Memory
Venue: arXiv cs.AR 2024
Year: 2024

Genome analysis has revolutionized fields such as personalized medicine and forensics. Modern sequencing machines generate vast amounts of fragmented strings of genome data called reads. The alignment of these reads into a complete DNA sequence of an organism (the read mapping process) requires extensive data transfer between processing units and memory, leading to execution bottlenecks. Prior studies have primarily focused on accelerating specific stages of the read-mapping task. Conversely, this paper introduces a holistic framework called DART-PIM that accelerates the entire read-mapping process. DART-PIM facilitates digital processing-in-memory (PIM) for an end-to-end acceleration of the entire read-mapping process, from indexing using a unique data organization schema to filtering and read alignment with an optimized Wagner Fischer algorithm. A comprehensive performance evaluation with real genomic data shows that DART-PIM achieves a 5.7x and 257x improvement in throughput and a 92x and 27x energy efficiency enhancement compared to state-of-the-art GPU and PIM implementations, respectively.
Attribute-Based Encryption With Payable Outsourced Decryption Using Blockchain and Responsive Zero Knowledge Proof
Venue: arXiv cs.CR 2024
Year: 2024

Attribute-Based Encryption (ABE) is a promising solution for access control in cloud services. However, the heavy decryption overhead hinders its widespread adoption. A general approach to address this issue is to outsource decryption to decryption cloud service(DCS). Existing schemes have utilized various methods to enable users to verify outsourced results; however, they lack an effective mechanism to achieve exemptibility which enables the honest DCS to escape from wrong claims. And it is impractical to assume that the DCS will provide free services. In this paper, we propose a blockchain-based payable outsourced decryption ABE scheme that achieves both verifiability and exemptibility without adding redundant information to ABE ciphertext. We use zero-knowledge proof to verify outsourced results on blockchain and introduce an optional single-round challenge game under optimistic assumption to address the high cost of proof generation. Moreover, our system achieves fairness and decentralized outsourcing to protect the interests of all parties. Finally, we implement and evaluate our scheme on Ethereum to demonstrate its feasibility and efficiency, the gas usage in attribute numbers from 5 to 60 is 11$\times$ to 140$\times$ in the happy case and 4$\times$ to 55$\times$ in the challenge case lower than the scheme of Ge et al. (TDSC'23).
WiP: Towards a Secure SECP256K1 for Crypto Wallets: Hardware Architecture and Implementation
Venue: arXiv cs.CR 2024
Year: 2024

The SECP256K1 elliptic curve algorithm is fundamental in cryptocurrency wallets for generating secure public keys from private keys, thereby ensuring the protection and ownership of blockchain-based digital assets. However, the literature highlights several successful side-channel attacks on hardware wallets that exploit SECP256K1 to extract private keys. This work proposes a novel hardware architecture for SECP256K1, optimized for side-channel attack resistance and efficient resource utilization. The architecture incorporates complete addition formulas, temporary registers, and parallel processing techniques, making elliptic curve point addition and doubling operations indistinguishable. Implementation results demonstrate an average reduction of 45% in LUT usage compared to similar works, emphasizing the design's resource efficiency.
Game-Theoretic Machine Unlearning: Mitigating Extra Privacy Leakage
Venue: arXiv cs.CY 2024
Year: 2024

With the extensive use of machine learning technologies, data providers encounter increasing privacy risks. Recent legislation, such as GDPR, obligates organizations to remove requested data and its influence from a trained model. Machine unlearning is an emerging technique designed to enable machine learning models to erase users' private information. Although several efficient machine unlearning schemes have been proposed, these methods still have limitations. First, removing the contributions of partial data may lead to model performance degradation. Second, discrepancies between the original and generated unlearned models can be exploited by attackers to obtain target sample's information, resulting in additional privacy leakage risks. To address above challenges, we proposed a game-theoretic machine unlearning algorithm that simulates the competitive relationship between unlearning performance and privacy protection. This algorithm comprises unlearning and privacy modules. The unlearning module possesses a loss function composed of model distance and classification error, which is used to derive the optimal strategy. The privacy module aims to make it difficult for an attacker to infer membership information from the unlearned data, thereby reducing the privacy leakage risk during the unlearning process. Additionally, the experimental results on real-world datasets demonstrate that this game-theoretic unlearning algorithm's effectiveness and its ability to generate an unlearned model with a performance similar to that of the retrained one while mitigating extra privacy leakage risks.
Inter-Frame Coding for Dynamic Meshes via Coarse-to-Fine Anchor Mesh Generation
Venue: arXiv cs.MM 2024
Year: 2024

In the current Video-based Dynamic Mesh Coding (V-DMC) standard, inter-frame coding is restricted to mesh frames with constant topology. Consequently, temporal redundancy is not fully leveraged, resulting in suboptimal compression efficacy. To address this limitation, this paper introduces a novel coarse-to-fine scheme to generate anchor meshes for frames with time-varying topology. Initially, we generate a coarse anchor mesh using an octree-based nearest neighbor search. Motion estimation compensates for regions with significant motion changes during this process. However, the quality of the coarse mesh is low due to its suboptimal vertices. To enhance details, the fine anchor mesh is further optimized using the Quadric Error Metrics (QEM) algorithm to calculate more precise anchor points. The inter-frame anchor mesh generated herein retains the connectivity of the reference base mesh, while concurrently preserving superior quality. Experimental results show that our method achieves 7.2% ~ 10.3% BD-rate gain compared to the existing V-DMC test model version 7.
Design and control of a robotic payload stabilization mechanism for rocket flights
Venue: arXiv cs.RO 2024
Year: 2024

The use of parallel manipulators in aerospace engineering has gained significant attention due to their ability to provide improved stability and precision. This paper presents the design, control, and analysis of 'STEWIE', which is a three-degree-of-freedom (DoF) parallel manipulator robot developed by members of the thrustMIT rocketry team, as a payload stabilization mechanism for their sounding rocket, 'Altair'. The goal of the robot was to demonstrate the attitude control of the parallel plate against the continuous change in orientation experienced by the rocket during its flight, stabilizing the payloads. At the same time, the high gravitational forces (G-forces) and vibrations experienced by the sounding rocket are counteracted. A novel design of the mechanism, inspired by a standard Stewart platform, is proposed which was down-scaled to fit inside a 4U CubeSat within its space constraints. The robot uses three micro servo motors to actuate the links that control the alignment of the parallel plate. In addition to the actuation mechanism, a robust control system for its manipulation was developed for the robot. The robot represents a significant advancement in the field of space robotics in the aerospace industry by demonstrating the successful implementation of complex robotic mechanisms in small, confined spaces such as CubeSats, which are standard form factors for large payloads in the aerospace industry.
Memorized action chunking with Transformers: Imitation learning for vision-based tissue surface scanning
Venue: arXiv cs.RO 2024
Year: 2024

Optical sensing technologies are emerging technologies used in cancer surgeries to ensure the complete removal of cancerous tissue. While point-wise assessment has many potential applications, incorporating automated large area scanning would enable holistic tissue sampling. However, such scanning tasks are challenging due to their long-horizon dependency and the requirement for fine-grained motion. To address these issues, we introduce Memorized Action Chunking with Transformers (MACT), an intuitive yet efficient imitation learning method for tissue surface scanning tasks. It utilizes a sequence of past images as historical information to predict near-future action sequences. In addition, hybrid temporal-spatial positional embeddings were employed to facilitate learning. In various simulation settings, MACT demonstrated significant improvements in contour scanning and area scanning over the baseline model. In real-world testing, with only 50 demonstration trajectories, MACT surpassed the baseline model by achieving a 60-80% success rate on all scanning tasks. Our findings suggest that MACT is a promising model for adaptive scanning in surgical settings.