OpenRaiser/Intern-Atlas huggingface dataset:
Auto-calibrating Universal Programmable Photonic Circuits: Hardware Error-Correction and Defect Resilience
Venue: arXiv cs.ET 2023
Year: 2023

It is recently shown that discrete $N\times N$ linear unitary operators can be represented by interlacing $N+1$ phase shift layers with a fixed intervening operator such as Discrete Fractional Fourier Transform (DFrFT). Here, we show that introducing perturbations to the intervening operations does not compromise the universality of this architecture. Furthermore, we show that this architecture is resilient to defects in the phase shifters as long as no more than one faulty phase shifter is present in each layer. These properties enable post-fabrication auto-calibration of such universal photonic circuits, effectively compensating for fabrication errors and defects in phase components.
Algorithms and Computational Study on a Transportation System Integrating Public Transit and Ridesharing of Personal Vehicles
Venue: arXiv cs.DS 2023
Year: 2023

The potential of integrating public transit with ridesharing includes shorter travel time for commuters and higher occupancy rate of personal vehicles and public transit ridership. In this paper, we describe a centralized transit system that integrates public transit and ridesharing to reduce travel time for commuters. In the system, a set of ridesharing providers (drivers) and a set of public transit riders are received. The optimization goal of the system is to assign riders to drivers by arranging public transit and ridesharing combined routes subject to shorter commuting time for as many riders as possible. We give an exact algorithm, which is an ILP formulation based on a hypergraph representation of the problem. By using the ILP and the hypergraph, we give approximation algorithms based on LP-rounding and hypergraph matching/weighted set packing, respectively. As a case study, we conduct an extensive computational study based on real-world public transit dataset and ridesharing dataset in Chicago city. To evaluate the effectiveness of the transit system and our algorithms, we generate data instances from the datasets. The experimental results show that more than 60% of riders are assigned to drivers on average, riders' commuting time is reduced by 23% and vehicle occupancy rate is improved to almost 3. Our proposed algorithms are efficient for practical scenarios.
A Comparative Study of Text Embedding Models for Semantic Text Similarity in Bug Reports
Venue: arXiv cs.SE 2023
Year: 2023

Bug reports are an essential aspect of software development, and it is crucial to identify and resolve them quickly to ensure the consistent functioning of software systems. Retrieving similar bug reports from an existing database can help reduce the time and effort required to resolve bugs. In this paper, we compared the effectiveness of semantic textual similarity methods for retrieving similar bug reports based on a similarity score. We explored several embedding models such as TF-IDF (Baseline), FastText, Gensim, BERT, and ADA. We used the Software Defects Data containing bug reports for various software projects to evaluate the performance of these models. Our experimental results showed that BERT generally outperformed the rest of the models regarding recall, followed by ADA, Gensim, FastText, and TFIDF. Our study provides insights into the effectiveness of different embedding methods for retrieving similar bug reports and highlights the impact of selecting the appropriate one for this task. Our code is available on GitHub.
GPU Accelerated Color Correction and Frame Warping for Real-time Video Stitching
Venue: arXiv cs.CV 2023
Year: 2023

Traditional image stitching focuses on a single panorama frame without considering the spatial-temporal consistency in videos. The straightforward image stitching approach will cause temporal flicking and color inconstancy when it is applied to the video stitching task. Besides, inaccurate camera parameters will cause artifacts in the image warping. In this paper, we propose a real-time system to stitch multiple video sequences into a panoramic video, which is based on GPU accelerated color correction and frame warping without accurate camera parameters. We extend the traditional 2D-Matrix (2D-M) color correction approach and a present spatio-temporal 3D-Matrix (3D-M) color correction method for the overlap local regions with online color balancing using a piecewise function on global frames. Furthermore, we use pairwise homography matrices given by coarse camera calibration for global warping followed by accurate local warping based on the optical flow. Experimental results show that our system can generate highquality panorama videos in real time.
SHAPFUZZ: Efficient Fuzzing via Shapley-Guided Byte Selection
Venue: arXiv cs.CR 2023
Year: 2023

Mutation-based fuzzing is popular and effective in discovering unseen code and exposing bugs. However, only a few studies have concentrated on quantifying the importance of input bytes, which refers to the degree to which a byte contributes to the discovery of new code. They often focus on obtaining the relationship between input bytes and path constraints, ignoring the fact that not all constraint-related bytes can discover new code. In this paper, we conduct Shapely analysis to understand the effect of byte positions on fuzzing performance, and find that some byte positions contribute more than others and this property often holds across seeds. Based on this observation, we propose a novel fuzzing solution, ShapFuzz, to guide byte selection and mutation. Specifically, ShapFuzz updates Shapley values (importance) of bytes when each input is tested during fuzzing with a low overhead, and utilizes contextual multi-armed bandit to trade off between mutating high Shapley value bytes and low-frequently chosen bytes. We implement a prototype of this solution based on AFL++, i.e., ShapFuzz. We evaluate ShapFuzz against ten state-of-the-art fuzzers, including five byte schedule-reinforced fuzzers and five commonly used fuzzers. Compared with byte schedule-reinforced fuzzers, ShapFuzz discovers more edges and exposes more bugs than the best baseline on three different sets of initial seeds. Compared with commonly used fuzzers, ShapFuzz exposes 20 more bugs than the best comparison fuzzer, and discovers 6 more CVEs than the best baseline on MAGMA. Furthermore, ShapFuzz discovers 11 new bugs on the latest versions of programs, and 3 of them are confirmed by vendors.
Progression-Guided Temporal Action Detection in Videos
Venue: arXiv cs.CV 2023
Year: 2023

We present a novel framework, Action Progression Network (APN), for temporal action detection (TAD) in videos. The framework locates actions in videos by detecting the action evolution process. To encode the action evolution, we quantify a complete action process into 101 ordered stages (0\%, 1\%, ..., 100\%), referred to as action progressions. We then train a neural network to recognize the action progressions. The framework detects action boundaries by detecting complete action processes in the videos, e.g., a video segment with detected action progressions closely follow the sequence 0\%, 1\%, ..., 100\%. The framework offers three major advantages: (1) Our neural networks are trained end-to-end, contrasting conventional methods that optimize modules separately; (2) The APN is trained using action frames exclusively, enabling models to be trained on action classification datasets and robust to videos with temporal background styles differing from those in training; (3) Our framework effectively avoids detecting incomplete actions and excels in detecting long-lasting actions due to the fine-grained and explicit encoding of the temporal structure of actions. Leveraging these advantages, the APN achieves competitive performance and significantly surpasses its counterparts in detecting long-lasting actions. With an IoU threshold of 0.5, the APN achieves a mean Average Precision (mAP) of 58.3\% on the THUMOS14 dataset and 98.9\% mAP on the DFMAD70 dataset.
Document Automation Architectures: Updated Survey in Light of Large Language Models
Venue: arXiv cs.CL 2023
Year: 2023

This paper surveys the current state of the art in document automation (DA). The objective of DA is to reduce the manual effort during the generation of documents by automatically creating and integrating input from different sources and assembling documents conforming to defined templates. There have been reviews of commercial solutions of DA, particularly in the legal domain, but to date there has been no comprehensive review of the academic research on DA architectures and technologies. The current survey of DA reviews the academic literature and provides a clearer definition and characterization of DA and its features, identifies state-of-the-art DA architectures and technologies in academic research, and provides ideas that can lead to new research opportunities within the DA field in light of recent advances in generative AI and large language models.
A Rank-One Optimization Framework and Its Applications to Transmit Beamforming
Venue: arXiv cs.IT 2023
Year: 2023

This paper proposes an elegant optimization framework consisting of a mix of linear-matrix-inequality and second-order-cone constraints. The proposed framework generalizes the semidefinite relaxation (SDR) enabled solution to the typical transmit beamforming problems presented in the form of quadratically constrained quadratic programs (QCQPs) in the literature. It is proved that the optimization problems subsumed under the framework always admit a rank-one optimal solution when they are feasible and their optimal solutions are not trivial. This finding indicates that the relaxation is tight as the optimal solution of the original beamforming QCQP can be straightforwardly obtained from that of the SDR counterpart without any loss of optimality. Four representative examples of transmit beamforming, i.e., transmit beamforming with perfect channel state information (CSI), transmit beamforming with imperfect CSI, chance-constraint approach for imperfect CSI, and reconfigurable-intelligent-surface (RIS) aided beamforming, are shown to demonstrate how the proposed optimization framework can be realized in deriving the SDR counterparts for different beamforming designs.
Fair Allocation Algorithms for Indivisible Items under Structured Conflict Constraints
Venue: arXiv cs.DM 2023
Year: 2023

We consider the fair allocation of indivisible items to several agents with additional conflict constraints. These are represented by a conflict graph where each item corresponds to a vertex of the graph and edges in the graph represent incompatible pairs of items which should not be allocated to the same agent. This setting combines the issues of Partition and Independent Set and can be seen as a partial coloring of the conflict graph. In the resulting optimization problem each agent has its own valuation function for the profits of the items. We aim at maximizing the lowest total profit obtained by any of the agents. In a previous paper this problem was shown to be strongly \NP-hard for several well-known graph classes, e.g., bipartite graphs and their line graphs. On the other hand, it was shown that pseudo-polynomial time algorithms exist for the classes of chordal graphs, cocomparability graphs, biconvex bipartite graphs, and graphs of bounded treewidth. In this contribution we extend this line of research by developing pseudo-polynomial time algorithms that solve the problem for the class of convex bipartite conflict graphs, graphs of bounded clique-width, and graphs of bounded tree-independence number. The algorithms are based on dynamic programming and also permit fully polynomial-time approximation schemes (FPTAS).
Transformer-based Detection of Microorganisms on High-Resolution Petri Dish Images
Venue: arXiv cs.CV 2023
Year: 2023

Many medical or pharmaceutical processes have strict guidelines regarding continuous hygiene monitoring. This often involves the labor-intensive task of manually counting microorganisms in Petri dishes by trained personnel. Automation attempts often struggle due to major challenges: significant scaling differences, low separation, low contrast, etc. To address these challenges, we introduce AttnPAFPN, a high-resolution detection pipeline that leverages a novel transformer variation, the efficient-global self-attention mechanism. Our streamlined approach can be easily integrated in almost any multi-scale object detection pipeline. In a comprehensive evaluation on the publicly available AGAR dataset, we demonstrate the superior accuracy of our network over the current state-of-the-art. In order to demonstrate the task-independent performance of our approach, we perform further experiments on COCO and LIVECell datasets.
BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine
Venue: arXiv cs.CE 2023
Year: 2023

Foundation models (FMs) have exhibited remarkable performance across a wide range of downstream tasks in many domains. Nevertheless, general-purpose FMs often face challenges when confronted with domain-specific problems, due to their limited access to the proprietary training data in a particular domain. In biomedicine, there are various biological modalities, such as molecules, proteins, and cells, which are encoded by the language of life and exhibit significant modality gaps with human natural language. In this paper, we introduce BioMedGPT, an open multimodal generative pre-trained transformer (GPT) for biomedicine, to bridge the gap between the language of life and human natural language. BioMedGPT allows users to easily ``communicate'' with diverse biological modalities through free text, which is the first of its kind. BioMedGPT aligns different biological modalities with natural language via a large generative language model, namely, BioMedGPT-LM. We publish BioMedGPT-10B, which unifies the feature spaces of molecules, proteins, and natural language via encoding and alignment. Through fine-tuning, BioMedGPT-10B outperforms or is on par with human and significantly larger general-purpose foundation models on the biomedical QA task. It also demonstrates promising performance in the molecule QA and protein QA tasks, which could greatly accelerate the discovery of new drugs and therapeutic targets. In addition, BioMedGPT-LM-7B is the first large generative language model based on Llama2 in the biomedical domain, therefore is commercial friendly. Both BioMedGPT-10B and BioMedGPT-LM-7B are open-sourced to the research community. In addition, we publish the datasets that are meticulously curated for the alignment of multi-modalities, i.e., PubChemQA and UniProtQA. All the models, codes, and datasets are available at \url{https://github.com/PharMolix/OpenBioMed}.
A new branch-and-cut approach for integrated planning in additive manufacturing
Venue: arXiv cs.DM 2023
Year: 2023

In recent years, there has been considerable interest in the transformative potential of additive manufacturing (AM) since it allows for producing highly customizable and complex components while reducing lead times and costs. The rise of AM for traditional and new business models enforces the need for efficient planning procedures for AM facilities. In this area, the assignment and sequencing of components to be built by an AM machine, also called a 3D printer, is a complex problem joining the nesting and scheduling of parts to be printed. This paper proposes a new branch-and-cut algorithm for integrated planning for unrelated parallel machines. The algorithm is based on combinatorial Benders decomposition: The scheduling problem is considered in the master problem, while the feasibility of a solution is checked in the sub-problem. Current state-of-the-art techniques are extended to solve the orthogonal packing with rotation to speed up the solution of the sub-problem. Extensive computational tests on existing instances and a new benchmark instance set show the algorithm's superior performance compared to an existing integrated mixed-integer programming model.
Balancing Transparency and Risk: The Security and Privacy Risks of Open-Source Machine Learning Models
Venue: arXiv cs.LG 2023
Year: 2023

The field of artificial intelligence (AI) has experienced remarkable progress in recent years, driven by the widespread adoption of open-source machine learning models in both research and industry. Considering the resource-intensive nature of training on vast datasets, many applications opt for models that have already been trained. Hence, a small number of key players undertake the responsibility of training and publicly releasing large pre-trained models, providing a crucial foundation for a wide range of applications. However, the adoption of these open-source models carries inherent privacy and security risks that are often overlooked. To provide a concrete example, an inconspicuous model may conceal hidden functionalities that, when triggered by specific input patterns, can manipulate the behavior of the system, such as instructing self-driving cars to ignore the presence of other vehicles. The implications of successful privacy and security attacks encompass a broad spectrum, ranging from relatively minor damage like service interruptions to highly alarming scenarios, including physical harm or the exposure of sensitive user data. In this work, we present a comprehensive overview of common privacy and security threats associated with the use of open-source models. By raising awareness of these dangers, we strive to promote the responsible and secure use of AI systems.
3D Model-free Visual Localization System from Essential Matrix under Local Planar Motion
Venue: arXiv cs.RO 2023
Year: 2023

Visual localization plays a critical role in the functionality of low-cost autonomous mobile robots. Current state-of-the-art approaches for achieving accurate visual localization are 3D scene-specific, requiring additional computational and storage resources to construct a 3D scene model when facing a new environment. An alternative approach of directly using a database of 2D images for visual localization offers more flexibility. However, such methods currently suffer from limited localization accuracy. In this paper, we propose an accurate and robust multiple checking-based 3D model-free visual localization system to address the aforementioned issues. To ensure high accuracy, our focus is on estimating the pose of a query image relative to the retrieved database images using 2D-2D feature matches. Theoretically, by incorporating the local planar motion constraint into both the estimation of the essential matrix and the triangulation stages, we reduce the minimum required feature matches for absolute pose estimation, thereby enhancing the robustness of outlier rejection. Additionally, we introduce a multiple-checking mechanism to ensure the correctness of the solution throughout the solving process. For validation, qualitative and quantitative experiments are performed on both simulation and two real-world datasets and the experimental results demonstrate a significant enhancement in both accuracy and robustness afforded by the proposed 3D model-free visual localization system.
Disparity, Inequality, and Accuracy Tradeoffs in Graph Neural Networks for Node Classification
Venue: arXiv cs.LG 2023
Year: 2023

Graph neural networks (GNNs) are increasingly used in critical human applications for predicting node labels in attributed graphs. Their ability to aggregate features from nodes' neighbors for accurate classification also has the capacity to exacerbate existing biases in data or to introduce new ones towards members from protected demographic groups. Thus, it is imperative to quantify how GNNs may be biased and to what extent their harmful effects may be mitigated. To this end, we propose two new GNN-agnostic interventions namely, (i) PFR-AX which decreases the separability between nodes in protected and non-protected groups, and (ii) PostProcess which updates model predictions based on a blackbox policy to minimize differences between error rates across demographic groups. Through a large set of experiments on four datasets, we frame the efficacies of our approaches (and three variants) in terms of their algorithmic fairness-accuracy tradeoff and benchmark our results against three strong baseline interventions on three state-of-the-art GNN models. Our results show that no single intervention offers a universally optimal tradeoff, but PFR-AX and PostProcess provide granular control and improve model confidence when correctly predicting positive outcomes for nodes in protected groups.
Far3D: Expanding the Horizon for Surround-view 3D Object Detection
Venue: arXiv cs.CV 2023
Year: 2023

Recently 3D object detection from surround-view images has made notable advancements with its low deployment cost. However, most works have primarily focused on close perception range while leaving long-range detection less explored. Expanding existing methods directly to cover long distances poses challenges such as heavy computation costs and unstable convergence. To address these limitations, this paper proposes a novel sparse query-based framework, dubbed Far3D. By utilizing high-quality 2D object priors, we generate 3D adaptive queries that complement the 3D global queries. To efficiently capture discriminative features across different views and scales for long-range objects, we introduce a perspective-aware aggregation module. Additionally, we propose a range-modulated 3D denoising approach to address query error propagation and mitigate convergence issues in long-range tasks. Significantly, Far3D demonstrates SoTA performance on the challenging Argoverse 2 dataset, covering a wide range of 150 meters, surpassing several LiDAR-based approaches. Meanwhile, Far3D exhibits superior performance compared to previous methods on the nuScenes dataset. The code is available at https://github.com/megvii-research/Far3D.
A Potts model approach to unsupervised graph clustering with Graph Neural Networks
Venue: arXiv cs.SI 2023
Year: 2023

Numerous approaches have been explored for graph clustering, including those which optimize a global criteria such as modularity. More recently, Graph Neural Networks (GNNs), which have produced state-of-the-art results in graph analysis tasks such as node classification and link prediction, have been applied for unsupervised graph clustering using these modularity-based metrics. Modularity, though robust for many practical applications, suffers from the resolution limit problem, in which optimization may fail to identify clusters smaller than a scale that is dependent on properties of the network. In this paper, we propose a new GNN framework which draws from the Potts model in physics to overcome this limitation. Experiments on a variety of real world datasets show that this model achieves state-of-the-art clustering results.
Quantifying Uncertainties of Contact Classifications in a Human-Robot Collaboration with Parallel Robots
Venue: arXiv cs.RO 2023
Year: 2023

In human-robot collaboration, unintentional physical contacts occur in the form of collisions and clamping, which must be detected and classified separately for a reaction. If certain collision or clamping situations are misclassified, reactions might occur that make the true contact case more dangerous. This work analyzes data-driven modeling based on physically modeled features like estimated external forces for clamping and collision classification with a real parallel robot. The prediction reliability of a feedforward neural network is investigated. Quantification of the classification uncertainty enables the distinction between safe versus unreliable classifications and optimal reactions like a retraction movement for collisions, structure opening for the clamping joint, and a fallback reaction in the form of a zero-g mode. This hypothesis is tested with experimental data of clamping and collision cases by analyzing dangerous misclassifications and then reducing them by the proposed uncertainty quantification. Finally, it is investigated how the approach of this work influences correctly classified clamping and collision scenarios.