OpenRaiser/Intern-Atlas huggingface dataset:
VisDA 2022 Challenge: Domain Adaptation for Industrial Waste Sorting
Venue: arXiv cs.CV 2023
Year: 2023

Label-efficient and reliable semantic segmentation is essential for many real-life applications, especially for industrial settings with high visual diversity, such as waste sorting. In industrial waste sorting, one of the biggest challenges is the extreme diversity of the input stream depending on factors like the location of the sorting facility, the equipment available in the facility, and the time of year, all of which significantly impact the composition and visual appearance of the waste stream. These changes in the data are called ``visual domains'', and label-efficient adaptation of models to such domains is needed for successful semantic segmentation of industrial waste. To test the abilities of computer vision models on this task, we present the VisDA 2022 Challenge on Domain Adaptation for Industrial Waste Sorting. Our challenge incorporates a fully-annotated waste sorting dataset, ZeroWaste, collected from two real material recovery facilities in different locations and seasons, as well as a novel procedurally generated synthetic waste sorting dataset, SynthWaste. In this competition, we aim to answer two questions: 1) can we leverage domain adaptation techniques to minimize the domain gap? and 2) can synthetic data augmentation improve performance on this task and help adapt to changing data distributions? The results of the competition show that industrial waste detection poses a real domain adaptation problem, that domain generalization techniques such as augmentations, ensembling, etc., improve the overall performance on the unlabeled target domain examples, and that leveraging synthetic data effectively remains an open problem. See https://ai.bu.edu/visda-2022/
Lightweight Estimation of Hand Mesh and Biomechanically Feasible Kinematic Parameters
Venue: arXiv cs.CV 2023
Year: 2023

3D hand pose estimation is a long-standing challenge in both robotics and computer vision communities due to its implicit depth ambiguity and often strong self-occlusion. Recently, in addition to the hand skeleton, jointly estimating hand pose and shape has gained more attraction. State-of-the-art methods adopt a model-free approach, estimating the vertices of the hand mesh directly and providing superior accuracy compared to traditional model-based methods directly regressing the parameters of the parametric hand mesh. However, with the large number of mesh vertices to estimate, these methods are often slow in inference. We propose an efficient variation of the previously proposed image-to-lixel approach to efficiently estimate hand meshes from the images. Leveraging recent developments in efficient neural architectures, we significantly reduce the computation complexity without sacrificing the estimation accuracy. Furthermore, we introduce an inverted kinematic(IK) network to translate the estimated hand mesh to a biomechanically feasible set of joint rotation parameters, which is necessary for applications that leverage pose estimation for controlling robotic hands. Finally, an optional post-processing module is proposed to refine the rotation and shape parameters to compensate for the error introduced by the IK net. Our Lite I2L Mesh Net achieves state-of-the-art joint and mesh estimation accuracy with less than $13\%$ of the total computational complexity of the original I2L hand mesh estimator. Adding the IK net and post-optimization modules can improve the accuracy slightly at a small computation cost, but more importantly, provide the kinematic parameters required for robotic applications.
TransCODE: Co-design of Transformers and Accelerators for Efficient Training and Inference
Venue: arXiv cs.LG 2023
Year: 2023

Automated co-design of machine learning models and evaluation hardware is critical for efficiently deploying such models at scale. Despite the state-of-the-art performance of transformer models, they are not yet ready for execution on resource-constrained hardware platforms. High memory requirements and low parallelizability of the transformer architecture exacerbate this problem. Recently-proposed accelerators attempt to optimize the throughput and energy consumption of transformer models. However, such works are either limited to a one-sided search of the model architecture or a restricted set of off-the-shelf devices. Furthermore, previous works only accelerate model inference and not training, which incurs substantially higher memory and compute resources, making the problem even more challenging. To address these limitations, this work proposes a dynamic training framework, called DynaProp, that speeds up the training process and reduces memory consumption. DynaProp is a low-overhead pruning method that prunes activations and gradients at runtime. To effectively execute this method on hardware for a diverse set of transformer architectures, we propose ELECTOR, a framework that simulates transformer inference and training on a design space of accelerators. We use this simulator in conjunction with the proposed co-design technique, called TransCODE, to obtain the best-performing models with high accuracy on the given task and minimize latency, energy consumption, and chip area. The obtained transformer-accelerator pair achieves 0.3% higher accuracy than the state-of-the-art pair while incurring 5.2$\times$ lower latency and 3.0$\times$ lower energy consumption.
GelSight Baby Fin Ray: A Compact, Compliant, Flexible Finger with High-Resolution Tactile Sensing
Venue: arXiv cs.RO 2023
Year: 2023

The synthesis of tactile sensing with compliance is essential to many fields, from agricultural usages like fruit picking, to sustainability practices such as sorting recycling, to the creation of safe home-care robots for the elderly to age with dignity. From tactile sensing, we can discern material properties, recognize textures, and determine softness, while with compliance, we are able to securely and safely interact with the objects and the environment around us. These two abilities can culminate into a useful soft robotic gripper, such as the original GelSight Fin Ray, which is able to grasp a large variety of different objects and also perform a simple household manipulation task: wine glass reorientation. Although the original GelSight Fin Ray solves the problem of interfacing a generally rigid, high-resolution sensor with a soft, compliant structure, we can improve the robustness of the sensor and implement techniques that make such camera-based tactile sensors applicable to a wider variety of soft robot designs. We first integrate flexible mirrors and incorporate the rigid electronic components into the base of the gripper, which greatly improves the compliance of the Fin Ray structure. Then, we synthesize a flexible and high-elongation silicone adhesive-based fluorescent paint, which can provide good quality 2D tactile localization results for our sensor. Finally, we incorporate all of these techniques into a new design: the Baby Fin Ray, which we use to dig through clutter, and perform successful classification of nuts in their shells. The supplementary video can be found here: https://youtu.be/_oD_QFtYTPM
Directed Greybox Fuzzing with Stepwise Constraint Focusing
Venue: arXiv cs.CR 2023
Year: 2023

Dynamic data flow analysis has been widely used to guide greybox fuzzing. However, traditional dynamic data flow analysis tends to go astray in the massive path tracking and requires to process a large volume of data, resulting in low efficiency in reaching the target location. In this paper, we propose a directed greybox fuzzer based on dynamic constraint filtering and focusing (CONFF). First, all path constraints are tracked, and those with high priority are filtered as the next solution targets. Next, focusing on a single path constraint to be satisfied, we obtain its data condition and probe the mapping relationship between it and the input bytes through multi-byte mapping and single-byte mapping. Finally, various mutation strategies are utilized to solve the path constraint currently focused on, and the target location of the program is gradually approached through path selection. The CONFF fuzzer can reach a specific location faster in the target program, thus efficiently triggering the crash. We designed and implemented a prototype of the CONFF fuzzer and evaluated it with the LAVA-1 dataset and some real-world vulnerabilities. The results show that the CONFF fuzzer can reproduce crashes on the LAVA-1 dataset and most of the real-world vulnerabilities. For most vulnerabilities, the CONFF fuzzer reproduced the crashes with significantly reduced time compared to state-of-the-art fuzzers. On average, the CONFF fuzzer was 23.7x faster than the state-of-the-art code coverage-based fuzzer Angora and 27.3x faster than the classical directed greybox fuzzer AFLGo.
Finite Strain Topology Optimization with Nonlinear Stability Constraints
Venue: arXiv cs.CE 2023
Year: 2023

This paper proposes a computational framework for the design optimization of stable structures under large deformations by incorporating nonlinear buckling constraints. A novel strategy for suppressing spurious buckling modes related to low-density elements is proposed. The strategy depends on constructing a pseudo-mass matrix that assigns small pseudo masses for DOFs surrounded by only low-density elements and degenerates to an identity matrix for the solid region. A novel optimization procedure is developed that can handle both simple and multiple eigenvalues wherein consistent sensitivities of simple eigenvalues and directional derivatives of multiple eigenvalues are derived and utilized in a gradient-based optimization algorithm - the method of moving asymptotes. An adaptive linear energy interpolation method is also incorporated in nonlinear analyses to handle the low-density elements distortion under large deformations. The numerical results demonstrate that, for systems with either low or high symmetries, the nonlinear stability constraints can ensure structural stability at the target load under large deformations. Post-analysis on the B-spline fitted designs shows that the safety margin, i.e., the gap between the target load and the 1st critical load, of the optimized structures can be well controlled by selecting different stability constraint values. Interesting structural behaviors such as mode switching and multiple bifurcations are also demonstrated.
Adapting Pretrained Language Models for Solving Tabular Prediction Problems in the Electronic Health Record
Venue: arXiv cs.CL 2023
Year: 2023

We propose an approach for adapting the DeBERTa model for electronic health record (EHR) tasks using domain adaptation. We pretrain a small DeBERTa model on a dataset consisting of MIMIC-III discharge summaries, clinical notes, radiology reports, and PubMed abstracts. We compare this model's performance with a DeBERTa model pre-trained on clinical texts from our institutional EHR (MeDeBERTa) and an XGBoost model. We evaluate performance on three benchmark tasks for emergency department outcomes using the MIMIC-IV-ED dataset. We preprocess the data to convert it into text format and generate four versions of the original datasets to compare data processing and data inclusion. The results show that our proposed approach outperforms the alternative models on two of three tasks (p<0.001) and matches performance on the third task, with the use of descriptive columns improving performance over the original column names.
Verification Of Partial Quantifier Elimination
Venue: arXiv cs.LO 2023
Year: 2023

Quantifier elimination (QE) is an important problem that has numerous applications. Unfortunately, QE is computationally very hard. Earlier we introduced a generalization of QE called $\mathit{partial}$ QE (or PQE for short). PQE allows to unquantify a $\mathit{part}$ of the formula. The appeal of PQE is twofold. First, many important problems can be solved in terms of PQE. Second, PQE can be drastically faster than QE if only a small part of the formula gets unquantified. To make PQE practical, one needs an algorithm for verifying the solution produced by a PQE solver. In this paper, we describe a very simple SAT-based verifier called $\mathit{VerPQE}$ and provide some experimental results.
Implementation-First Approach of Developing Formal Semantics of a Simulation Language in VDM-SL
Venue: arXiv cs.SE 2023
Year: 2023

Formal specification is a basis for rigorous software implementation. VDM-SL is a formal specification language with an extensive executable subset. Successful cases of VDM-family including VDM-SL have shown that producing a well-tested executable specification can reduce the cost of the implementation phase. This paper introduces and discusses the reversed order of specification and implementation. The development of a multi-agent simulation language called \remobidyc is described and examined as a case study of defining a formal specification after initial implementation and reflecting the specification into the implementation code.
Diffusion Denoised Smoothing for Certified and Adversarial Robust Out-Of-Distribution Detection
Venue: arXiv cs.LG 2023
Year: 2023

As the use of machine learning continues to expand, the importance of ensuring its safety cannot be overstated. A key concern in this regard is the ability to identify whether a given sample is from the training distribution, or is an "Out-Of-Distribution" (OOD) sample. In addition, adversaries can manipulate OOD samples in ways that lead a classifier to make a confident prediction. In this study, we present a novel approach for certifying the robustness of OOD detection within a $\ell_2$-norm around the input, regardless of network architecture and without the need for specific components or additional training. Further, we improve current techniques for detecting adversarial attacks on OOD samples, while providing high levels of certified and adversarial robustness on in-distribution samples. The average of all OOD detection metrics on CIFAR10/100 shows an increase of $\sim 13 \% / 5\%$ relative to previous approaches.
An End-to-End Framework For Universal Lesion Detection With Missing Annotations
Venue: arXiv cs.CV 2023
Year: 2023

Fully annotated large-scale medical image datasets are highly valuable. However, because labeling medical images is tedious and requires specialized knowledge, the large-scale datasets available often have missing annotation issues. For instance, DeepLesion, a large-scale CT image dataset with labels for various kinds of lesions, is reported to have a missing annotation rate of 50\%. Directly training a lesion detector on it would suffer from false negative supervision caused by unannotated lesions. To address this issue, previous works have used sophisticated multi-stage strategies to switch between lesion mining and detector training. In this work, we present a novel end-to-end framework for mining unlabeled lesions while simultaneously training the detector. Our framework follows the teacher-student paradigm. In each iteration, the teacher model infers the input data and creates a set of predictions. High-confidence predictions are combined with partially-labeled ground truth for training the student model. On the DeepLesion dataset, using the original partially labeled training set, our model can outperform all other more complicated methods and surpass the previous best method by 2.3\% on average sensitivity and 2.7\% on average precision, achieving state-of-the-art universal lesion detection results.
Improving the Transferability of Adversarial Examples via Direction Tuning
Venue: arXiv cs.CV 2023
Year: 2023

In the transfer-based adversarial attacks, adversarial examples are only generated by the surrogate models and achieve effective perturbation in the victim models. Although considerable efforts have been developed on improving the transferability of adversarial examples generated by transfer-based adversarial attacks, our investigation found that, the big deviation between the actual and steepest update directions of the current transfer-based adversarial attacks is caused by the large update step length, resulting in the generated adversarial examples can not converge well. However, directly reducing the update step length will lead to serious update oscillation so that the generated adversarial examples also can not achieve great transferability to the victim models. To address these issues, a novel transfer-based attack, namely direction tuning attack, is proposed to not only decrease the update deviation in the large step length, but also mitigate the update oscillation in the small sampling step length, thereby making the generated adversarial examples converge well to achieve great transferability on victim models. In addition, a network pruning method is proposed to smooth the decision boundary, thereby further decreasing the update oscillation and enhancing the transferability of the generated adversarial examples. The experiment results on ImageNet demonstrate that the average attack success rate (ASR) of the adversarial examples generated by our method can be improved from 87.9\% to 94.5\% on five victim models without defenses, and from 69.1\% to 76.2\% on eight advanced defense methods, in comparison with that of latest gradient-based attacks.
LMCanvas: Object-Oriented Interaction to Personalize Large Language Model-Powered Writing Environments
Venue: arXiv cs.HC 2023
Year: 2023

Large language models (LLMs) can enhance writing by automating or supporting specific tasks in writers' workflows (e.g., paraphrasing, creating analogies). Leveraging this capability, a collection of interfaces have been developed that provide LLM-powered tools for specific writing tasks. However, these interfaces provide limited support for writers to create personal tools for their own unique tasks, and may not comprehensively fulfill a writer's needs -- requiring them to continuously switch between interfaces during writing. In this work, we envision LMCanvas, an interface that enables writers to create their own LLM-powered writing tools and arrange their personal writing environment by interacting with "blocks" in a canvas. In this interface, users can create text blocks to encapsulate writing and LLM prompts, model blocks for model parameter configurations, and connect these to create pipeline blocks that output generations. In this workshop paper, we discuss the design for LMCanvas and our plans to develop this concept.
Hybrid Augmented Automated Graph Contrastive Learning
Venue: arXiv cs.LG 2023
Year: 2023

Graph augmentations are essential for graph contrastive learning. Most existing works use pre-defined random augmentations, which are usually unable to adapt to different input graphs and fail to consider the impact of different nodes and edges on graph semantics. To address this issue, we propose a framework called Hybrid Augmented Automated Graph Contrastive Learning (HAGCL). HAGCL consists of a feature-level learnable view generator and an edge-level learnable view generator. The view generators are end-to-end differentiable to learn the probability distribution of views conditioned on the input graph. It insures to learn the most semantically meaningful structure in terms of features and topology, respectively. Furthermore, we propose an improved joint training strategy, which can achieve better results than previous works without resorting to any weak label information in the downstream tasks and extensive evaluation of additional work.
Probing optimisation in physics-informed neural networks
Venue: arXiv cs.LG 2023
Year: 2023

A novel comparison is presented of the effect of optimiser choice on the accuracy of physics-informed neural networks (PINNs). To give insight into why some optimisers are better, a new approach is proposed that tracks the training trajectory curvature and can be evaluated on the fly at a low computational cost. The linear advection equation is studied for several advective velocities, and we show that the optimiser choice substantially impacts PINNs model performance and accuracy. Furthermore, using the curvature measure, we found a negative correlation between the convergence error and the curvature in the optimiser local reference frame. It is concluded that, in this case, larger local curvature values result in better solutions. Consequently, optimisation of PINNs is made more difficult as minima are in highly curved regions.
Digital Twin of a Network and Operating Environment Using Augmented Reality
Venue: arXiv cs.NI 2023
Year: 2023

We demonstrate the digital twin of a network, network elements, and operating environment using machine learning. We achieve network card failure localization and remote collaboration over 86 km of fiber using augmented reality.
Unconditionally secure ciphers with a short key for a source with unknown statistics
Venue: arXiv cs.CR 2023
Year: 2023

We consider the problem of constructing an unconditionally secure cipher with a short key for the case where the probability distribution of encrypted messages is unknown. Note that unconditional security means that an adversary with no computational constraints can obtain only a negligible amount of information ("leakage") about an encrypted message (without knowing the key). Here we consider the case of a priori (partially) unknown message source statistics. More specifically, the message source probability distribution belongs to a given family of distributions. We propose an unconditionally secure cipher for this case. As an example, one can consider constructing a single cipher for texts written in any of the languages of the European Union. That is, the message to be encrypted could be written in any of these languages.
From Self-Adaptation to Self-Evolution Leveraging the Operational Design Domain
Venue: arXiv cs.SE 2023
Year: 2023

Engineering long-running computing systems that achieve their goals under ever-changing conditions pose significant challenges. Self-adaptation has shown to be a viable approach to dealing with changing conditions. Yet, the capabilities of a self-adaptive system are constrained by its operational design domain (ODD), i.e., the conditions for which the system was built (requirements, constraints, and context). Changes, such as adding new goals or dealing with new contexts, require system evolution. While the system evolution process has been automated substantially, it remains human-driven. Given the growing complexity of computing systems, human-driven evolution will eventually become unmanageable. In this paper, we provide a definition for ODD and apply it to a self-adaptive system. Next, we explain why conditions not covered by the ODD require system evolution. Then, we outline a new approach for self-evolution that leverages the concept of ODD, enabling a system to evolve autonomously to deal with conditions not anticipated by its initial ODD. We conclude with open challenges to realise self-evolution.
An Analytical Representation of the 2d Generalized Balanced Power Diagram
Venue: arXiv cs.CG 2023
Year: 2023

Tessellations are an important tool to model the microstructure of cellular and polycrystalline materials. Classical tessellation models include the Voronoi diagram and Laguerre tessellation whose cells are polyhedra. Due to the convexity of their cells, those models may be too restrictive to describe data that includes possibly anisotropic grains with curved boundaries. Several generalizations exist. The cells of the generalized balanced power diagram are induced by elliptic distances leading to more diverse structures. So far, methods for computing the generalized balanced power diagram are restricted to discretized versions in the form of label images. In this work, we derive an analytic representation of the vertices and edges of the generalized balanced power diagram in 2d. Based on that, we propose a novel algorithm to compute the whole diagram.