OpenRaiser/Intern-Atlas huggingface dataset:
Control3Diff: Learning Controllable 3D Diffusion Models from Single-view Images
Venue: arXiv cs.CV 2023
Year: 2023

Diffusion models have recently become the de-facto approach for generative modeling in the 2D domain. However, extending diffusion models to 3D is challenging due to the difficulties in acquiring 3D ground truth data for training. On the other hand, 3D GANs that integrate implicit 3D representations into GANs have shown remarkable 3D-aware generation when trained only on single-view image datasets. However, 3D GANs do not provide straightforward ways to precisely control image synthesis. To address these challenges, We present Control3Diff, a 3D diffusion model that combines the strengths of diffusion models and 3D GANs for versatile, controllable 3D-aware image synthesis for single-view datasets. Control3Diff explicitly models the underlying latent distribution (optionally conditioned on external inputs), thus enabling direct control during the diffusion process. Moreover, our approach is general and applicable to any type of controlling input, allowing us to train it with the same diffusion objective without any auxiliary supervision. We validate the efficacy of Control3Diff on standard image generation benchmarks, including FFHQ, AFHQ, and ShapeNet, using various conditioning inputs such as images, sketches, and text prompts. Please see the project website (\url{https://jiataogu.me/control3diff}) for video comparisons.
How Will It Drape Like? Capturing Fabric Mechanics from Depth Images
Venue: arXiv cs.CV 2023
Year: 2023

We propose a method to estimate the mechanical parameters of fabrics using a casual capture setup with a depth camera. Our approach enables to create mechanically-correct digital representations of real-world textile materials, which is a fundamental step for many interactive design and engineering applications. As opposed to existing capture methods, which typically require expensive setups, video sequences, or manual intervention, our solution can capture at scale, is agnostic to the optical appearance of the textile, and facilitates fabric arrangement by non-expert operators. To this end, we propose a sim-to-real strategy to train a learning-based framework that can take as input one or multiple images and outputs a full set of mechanical parameters. Thanks to carefully designed data augmentation and transfer learning protocols, our solution generalizes to real images despite being trained only on synthetic data, hence successfully closing the sim-to-real loop.Key in our work is to demonstrate that evaluating the regression accuracy based on the similarity at parameter space leads to an inaccurate distances that do not match the human perception. To overcome this, we propose a novel metric for fabric drape similarity that operates on the image domain instead on the parameter space, allowing us to evaluate our estimation within the context of a similarity rank. We show that out metric correlates with human judgments about the perception of drape similarity, and that our model predictions produce perceptually accurate results compared to the ground truth parameters.
STU-Net: Scalable and Transferable Medical Image Segmentation Models Empowered by Large-Scale Supervised Pre-training
Venue: arXiv cs.CV 2023
Year: 2023

Large-scale models pre-trained on large-scale datasets have profoundly advanced the development of deep learning. However, the state-of-the-art models for medical image segmentation are still small-scale, with their parameters only in the tens of millions. Further scaling them up to higher orders of magnitude is rarely explored. An overarching goal of exploring large-scale models is to train them on large-scale medical segmentation datasets for better transfer capacities. In this work, we design a series of Scalable and Transferable U-Net (STU-Net) models, with parameter sizes ranging from 14 million to 1.4 billion. Notably, the 1.4B STU-Net is the largest medical image segmentation model to date. Our STU-Net is based on nnU-Net framework due to its popularity and impressive performance. We first refine the default convolutional blocks in nnU-Net to make them scalable. Then, we empirically evaluate different scaling combinations of network depth and width, discovering that it is optimal to scale model depth and width together. We train our scalable STU-Net models on a large-scale TotalSegmentator dataset and find that increasing model size brings a stronger performance gain. This observation reveals that a large model is promising in medical image segmentation. Furthermore, we evaluate the transferability of our model on 14 downstream datasets for direct inference and 3 datasets for further fine-tuning, covering various modalities and segmentation targets. We observe good performance of our pre-trained model in both direct inference and fine-tuning. The code and pre-trained models are available at https://github.com/Ziyan-Huang/STU-Net.
End-to-end codesign of Hessian-aware quantized neural networks for FPGAs and ASICs
Venue: arXiv cs.LG 2023
Year: 2023

We develop an end-to-end workflow for the training and implementation of co-designed neural networks (NNs) for efficient field-programmable gate array (FPGA) and application-specific integrated circuit (ASIC) hardware. Our approach leverages Hessian-aware quantization (HAWQ) of NNs, the Quantized Open Neural Network Exchange (QONNX) intermediate representation, and the hls4ml tool flow for transpiling NNs into FPGA and ASIC firmware. This makes efficient NN implementations in hardware accessible to nonexperts, in a single open-sourced workflow that can be deployed for real-time machine learning applications in a wide range of scientific and industrial settings. We demonstrate the workflow in a particle physics application involving trigger decisions that must operate at the 40 MHz collision rate of the CERN Large Hadron Collider (LHC). Given the high collision rate, all data processing must be implemented on custom ASIC and FPGA hardware within a strict area and latency. Based on these constraints, we implement an optimized mixed-precision NN classifier for high-momentum particle jets in simulated LHC proton-proton collisions.
Murder by design: Design thinking approach for pre-emptive cybernetic security design
Venue: arXiv cs.HC 2023
Year: 2023

In an ever more connected world, awareness has grown towards the hazards and vulnerabilities that the networking on sensitive digitized information pose for all parties involved. This vulnerability rests in a number of factors, both human and technical.From an ethical perspective, this means people seeking to maximise their own gain, and accomplish their goals through exploiting information existing in cyber space at the expense of other individuals and parties. One matter that is yet to be fully explored is the eventuality of not only financial information and other sensitive material being globally connected on the information highways, but also the people themselves as physical beings. Humans are natural born cyborgs who have integrated technology into their being throughout history. Issues of cyber security are extended to cybernetic security, which not only has severe ethical implications for how we, policy makers, academics, scientists, designers etc., define ethics in relation to humanity and human rights, but also the security and safety of merged organic and artificial systems and ecosystems.
A Dynamic Adverse Selection Multiagent Model with Off-Menu Actions
Venue: arXiv cs.GT 2023
Year: 2023

In dynamic mechanism design literature, one critical aspect has been typically ignored-the agents' periodic participation, which they can adapt and plan strategically. We propose a framework for dynamic principal-multiagent problems, augmenting the classic model by incorporating agents' periodic coupled decisions on participation and regular action selections. The principal faces adverse selection and designs a mechanism comprising a task policy profile (defining evolving agent action menus), a coupling policy profile (affecting agent utilities), and an off-switch function profile (assigning rewards or penalties upon agent withdrawal). Firstly, we introduce payoff-flow conservation-a sufficient condition to ensure dynamic incentive compatibility for regular actions. Secondly, we formulate a unique process, persistence transformation, which integrates task policy's implicit functions, enabling a closed-form off-switch function derivation, hence securing sufficient conditions for agents' coupled decisions' incentive compatibility, aligning with the principal's preferences. Thirdly, we go beyond the traditional envelope theorem by presenting a necessary condition for incentive compatibility, leveraging the coupled optimality of principal-desired actions. This approach helps explicitly formulate both the coupling and off-switch functions. Finally, we establish envelope-like conditions exclusively on the task policies, facilitating the application of the first-order approach.
Vax-Culture: A Dataset for Studying Vaccine Discourse on Twitter
Venue: arXiv cs.SI 2023
Year: 2023

Vaccine hesitancy continues to be a main challenge for public health officials during the COVID-19 pandemic. As this hesitancy undermines vaccine campaigns, many researchers have sought to identify its root causes, finding that the increasing volume of anti-vaccine misinformation on social media platforms is a key element of this problem. We explored Twitter as a source of misleading content with the goal of extracting overlapping cultural and political beliefs that motivate the spread of vaccine misinformation. To do this, we have collected a data set of vaccine-related Tweets and annotated them with the help of a team of annotators with a background in communications and journalism. Ultimately we hope this can lead to effective and targeted public health communication strategies for reaching individuals with anti-vaccine beliefs. Moreover, this information helps with developing Machine Learning models to automatically detect vaccine misinformation posts and combat their negative impacts. In this paper, we present Vax-Culture, a novel Twitter COVID-19 dataset consisting of 6373 vaccine-related tweets accompanied by an extensive set of human-provided annotations including vaccine-hesitancy stance, indication of any misinformation in tweets, the entities criticized and supported in each tweet and the communicated message of each tweet. Moreover, we define five baseline tasks including four classification and one sequence generation tasks, and report the results of a set of recent transformer-based models for them. The dataset and code are publicly available at https://github.com/mrzarei5/Vax-Culture.
AutoSplice: A Text-prompt Manipulated Image Dataset for Media Forensics
Venue: arXiv cs.CV 2023
Year: 2023

Recent advancements in language-image models have led to the development of highly realistic images that can be generated from textual descriptions. However, the increased visual quality of these generated images poses a potential threat to the field of media forensics. This paper aims to investigate the level of challenge that language-image generation models pose to media forensics. To achieve this, we propose a new approach that leverages the DALL-E2 language-image model to automatically generate and splice masked regions guided by a text prompt. To ensure the creation of realistic manipulations, we have designed an annotation platform with human checking to verify reasonable text prompts. This approach has resulted in the creation of a new image dataset called AutoSplice, containing 5,894 manipulated and authentic images. Specifically, we have generated a total of 3,621 images by locally or globally manipulating real-world image-caption pairs, which we believe will provide a valuable resource for developing generalized detection methods in this area. The dataset is evaluated under two media forensic tasks: forgery detection and localization. Our extensive experiments show that most media forensic models struggle to detect the AutoSplice dataset as an unseen manipulation. However, when fine-tuned models are used, they exhibit improved performance in both tasks.
Toward Real-Time Image Annotation Using Marginalized Coupled Dictionary Learning
Venue: arXiv cs.CV 2023
Year: 2023

In most image retrieval systems, images include various high-level semantics, called tags or annotations. Virtually all the state-of-the-art image annotation methods that handle imbalanced labeling are search-based techniques which are time-consuming. In this paper, a novel coupled dictionary learning approach is proposed to learn a limited number of visual prototypes and their corresponding semantics simultaneously. This approach leads to a real-time image annotation procedure. Another contribution of this paper is that utilizes a marginalized loss function instead of the squared loss function that is inappropriate for image annotation with imbalanced labels. We have employed a marginalized loss function in our method to leverage a simple and effective method of prototype updating. Meanwhile, we have introduced ${\ell}_1$ regularization on semantic prototypes to preserve the sparse and imbalanced nature of labels in learned semantic prototypes. Finally, comprehensive experimental results on various datasets demonstrate the efficiency of the proposed method for image annotation tasks in terms of accuracy and time. The reference implementation is publicly available on https://github.com/hamid-amiri/MCDL-Image-Annotation.
Energy-Efficient UAV Communications in the Presence of Wind: 3D Modeling and Trajectory Design
Venue: arXiv cs.IT 2023
Year: 2023

The rapid development of unmanned aerial vehicle (UAV) technology provides flexible communication services to terrestrial nodes. Energy efficiency is crucial to the deployment of UAVs, especially rotary-wing UAVs whose propulsion power is sensitive to the wind effect. In this paper, we first derive a three-dimensional (3D) generalised propulsion energy consumption model (GPECM) for rotary-wing UAVs under the consideration of stochastic wind modeling and 3D force analysis. Based on the GPECM, we study a UAV-enabled downlink communication system, where a rotary-wing UAV flies subject to stochastic wind disturbance and provides communication services for ground users (GUs). We aim to maximize the energy efficiency (EE) of the UAV by jointly optimizing the 3D trajectory and user scheduling among the GUs based on the GPECM. We formulate the problem as stochastic optimization, which is difficult to solve due to the lack of real-time wind information. To address this issue, we propose an offline-based online adaptive (OBOA) design with two phases, namely, an offline phase and an online phase. In the offline phase, we average the wind effect on the UAV by leveraging stochastic programming (SP) based on wind statistics; then, in the online phase, we further optimize the instantaneous velocity to adapt the real-time wind. Simulation results show that the optimized trajectories of the UAV in both two phases can better adapt to the wind in changing speed and direction, and achieves a higher EE compared with the windless scheme. In particular, our proposed OBOA design can be applied in the scenario with dramatic wind changes, and makes the UAV adjust its velocity dynamically to achieve a better performance in terms of EE.
Interpretability is a Kind of Safety: An Interpreter-based Ensemble for Adversary Defense
Venue: arXiv cs.LG 2023
Year: 2023

While having achieved great success in rich real-life applications, deep neural network (DNN) models have long been criticized for their vulnerability to adversarial attacks. Tremendous research efforts have been dedicated to mitigating the threats of adversarial attacks, but the essential trait of adversarial examples is not yet clear, and most existing methods are yet vulnerable to hybrid attacks and suffer from counterattacks. In light of this, in this paper, we first reveal a gradient-based correlation between sensitivity analysis-based DNN interpreters and the generation process of adversarial examples, which indicates the Achilles's heel of adversarial attacks and sheds light on linking together the two long-standing challenges of DNN: fragility and unexplainability. We then propose an interpreter-based ensemble framework called X-Ensemble for robust adversary defense. X-Ensemble adopts a novel detection-rectification process and features in building multiple sub-detectors and a rectifier upon various types of interpretation information toward target classifiers. Moreover, X-Ensemble employs the Random Forests (RF) model to combine sub-detectors into an ensemble detector for adversarial hybrid attacks defense. The non-differentiable property of RF further makes it a precious choice against the counterattack of adversaries. Extensive experiments under various types of state-of-the-art attacks and diverse attack scenarios demonstrate the advantages of X-Ensemble to competitive baseline methods.
A Design Guideline to Overcome Web Accessibility Issues Challenged by Visually Impaired Community in Sri Lanka
Venue: arXiv cs.CY 2023
Year: 2023

Visual-impaired communities are one of the hindrances groups to accessing web content access in the world. The obstacles encountered by this community in their current practices and to develop best practice guidelines to overcome the digital divide in Sri Lanka become gap filling of this domain. A preliminary survey indicated five main problems including access limited by the impairment, usability issues due to lack of design, unavailability of visually impaired-friendly applications, lack of communication, and web navigation issues are the most dominant pertaining issues. To overcome those issues, solutions are tested and validated using the Design Science approach. Result Indicate that significant factors need to be incorporated, ensuring keyboard-friendly websites, easy accessibility and support with semantic annotation by adding alternative text for images. Furthermore, use headers to structure the content correctly, design all forms to support accessibility in mind including Content developing and designing, navigation, the best colour combination, Pre-recorded video with the audio facilities, braille support on the web, the designing option has no significant impact on visually impaired web users. Introducing a rating widget option to a website identifies the level of accessibility features availability facilitates, thereby overcoming the disability digital divide. The results further conclude that a significant difference exists in websites, with and without the involvement of the visually impaired community. Semantic web and semantic annotations of the context of page elements, content serialization, and navigation by special keyboard commands are also highly influencing the effective use of the web and increasing the satisfaction level in the website accessing process.
Study on Soft Robotic Pinniped Locomotion
Venue: arXiv cs.RO 2023
Year: 2023

Legged locomotion is a highly promising but under-researched subfield within the field of soft robotics. The compliant limbs of soft-limbed robots offer numerous benefits, including the ability to regulate impacts, tolerate falls, and navigate through tight spaces. These robots have the potential to be used for various applications, such as search and rescue, inspection, surveillance, and more. The state-of-the-art still faces many challenges, including limited degrees of freedom, a lack of diversity in gait trajectories, insufficient limb dexterity, and limited payload capabilities. To address these challenges, we develop a modular soft-limbed robot that can mimic the locomotion of pinnipeds. By using a modular design approach, we aim to create a robot that has improved degrees of freedom, gait trajectory diversity, limb dexterity, and payload capabilities. We derive a complete floating-base kinematic model of the proposed robot and use it to generate and experimentally validate a variety of locomotion gaits. Results show that the proposed robot is capable of replicating these gaits effectively. We compare the locomotion trajectories under different gait parameters against our modeling results to demonstrate the validity of our proposed gait models.
Self-Supervised Learning based Depth Estimation from Monocular Images
Venue: arXiv cs.CV 2023
Year: 2023

Depth Estimation has wide reaching applications in the field of Computer vision such as target tracking, augmented reality, and self-driving cars. The goal of Monocular Depth Estimation is to predict the depth map, given a 2D monocular RGB image as input. The traditional depth estimation methods are based on depth cues and used concepts like epipolar geometry. With the evolution of Convolutional Neural Networks, depth estimation has undergone tremendous strides. In this project, our aim is to explore possible extensions to existing SoTA Deep Learning based Depth Estimation Models and to see whether performance metrics could be further improved. In a broader sense, we are looking at the possibility of implementing Pose Estimation, Efficient Sub-Pixel Convolution Interpolation, Semantic Segmentation Estimation techniques to further enhance our proposed architecture and to provide fine-grained and more globally coherent depth map predictions. We also plan to do away with camera intrinsic parameters during training and apply weather augmentations to further generalize our model.
Domain shifts in dermoscopic skin cancer datasets: Evaluation of essential limitations for clinical translation
Venue: arXiv cs.CV 2023
Year: 2023

The limited ability of Convolutional Neural Networks to generalize to images from previously unseen domains is a major limitation, in particular, for safety-critical clinical tasks such as dermoscopic skin cancer classification. In order to translate CNN-based applications into the clinic, it is essential that they are able to adapt to domain shifts. Such new conditions can arise through the use of different image acquisition systems or varying lighting conditions. In dermoscopy, shifts can also occur as a change in patient age or occurence of rare lesion localizations (e.g. palms). These are not prominently represented in most training datasets and can therefore lead to a decrease in performance. In order to verify the generalizability of classification models in real world clinical settings it is crucial to have access to data which mimics such domain shifts. To our knowledge no dermoscopic image dataset exists where such domain shifts are properly described and quantified. We therefore grouped publicly available images from ISIC archive based on their metadata (e.g. acquisition location, lesion localization, patient age) to generate meaningful domains. To verify that these domains are in fact distinct, we used multiple quantification measures to estimate the presence and intensity of domain shifts. Additionally, we analyzed the performance on these domains with and without an unsupervised domain adaptation technique. We observed that in most of our grouped domains, domain shifts in fact exist. Based on our results, we believe these datasets to be helpful for testing the generalization capabilities of dermoscopic skin cancer classifiers.