Miracle-dz/UniFAR-FTUs huggingface dataset:
EvHandPose: Event-based 3D Hand Pose Estimation with Sparse Supervision

Event camera shows great potential in 3D hand pose estimation, especially addressing the challenges of fast motion and high dynamic range in a low-power way. However, due to the asynchronous differential imaging mechanism, it is challenging to design event representation to encode hand motion information especially when the hands are not moving (causing motion ambiguity), and it is infeasible to fully annotate the temporally dense event stream. In this paper, we propose EvHandPose with novel hand flow representations in Event-to-Pose module for accurate hand pose estimation and alleviating the motion ambiguity issue. To solve the problem under sparse annotation, we design contrast maximization and hand-edge constraints in Pose-to-IWE (Image with Warped Events) module and formulate EvHandPose in a weakly-supervision framework. We further build EvRealHands, the first large-scale real-world event-based hand pose dataset on several challenging scenes to bridge the real-synthetic domain gap. Experiments on EvRealHands demonstrate that EvHandPose outperforms previous event-based methods under all evaluation scenes, achieves accurate and stable hand pose estimation with high temporal resolution in fast motion and strong light scenes compared with RGB-based methods, generalizes well to outdoor scenes and another type of event camera, and shows the potential for the hand gesture recognition task.
The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges

Recent advancements in visiolinguistic (VL) learning have allowed the development of multiple models and techniques that offer several impressive implementations, able to currently resolve a variety of tasks that require the collaboration of vision and language. Current datasets used for VL pre-training only contain a limited amount of visual and linguistic knowledge, thus significantly limiting the generalization capabilities of many VL models. External knowledge sources such as knowledge graphs (KGs) and Large Language Models (LLMs) are able to cover such generalization gaps by filling in missing knowledge, resulting in the emergence of hybrid architectures. In the current survey, we analyze tasks that have benefited from such hybrid approaches. Moreover, we categorize existing knowledge sources and types, proceeding to discussion regarding the KG vs LLM dilemma and its potential impact to future hybrid approaches.
X-RAY AND GAMMA-RAY EMISSIONS FROM ROTATION POWERED MILLISECOND PULSARS

The Fermi Large Area Telescope has revealed that rotation powered millisecond pulsars (MSPs) are a major contributor to the Galactic γ-ray source population. Such pulsars may also be important in modeling the quiescent state of several low-mass X-ray binaries (LMXBs), where optical observations of the companion star suggest the possible existence of rotation powered MSPs. To understand the observational properties of the different evolutionary stages of MSPs, the X-ray and γ-ray emissions associated with the outer gap model are investigated. For rotation powered MSPs, the size of the outer gap and the properties of the high-energy emission are controlled by either the photon–photon pair-creation process or magnetic pair-creation process near the surface. For these pulsars, we find that the outer gap model controlled by the magnetic pair-creation process is preferable in explaining the possible correlations between the γ-ray luminosity or non-thermal X-ray luminosity versus the spin-down power. For the accreting MSPs in quiescent LMXBs, the thermal X-ray emission at the neutron star (NS) surface resulting from deep crustal heating can control the conditions in the outer gap. We argue that the optical modulation observed in the quiescent state of several LMXBs originates from the irradiation of the donor star by γ-rays from the outer gap. In these systems, the irradiation luminosity required for the optical modulation of the source such as SAX J1808.4−3658 can be achieved for a NS of high mass. Finally, we discuss the high-energy emission associated with an intra-binary shock in black widow systems, e.g., PSR B1957+20.
Open-Vocabulary Affordance Detection in 3D Point Clouds

Affordance detection is a challenging problem with a wide variety of robotic applications. Traditional affordance detection methods are limited to a predefined set of affordance labels, hence potentially restricting the adaptability of intelligent robots in complex and dynamic environments. In this paper, we present the Open-Vocabulary Affordance Detection (OpenAD) method, which is capable of detecting an unbounded number of affordances in 3D point clouds. By simultaneously learning the affordance text and the point feature, OpenAD successfully exploits the semantic relationships between affordances. Therefore, our proposed method enables zero-shot detection and can be able to detect previously unseen affordances without a single annotation example. Intensive experimental results show that OpenAD works effectively on a wide range of affordance detection setups and outperforms other baselines by a large margin. Additionally, we demonstrate the practicality of the proposed OpenAD in real-world robotic applications with a fast inference speed. Our project is available at https://openad2023.github. io.
Annealing time, temperature, and field dependence of pinned magnetic moments in the collinear antiferromagnet PdMn

Magnetic annealing of the collinear antiferromagnet PdMn with excess Pd produces strongly pinned magnetic moments in the annealing field direction. This behaviour can be understood with the help of the magnetic-field-biased diffusion model. Here, the magnetic field creates an energy difference between the two possible occupations of the antiferromagnetic Mn-sublattices by the Pd-excess atoms. This, mediated by diffusion, leads to an imbalance in the amount of the Pd-excess atoms in these sublattices and, subsequently, to an imbalance in the total magnetization of the sublattices. We investigate this effect's dependence on the annealing field, time, and temperature. The results are then compared to the results of the magnetic-field-biased diffusion model, which gives good agreement.
CBCT-Based synthetic CT image generation using conditional denoising diffusion probabilistic model.

BACKGROUND Daily or weekly cone-beam computed tomography (CBCT) scans are commonly used for accurate patient positioning during the image-guided radiotherapy (IGRT) process, making it an ideal option for adaptive radiotherapy (ART) replanning. However, the presence of severe artifacts and inaccurate Hounsfield unit (HU) values prevent its use for quantitative applications such as organ segmentation and dose calculation. To enable the clinical practice of online ART, it is crucial to obtain CBCT scans with a quality comparable to that of a CT scan. PURPOSE This work aims to develop a conditional diffusion model to perform image translation from the CBCT to the CT distribution for the image quality improvement of CBCT. METHODS The proposed method is a conditional denoising diffusion probabilistic model (DDPM) that utilizes a time-embedded U-net architecture with residual and attention blocks to gradually transform the white Gaussian noise sample to the target CT distribution conditioned on the CBCT. The model was trained on deformed planning CT (dpCT) and CBCT image pairs, and its feasibility was verified in brain patient study and head-and-neck (H&N) patient study. The performance of the proposed algorithm was evaluated using mean absolute error (MAE), peak signal-to-noise ratio (PSNR) and normalized cross-correlation (NCC) metrics on generated synthetic CT (sCT) samples. The proposed method was also compared to four other diffusion model-based sCT generation methods. RESULTS In the brain patient study, the MAE, PSNR, and NCC of the generated sCT were 25.99 HU, 30.49 dB, and 0.99, respectively, compared to 40.63 HU, 27.87 dB, and 0.98 of the CBCT images. In the H&N patient study, the metrics were 32.56 HU, 27.65 dB, 0.98 and 38.99 HU, 27.00, 0.98 for sCT and CBCT, respectively. Compared to the other four diffusion models and one Cycle generative adversarial network (Cycle GAN), the proposed method showed superior results in both visual quality and quantitative analysis. CONCLUSIONS The proposed conditional DDPM method can generate sCT from CBCT with accurate HU numbers and reduced artifacts, enabling accurate CBCT-based organ segmentation and dose calculation for online ART.
Temperature dependence of QCSE device characteristics and performance

The normal method of operation for a GaAs MQW quantum confined Stark effect (QCSE) reflection modulator is given for a device operating wavelength, taken as 852 nm for the analysis in this paper. Modulation in the intensity of a reference optical beam is achieved by shifting the wavelength to achieve both low reflectivity and high reflectivity. The figure of merit for a QCSE modulator is its change in reflectivity from the low state to the high state is given.
A NOTE ON FUNDAMENTAL SOLUTIONS AND GAUSSIAN BOUNDS FOR DEGENERATE PARABOLIC EQUATIONS WITH TIME-DEPENDENT COEFFICIENTS

We consider second order degenerate parabolic equations with real, measurable, and time-dependent coeﬃcients. We allow for degenerate ellipticity dictated by a spatial A 2 -weight. We prove the existence of a fundamental solution and derive Gaussian bounds. Our construction is based on the original work of Kato [15].
Constrained Reinforcement Learning and Formal Verification for Safe Colonoscopy Navigation

The field of robotic Flexible Endoscopes (FEs) has progressed significantly, offering a promising solution to reduce patient discomfort. However, the limited autonomy of most robotic FEs results in non-intuitive and challenging manoeuvres, constraining their application in clinical settings. While previous studies have employed lumen tracking for autonomous navigation, they fail to adapt to the presence of obstructions and sharp turns when the endoscope faces the colon wall. In this work, we propose a Deep Reinforcement Learning (DRL)-based navigation strategy that eliminates the need for lumen tracking. However, the use of DRL methods poses safety risks as they do not account for potential hazards associated with the actions taken. To ensure safety, we exploit a Constrained Reinforcement Learning (CRL) method to restrict the policy in a predefined safety regime. Moreover, we present a model selection strategy that utilises Formal Verification (FV) to choose a policy that is entirely safe before deployment. We validate our approach in a virtual colonoscopy environment and report that out of the 300 trained policies, we could identify three policies that are entirely safe. Our work demonstrates that CRL, combined with model selection through FV, can improve the robustness and safety of robotic behaviour in surgical applications.
Referring Multi-Object Tracking

Existing referring understanding tasks tend to involve the detection of a single text-referred object. In this paper, we propose a new and general referring understanding task, termed referring multi-object tracking (RMOT). Its core idea is to employ a language expression as a semantic cue to guide the prediction of multi-object tracking. To the best of our knowledge, it is the first work to achieve an arbitrary number of referent object predictions in videos. To push forward RMOT, we construct one benchmark with scalable expressions based on KITTI, named Refer-KITTI. Specifically, it provides 18 videos with 818 expressions, and each expression in a video is annotated with an average of 10.7 objects. Further, we develop a transformer-based architecture TransRMOT to tackle the new task in an online manner, which achieves impressive detection performance and out-performs other counterparts. The Refer-KITTI dataset and the code are released at https://referringmot.github. io.
Coupled Multiwavelet Neural Operator Learning for Coupled Partial Differential Equations

Coupled partial differential equations (PDEs) are key tasks in modeling the complex dynamics of many physical processes. Recently, neural operators have shown the ability to solve PDEs by learning the integral kernel directly in Fourier/Wavelet space, so the difficulty for solving the coupled PDEs depends on dealing with the coupled mappings between the functions. Towards this end, we propose a \textit{coupled multiwavelets neural operator} (CMWNO) learning scheme by decoupling the coupled integral kernels during the multiwavelet decomposition and reconstruction procedures in the Wavelet space. The proposed model achieves significantly higher accuracy compared to previous learning-based solvers in solving the coupled PDEs including Gray-Scott (GS) equations and the non-local mean field game (MFG) problem. According to our experimental results, the proposed model exhibits a $2\times \sim 4\times$ improvement relative $L$2 error compared to the best results from the state-of-the-art models.
Demystifying What Code Summarization Models Learned

Study patterns that models have learned has long been a focus of pattern recognition research. Explaining what patterns are discovered from training data, and how patterns are generalized to unseen data are instrumental to understanding and advancing the pattern recognition methods. Unfortunately, the vast majority of the application domains deal with continuous data (i.e. statistical in nature) out of which extracted patterns can not be formally defined. For example, in image classification, there does not exist a principle definition for a label of cat or dog. Even in natural language, the meaning of a word can vary with the context it is surrounded by. Unlike the aforementioned data format, programs are a unique data structure with a well-defined syntax and semantics, which creates a golden opportunity to formalize what models have learned from source code. This paper presents the first formal definition of patterns discovered by code summarization models (i.e. models that predict the name of a method given its body), and gives a sound algorithm to infer a context-free grammar (CFG) that formally describes the learned patterns. We realize our approach in PATIC which produces CFGs for summarizing the patterns discovered by code summarization models. In particular, we pick two prominent instances, code2vec and code2seq, to evaluate PATIC. PATIC shows that the patterns extracted by each model are heavily restricted to local, and syntactic code structures with little to none semantic implication. Based on these findings, we present two example uses of the formal definition of patterns: a new method for evaluating the robustness and a new technique for improving the accuracy of code summarization models. Our work opens up this exciting, new direction of studying what models have learned from source code.
Chasing Fairness Under Distribution Shift: A Model Weight Perturbation Approach

Fairness in machine learning has attracted increasing attention in recent years. The fairness methods improving algorithmic fairness for in-distribution data may not perform well under distribution shifts. In this paper, we first theoretically demonstrate the inherent connection between distribution shift, data perturbation, and model weight perturbation. Subsequently, we analyze the sufficient conditions to guarantee fairness (i.e., low demographic parity) for the target dataset, including fairness for the source dataset, and low prediction difference between the source and target datasets for each sensitive attribute group. Motivated by these sufficient conditions, we propose robust fairness regularization (RFR) by considering the worst case within the model weight perturbation ball for each sensitive attribute group. We evaluate the effectiveness of our proposed RFR algorithm on synthetic and real distribution shifts across various datasets. Experimental results demonstrate that RFR achieves better fairness-accuracy trade-off performance compared with several baselines. The source code is available at \url{https://github.com/zhimengj0326/RFR_NeurIPS23}.
Ternary oxides of $\textit{s}$- and $\textit{p}$-block metals for photocatalytic solar-to-hydrogen conversion

Oxides containing metals or metalloids from the {\it p}-block of the periodic table ({\it e.g.}, In, Sn, Sb, Pb, Bi) are of technological interest as transparent conductors and light absorbers for solar energy conversion due to the tunability of their electronic conductivity and optical absorption. Comparatively, these oxides have found limited applications in hydrogen photoelectrolysis primarily due to their high electronegativity, which impedes electron transfer for reducing protons into hydrogen. We have shown recently that inserting {\it s}-block cations into {\it p}-block metal oxides is effective at lowering electronegativities while affording further control of band gaps. Here, we explain the origins of this dual tunability by demonstrating the mediator role of {\it s}-block cations in modulating orbital hybridization while not contributing to frontier electronic states. From this result, we carry out a comprehensive computational study of 109 ternary oxides of {\it s}- and {\it p}-block metal elements as candidate photocatalysts for solar hydrogen generation. We downselect the most desirable materials using band gaps and band edges obtained from Hubbard-corrected density-functional theory with Hubbard parameters computed entirely from first principles, evaluate the stability of these oxides in aqueous conditions, and characterize experimentally four of the remaining materials, synthesized with high phase uniformity, to validate and further develop the computational models. We thus propose nine oxide semiconductors, including CsIn$_3$O$_5$, Sr$_2$In$_2$O$_5$, and KSbO$_2$ which, to the extent of our literature review, have not been previously considered as water-splitting photocatalysts.
Federated Virtual Learning on Heterogeneous Data with Local-global Distillation

Despite Federated Learning (FL)'s trend for learning machine learning models in a distributed manner, it is susceptible to performance drops when training on heterogeneous data. In addition, FL inevitability faces the challenges of synchronization, efficiency, and privacy. Recently, dataset distillation has been explored in order to improve the efficiency and scalability of FL by creating a smaller, synthetic dataset that retains the performance of a model trained on the local private datasets. We discover that using distilled local datasets can amplify the heterogeneity issue in FL. To address this, we propose a new method, called Federated Virtual Learning on Heterogeneous Data with Local-Global Distillation (FedLGD), which trains FL using a smaller synthetic dataset (referred as virtual data) created through a combination of local and global dataset distillation. Specifically, to handle synchronization and class imbalance, we propose iterative distribution matching to allow clients to have the same amount of balanced local virtual data; to harmonize the domain shifts, we use federated gradient matching to distill global virtual data that are shared with clients without hindering data privacy to rectify heterogeneous local training via enforcing local-global feature similarity. We experiment on both benchmark and real-world datasets that contain heterogeneous data from different sources, and further scale up to an FL scenario that contains large number of clients with heterogeneous and class imbalance data. Our method outperforms state-of-the-art heterogeneous FL algorithms under various settings with a very limited amount of distilled virtual data.
Scaling behaviour of the helical and skyrmion phases of Cu2OSeO3 determined by single crystal small angle neutron scattering

Skyrmions are topologically protected quantum objects at the nanometre scale. They form perpendicular to an applied magnetic field at a certain temperature and arrange themselves in a typically hexagonal lattice. Using small angle neutron scattering we have determined the magnetic field versus temperature phase diagrams of the stability range of the different magnetic phases, the helical as well as the skyrmion phases in the multiferroic skyrmion material Cu$_2$OSeO$_3$. Therefore, a single crystal was mounted in different crystal orientations, i.e. unoriented and with the crystallographic $\langle1 1 0\rangle$ and $\langle1 0 0\rangle$ axis aligned parallel to the magnetic field. Furthermore, different cooling procedures were tested, cooling from the paramagnetic phase at zero magnetic field and field cooling through the skyrmion phase where metastable skyrmions are nucleated. From this, not only were the stability ranges of both the helical and skyrmion phases in this multiferroic skyrmion material determined, but the length of the spin helix and the skyrmion distances at different conditions was also determined through detailed analysis of the positions of the observed Bragg peaks. The obtained data provide valuable information about the scaling of the skyrmion distances and therefore their packing density. The knowledge about this tunability will serve as an important input for the theoretical understanding of the formation of skyrmions. Concerning technological applications, for example as high-density data storage devices in information technology, the temperature and magnetic field dependence of the packing density of skyrmions is of critical importance for their design. Therefore, the obtained knowledge is an essential input for future skyrmionics applications in high-density data storage, in low-energy spintronics or in skyrmionics quantum computation.
Dynamic Prompting: A Unified Framework for Prompt Tuning

It has been demonstrated that the art of prompt tuning is highly effective in efficiently extracting knowledge from pretrained foundation models, encompassing pretrained language models (PLMs), vision pretrained models, and vision-language (V-L) models. However, the efficacy of employing fixed soft prompts with a predetermined position for concatenation with inputs for all instances, irrespective of their inherent disparities, remains uncertain. Variables such as the position, length, and representations of prompts across diverse instances and tasks can substantially influence the performance of prompt tuning. In this context, we provide a theoretical analysis, which reveals that optimizing the position of the prompt to encompass the input can capture additional semantic information that traditional prefix or postfix prompt tuning methods fail to capture. Building upon our analysis, we present a unified dynamic prompt (DP) tuning strategy that dynamically determines different factors of prompts based on specific tasks and instances. To accomplish this, we employ a lightweight learning network with Gumble-Softmax, allowing us to learn instance-dependent guidance. Experimental results underscore the significant performance improvement achieved by dynamic prompt tuning across a wide range of tasks, including NLP tasks, vision recognition tasks, and vision-language tasks. Furthermore, we establish the universal applicability of our approach under full-data, few-shot, and multitask scenarios. Codes are available at https://github.com/Xianjun-Yang/DPT.