Reviewerly/exHarmony huggingface dataset:
Image synthesis with disentangled attributes for chest X-ray nodule augmentation and detection

Lung nodule detection in chest X-ray (CXR) images is common to early screening of lung cancers. Deep-learning-based Computer-Assisted Diagnosis (CAD) systems can support radiologists for nodule screening in CXR images. However, it requires large-scale and diverse medical data with high-quality annotations to train such robust and accurate CADs. To alleviate the limited availability of such datasets, lung nodule synthesis methods are proposed for the sake of data augmentation. Nevertheless, previous methods lack the ability to generate nodules that are realistic with the shape/size attributes desired by the detector. To address this issue, we introduce a novel lung nodule synthesis framework in this paper, which decomposes nodule attributes into three main aspects including the shape, the size, and the texture, respectively. A GAN-based Shape Generator firstly models nodule shapes by generating diverse shape masks. The following Size Modulation then enables quantitative control on the diameters of the generated nodule shapes in pixel-level granularity. A coarse-to-fine gated convolutional Texture Generator finally synthesizes visually plausible nodule textures conditioned on the modulated shape masks. Moreover, we propose to synthesize nodule CXR images by controlling the disentangled nodule attributes for data augmentation, in order to better compensate for the nodules that are easily missed in the detection task. Our experiments demonstrate the enhanced image quality, diversity, and controllability of the proposed lung nodule synthesis framework. We also validate the effectiveness of our data augmentation strategy on greatly improving nodule detection performance.
Human-Avatar Interaction in Metaverse

The metaverse is a network of shared virtual environments where people can interact synchronously through their avatars. To enable this, it is necessary to accurately capture and recreate (physical) human motion. This is used to render avatars correctly, reflecting the motion of their corresponding users. In large-scale environments this must be done in real-time. This paper proposes a human-avatar framework with full-body motion capture. Its goal is to deliver high-accuracy capture with low computational and network overheads. It relies on a lightweight Octree data structure to record and transmit motion to other users. We conduct a user study with 22 participants and perform a preliminary evaluation of its scalability. Our user study shows that Octree with Inverse Kinematic achieves the best trade-off, achieving low delay and high accuracy. Our proposed solution delivers the lowest delay, with an average of 67ms in an environment of 8 concurrent users. It attains a 55.7% improvement over the prior techniques.
Understanding situation awareness in SOCs, a systematic literature review

Situation awareness is shown through human factors research to be a valuable construct to understand and improve how humans perform while operating complex systems in critical environments. Within cyber security one such environment is the Security Operations Center (SOC). With the increasing threat of hybrid warfare, knowledge about situation awareness within SOC environments, where human error or low performance may be detrimental, must be developed. This paper reports on the results of a Systematic Descriptive Literature Review of the current research on situation awareness within SOCs. The goal of the paper is to analyze how situation awareness is understood in the current research. To achieve this goal three aspects of understanding were addressed: Theoretical foundations; levels of conceptualization; and measurement of situation awareness. Theoretical foundations in the literature were assessed by how situation awareness was defined and the presence of references to theoretical models of SA. The results show a clear trend of basing the research on Endsley's three level situation awareness model; this model has been developed into a domain specific formulation called “Cyber Situation Awareness”. Some parts of the literature, particularly in research aimed at developing tools for improving situation awareness, lack a theoretical foundation; some refer to alternative theoretical foundations of situation awareness like Stanton et al.’s Distributed Situation Awareness. Further, a balance between conceptualizations on the individual, group and system level has been identified. Within research aimed at developing tools for improving situation awareness there are some examples of specialized and precise measurements of situation awareness, but in general the research seems too reliant on indirect measures of situation awareness. The paper concludes with the proposition of connecting the systems-based theoretical perspective of distributed situation awareness into the research, utilizing a systems level conceptualization of situation awareness. This might prove to be a useful bridge between the human cognitive perspective of situation awareness and the development of the complex technical environment of critical importance that SOCs represent.
Distributed Quantum Error Correction for Chip-Level Catastrophic Errors

Quantum error correction holds the key to scaling up quantum computers. Cosmic ray events severely impact the operation of a quantum computer by causing chip-level catastrophic errors, essentially erasing the information encoded in a chip. Here, we present a distributed error correction scheme to combat the devastating effect of such events by introducing an additional layer of quantum erasure error correcting code across separate chips. We show that our scheme is fault tolerant against chip-level catastrophic errors and discuss its experimental implementation using superconducting qubits with microwave links. Our analysis shows that in state-of-the-art experiments, it is possible to suppress the rate of these errors from 1 per 10 s to less than 1 per month.
Supervised GAN Watermarking for Intellectual Property Protection

We propose a watermarking method for protecting the Intellectual Property (IP) of Generative Adversarial Networks (GANs). The aim is to watermark the GAN model so that any image generated by the GAN contains an invisible watermark (signature), whose presence inside the image can be checked at a later stage for ownership verification. To achieve this goal, a pre-trained CNN watermarking decoding block is inserted at the output of the generator. The generator loss is then modified by including a watermark loss term, to ensure that the prescribed watermark can be extracted from the generated images. The watermark is embedded via fine-tuning, with reduced time complexity. Results show that our method can effectively embed an invisible watermark inside the generated images. Moreover, our method is a general one and can work with different GAN architectures, different tasks, and different resolutions of the output image. We also demonstrate the good robustness performance of the embedded watermark against several post-processing, among them, JPEG compression, noise addition, blurring, and color transformations.
AGILEST approach: Using machine learning agents to facilitate kinesthetic learning in STEM education through real-time touchless hand interaction

There is an increasing interest in creating interactive learning applications using innovative interaction technologies, especially in STEM (Science, technology, engineering, and mathematics) subjects. Recent developments in machine learning have allowed for nearly perfect hand-tracking recognition, introducing a touchless modality for interaction within Augmented Reality (AR) environments. However, the research community has not explored the pedagogical approach of Kinesthetic Learning or "Learning by Doing", hand tracking, and machine learning agents combined with Augmented Reality technology. Fundamentally, this exploration of touchless interaction technologies has taken on new importance in the new post-COVID world. Meanwhile, machine learning has gained attention for its ability to enhance personalized learning and play a vital new role as a virtual instructor. This paper proposes a novel approach called the AGILEST approach, which uses machine learning Agents to facilitate interactive kinesthetic learning in STEM education through touchless interaction. The first case study for this approach will be an AR learning application for chemistry. This application uses real-time touchless hand interaction for kinesthetic learning and uses a machine learning agent to act as both trainer and assessor of the user. The evaluation of this research has been conducted remotely through a usability study with expert reviewers, which includes 15 young researchers with peer-reviewed work in Human-Computer Interaction & AR and 2 subject experts STEM teachers at the secondary school level. The usability evaluation through NASA Task Load Index (NASA-TLX), Perceived Ease of Use (PUEU), and Perceived Usefulness (PU) with expert reviewers provide positive feedback about this approach for productive learning gain, engagement and interactiveness in learning STEM subjects.
HOMDA: High-Order Moment-Based Domain Alignment for unsupervised domain adaptation

Unsupervised domain adaptation aims to annotate unlabeled target domain samples by utilizing transferable knowledge learned from the source domain. Optimal transport (OT) has become a popular probability metric to measure the distribution discrepancy across domains. However, most OT-based methods inevitably suffer from false matching problem, resulting in poor domain alignment. Furthermore, because the domain shift cannot be completely eliminated, some target domain data distributed near cluster edges or far from their corresponding class centers are highly likely to be misclassified by the decision hyperplane learned from the source samples. We propose a High-Order Moment-Based Domain Alignment (HOMDA) method that uses an improved partial optimal transport (POT) strategy and a discriminative centroid-wise clustering regularization scheme to solve these issues. The improved POT strategy performs fine-grained domain alignment by limiting the amount of masses during mapping. The centroid-wise clustering scheme can promote the shared features with better intra-class compactness and inter-class separability, effectively reducing negative transfer caused by hard-aligned target samples. To better capture discriminative representations, HOMDA also includes a high-order statistics mixer. Extensive experiments show that our proposal can improve adaptation performance significantly and is competitive with state-of-the-art methods.
A novel ensemble model with two-stage learning for joint dialog act recognition and sentiment classification

Dialog act recognition (DAR) and sentiment classification (SC) are two correlative tasks to capture speakers’ intentions, where dialog act and sentiment can indicate the explicit and the implicit intentions separately. Current state-of-the-art methods usually leverage the graph neural networks or attention mechanism to capture the contextual dependency of an utterance, thereby aiding in efficient identification of the speakers’ intentions. Although existing attention mechanisms can automatically obtain the corresponding coefficient of each utterance based on the utterance-level embeddings, they often ignore the reliability of the utterance-level embeddings. This promotes us to propose an ensemble model with two-stage learning for joint DAR and SC. Firstly, we employ the BiLSTM as the encoder to capture the features of utterances, which are referred to “confidence vector” of the utterance-level embedding. Further, we introduce an edge-aware graph attention network to improve the classifier’s performance by using the confidence vector to selectively leverage the contextual information. Experimental results on two benchmark datasets show that our framework obtains state-of-the-art performances against all baselines, demonstrating the effectiveness of our method.
Joint depth map super-resolution method via deep hybrid-cross guidance filter

Nowadays color-guided Depth map Super-Resolution (DSR) methods mainly have three thorny problems: (1) joint DSR methods have serious detail and structure loss at very high sampling rate; (2) existing DSR networks have high computational complexity; (3) color-depth inconsistency makes it hard to fuse dual-modality features. To resolve these problems, we propose a joint hybrid-cross guidance filter method to progressively recover the quality of degraded Low-Resolution (LR) depth maps by exploiting color-depth consistency from multiple perspectives. Specifically, the proposed method leverages pyramid structure to extract multi-scale features from High-Resolution (HR) color image. At each scale, hybrid side window filter block is proposed to achieve high-efficiency color feature extraction after each down-sampling for HR color image. This block is also used to extract depth features from the LR depth map. Meanwhile, we propose a multi-perspective cross-guided fusion filter block to progressively fuse high-quality multi-scale structure information of color image with corresponding enhanced depth features. In this filter block, two kinds of space-aware group-compensation modules are introduced to capture various spatial features from different perspectives. Meanwhile, color-depth cross-attention module is proposed to extract color-depth consistency features for impactful boundary preservation. Comprehensively qualitative and quantitative experimental results have demonstrated that our method can achieve superior performances against a lot of state-of-the-art depth SR approaches in terms of mean absolute deviation and root mean square error on Middlebury, NYU-v2 and RGB-D-D datasets.
Improving dynamic gesture recognition in untrimmed videos by an online lightweight framework and a new gesture dataset ZJUGesture

Human–computer interaction technology brings great convenience to people, and dynamic gesture recognition makes it possible for a man to interact naturally with a machine. However, recognizing gestures quickly and precisely in untrimmed videos remains a challenge in real-world systems since: (1) It is challenging to locate the temporal boundaries of performing gestures; (2) There are significant differences in performing gestures among different people, resulting in a variety of gestures; (3) There must be a trade-off between the accuracy and the computational consumption. In this work, we propose an online lightweight two-stage framework, including a detection module and a gesture recognition module, to precisely detect and classify dynamic gestures in untrimmed videos. Specifically, we first design a low-power detection module to locate gestures in time series, then a temporal relational reasoning module is employed for gesture recognition. Moreover, we present a new dynamic gesture dataset named ZJUGesture, which contains nine classes of common gestures in various scenarios. Extensive experiments on the proposed ZJUGesture and 20-bn-Jester dataset demonstrate the attractive performance of our method with high accuracy and a low computational cost.
Effective Extended Reality: A Mixed-Reality Simulation Demonstration with Digitized and Holographic Tools and Intelligent Avatars

How can we design engaging and effective medical education both online and onsite? Extended reality (XR) is a term referring to all real-and-virtual combined environments and human-machine interactions generated by computer technology and wearables. It includes representative forms such as virtual reality (VR), augmented reality (AR), or mixed reality (MR), and the areas interpolated among them. MR is a domain of particular interest today. It takes place not only in the physical world or in the virtual world, but is a mix of the real and the virtual. Metaverses can be enabled by MR wearable augments. Glasses-free MR is another very interesting dimension: e-REAL®, as a MR environment for hybrid simulation and medical education in general, can be a stand-alone solution or even networked between multiple places through a link to a special videoconferencing system. Digital humans and human-sized holograms are part of the e-REAL scenarios, making this solution unique, rich, and diversified.
Re-examining lexical and semantic attention: Dual-view graph convolutions enhanced BERT for academic paper rating

Automatically assessing academic papers has enormous potential to reduce peer-review burden and individual bias. Existing studies strive for building sophisticated deep neural networks to identify academic value based on comprehensive data, e.g., academic graphs and full papers. However, these data are not always easy to access. And the content of the paper rather than other features outside the paper should matter in a fair assessment. Furthermore, while BERT models can maintain general semantics by pre-training on large-scale corpora, they tend to be over-smoothing due to stacked self-attention layers among unfiltered input tokens. Therefore, it is nontrivial to figure out distinguishable value of an academic paper from its limited content. In this study, we propose a novel deep neural network, namely Dual-view Graph Convolutions Enhanced BERT (DGC-BERT), for academic paper acceptance estimation. We combine the title and abstract of the paper as input. Then, a pre-trained BERT model is employed to extract the paper's general representations. Apart from hidden representations of the final layer, we highlight the first and last few layers as lexical and semantic views. In particular, we re-examine the dual-view filtered self-attention matrices via constructing two graphs, respectively. After that, two multi-hop Graph Convolutional Networks (GCNs) are separately employed to capture pivotal and distant dependencies between the tokens. Moreover, the dual-view representations are facilitated by each other with biaffine attention modules. And a re-weighting gate is proposed to further streamline the dual-view representations with the help of the original BERT representation. Finally, whether the submitted paper could be acceptable is predicted based on the original language model features cooperated with the dual-view dependencies. Extensive data analyses and the full paper based MHCNN studies provide insights into the task and structural functions. Comparison experiments on two benchmark datasets demonstrate that the proposed DGC-BERT significantly outperforms alternative approaches, especially the state-of-the-art models like MHCNN and BERT variants. Additional analyses reveal significance and explainability of the proposed modules in the DGC-BERT. Our codes and settings have been released on Github (https://github.com/ECNU-Text-Computing/DGC-BERT).
Learning-Based Bending Stiffness Parameter Estimation by a Drape Tester

Real-world fabrics often possess complicated nonlinear, anisotropic bending stiffness properties. Measuring the physical parameters of such properties for physics-based simulation is difficult yet unnecessary, due to the persistent existence of numerical errors in simulation technology. In this work, we propose to adopt a simulation-in-the-loop strategy: instead of measuring the physical parameters, we estimate the simulation parameters to minimize the discrepancy between reality and simulation. This strategy offers good flexibility in test setups, but the associated optimization problem is computationally expensive to solve by numerical methods. Our solution is to train a regression-based neural network for inferring bending stiffness parameters, directly from drape features captured in the real world. Specifically, we choose the Cusick drape test method and treat multiple-view depth images as the feature vector. To effectively and efficiently train our network, we develop a highly expressive and physically validated bending stiffness model, and we use the traditional cantilever test to collect the parameters of this model for 618 real-world fabrics. Given the whole parameter data set, we then construct a parameter subspace, generate new samples within the sub-space, and finally simulate and augment synthetic data for training purposes. The experiment shows that our trained system can replace cantilever tests for quick, reliable and effective estimation of simulation-ready parameters. Thanks to the use of the system, our simulator can now faithfully simulate bending effects comparable to those in the real world.
Large-scale application of named entity recognition to biomedicine and epidemiology

Despite significant advancements in biomedical named entity recognition methods, the clinical application of these systems continues to face many challenges: (1) most of the methods are trained on a limited set of clinical entities; (2) these methods are heavily reliant on a large amount of data for both pre-training and prediction, making their use in production impractical; (3) they do not consider non-clinical entities, which are also related to patient's health, such as social, economic or demographic factors.In this paper, we develop Bio-Epidemiology-NER (https://pypi.org/project/Bio-Epidemiology-NER/) an open-source Python package for detecting biomedical named entities from the text. This approach is based on a Transformer-based system and trained on a dataset that is annotated with many named entities (medical, clinical, biomedical, and epidemiological). This approach improves on previous efforts in three ways: (1) it recognizes many clinical entity types, such as medical risk factors, vital signs, drugs, and biological functions; (2) it is easily configurable, reusable, and can scale up for training and inference; (3) it also considers non-clinical factors (age and gender, race and social history and so) that influence health outcomes. At a high level, it consists of the phases: pre-processing, data parsing, named entity recognition, and named entity enhancement.Experimental results show that our pipeline outperforms other methods on three benchmark datasets with macro-and micro average F1 scores around 90 percent and above.This package is made publicly available for researchers, doctors, clinicians, and anyone to extract biomedical named entities from unstructured biomedical texts.
Time–frequency recurrent transformer with diversity constraint for dense video captioning

Describing a long video using multiple sentences, i.e., dense video captioning, is a very challenging task. Existing methods neglect the important fact that the actions of several tempos (a.k.a., frequencies) evolve with the time in video, and do not well handle the phrase repetition issue. Therefore, we propose a Time-Frequency recurrent Transformer with Diversity constraint (TFTD) for dense video captioning. Its basic idea is to develop a time–frequency memory module, which not only stores the history of the past sentences and corresponding video segments to consider the temporal relations, but also models the motion dependency of action patterns with different frequencies. This contributes to producing more coherent sentences that well describe the video content. Moreover, we adopt the Determinantal Point Processes (DDP) for designing the diversity loss imposed to the objective function as constraint, such that the generated sentences are diverse with less redundancy. Extensive experiments on two benchmark datasets have verified the superior performance of our approach, e.g., it achieves 11.36, 16.56, 26.16, and 3.77, in terms of BLEU@4, METEOR, CIDEr-D, and R@4, respectively, on ActivityNet Captions (20,000 videos). Besides, TFTD outperforms the most competitive alternative by 8.0% on ActivityNet Captions and 9.8% on YouCookII (2,000 videos) in terms of coherence for human evaluation.