Reviewerly/exHarmony huggingface dataset:
On the Perception of Small Sub-graphs

Interpreting a node-link graph is enhanced if similar subgraphs (or motifs) are depicted in a similar manner; that is, they have the same visual form. Small motifs within graphs may be perceived to be identical when they are structurally dissimilar, or may be perceived to be dissimilar when they are identical. This issue primarily relates to the Gestalt principle of similarity, but may also include an element of quick, low-level pattern-matching. We believe that if motifs are identical, they should be depicted identically; if they are nearly-identical, they should be depicted nearly-identically. This principle is particularly important in domains where motifs hold meaning and where their identification is important. We identified five small motifs: bi-cliques, cliques, cycles, double-cycles, and stars. For each, we defined visual variations on two dimensions: same or different structure, same or different shape. We conducted a crowd-sourced empirical study to test the perception of similarity of these varied motifs, and found that determining whether motifs are identical or similar is affected by both shape and structure.
A new approach for evaluating internal cluster validation indices

A vast number of different methods are available for unsupervised classification. Since no algorithm and parameter setting performs best in all types of data, there is a need for cluster validation to select the actually best-performing algorithm. Several indices were proposed for this purpose without using any additional (external) information. These internal validation indices can be evaluated by applying them to classifications of datasets with a known cluster structure. Evaluation approaches differ in how they use the information on the ground-truth classification. This paper reviews these approaches, considering their advantages and disadvantages, and then suggests a new approach.
Stabilizing two-qubit entanglement with dynamically decoupled active feedback

We propose and analyze a protocol for stabilizing a maximally entangled state of two noninteracting qubits using active state-dependent feedback from a continuous two-qubit half-parity measurement in coordination with a concurrent, non-commuting dynamical decoupling drive. We demonstrate that such a drive can be simultaneous with the measurement and feedback, while also playing a key part in the feedback protocol itself. We show that robust stabilization with near-unit fidelity can be achieved even in the presence of realistic nonidealities, such as time delay in the feedback loop, imperfect state-tracking, inefficient measurements, dephasing from $1/f$-distributed qubit-frequency noise, and relaxation. We mitigate feedback-delay error by introducing a forward-state-estimation strategy in the feedback controller that tracks the effects of control signals already in transit. More generally, the steady state is globally attractive without the need for ancillas, regardless of the error state, in contrast to most known feedback and error correction schemes.
ForensiBlock: A Provenance-Driven Blockchain Framework for Data Forensics and Auditability

Maintaining accurate provenance records is paramount in digital forensics, as they underpin evidence credibility and integrity, addressing essential aspects like accountability and reproducibility. Blockchains have several properties that can address these requirements. Previous systems utilized public blockchains, i.e., treated blockchain as a black box, and benefiting from the immutability property. However, the blockchain was accessible to everyone, giving rise to security concerns and moreover, efficient extraction of provenance faces challenges due to the enormous scale and complexity of digital data. This necessitates a tailored blockchain design for digital forensics. Our solution, Forensiblock has a novel design that automates investigation steps, ensures secure data access, traces data origins, preserves records, and expedites provenance extraction. Forensiblock incorporates Role-Based Access Control with Staged Authorization (RBAC-SA) and a distributed Merkle root for case tracking. These features support authorized resource access with an efficient retrieval of provenance records. Particularly, comparing two methods for extracting provenance records off chain storage retrieval with Merkle root verification and a brute-force search the offchain method is significantly better, especially as the blockchain size and number of cases increase. We also found that our distributed Merkle root creation slightly increases smart contract processing time but significantly improves history access. Overall, we show that Forensiblock offers secure, efficient, and reliable handling of digital forensic data
The Prospect of Enhancing Large-Scale Heterogeneous Federated Learning with Transformers

Federated learning (FL) addresses data privacy concerns by enabling collaborative training of AI models across distributed data owners. Wide adoption of FL faces the fundamental challenges of data heterogeneity and the large scale of data owners involved. In this paper, we investigate the prospect of Transformer-based FL models for achieving generalization and personalization in this setting. We conduct extensive comparative experiments involving FL with Transformers, ResNet, and personalized ResNet-based FL approaches under various scenarios. These experiments consider varying numbers of data owners to demonstrate Transformers' advantages over deep neural networks in large-scale heterogeneous FL tasks. In addition, we analyze the superior performance of Transformers by comparing the Centered Kernel Alignment (CKA) representation similarity across different layers and FL models to gain insight into the reasons behind their promising capabilities.
Dead or Alive: Continuous Data Profiling for Interactive Data Science

Profiling data by plotting distributions and analyzing summary statistics is a critical step throughout data analysis. Currently, this process is manual and tedious since analysts must write extra code to examine their data after every transformation. This inefficiency may lead to data scientists profiling their data infrequently, rather than after each transformation, making it easy for them to miss important errors or insights. We propose continuous data profiling as a process that allows analysts to immediately see interactive visual summaries of their data throughout their data analysis to facilitate fast and thorough analysis. Our system, AutoProfiler, presents three ways to support continuous data profiling: it automatically displays data distributions and summary statistics to facilitate data comprehension; it is live, so visualizations are always accessible and update automatically as the data updates; it supports follow up analysis and documentation by authoring code for the user in the notebook. In a user study with 16 participants, we evaluate two versions of our system that integrate different levels of automation: both automatically show data profiles and facilitate code authoring, however, one version updates reactively and the other updates only on demand. We find that both tools facilitate insight discovery with 91% of user-generated insights originating from the tools rather than manual profiling code written by users. Participants found live updates intuitive and felt it helped them verify their transformations while those with on-demand profiles liked the ability to look at past visualizations. We also present a longitudinal case study on how AutoProfiler helped domain scientists find serendipitous insights about their data through automatic, live data profiles. Our results have implications for the design of future tools that offer automated data analysis support.
Prompted Contrast with Masked Motion Modeling: Towards Versatile 3D Action Representation Learning

Self-supervised learning has proved effective for skeleton-based human action understanding, which is an important yet challenging topic. Previous works mainly rely on contrastive learning or masked motion modeling paradigm to model the skeleton relations. However, the sequence-level and joint-level representation learning cannot be effectively and simultaneously handled by these methods. As a result, the learned representations fail to generalize to different downstream tasks. Moreover, combining these two paradigms in a naive manner leaves the synergy between them untapped and can lead to interference in training. To address these problems, we propose Prompted Contrast with Masked Motion Modeling, PCM$^{\rm 3}$, for versatile 3D action representation learning. Our method integrates the contrastive learning and masked prediction tasks in a mutually beneficial manner, which substantially boosts the generalization capacity for various downstream tasks. Specifically, masked prediction provides novel training views for contrastive learning, which in turn guides the masked prediction training with high-level semantic information. Moreover, we propose a dual-prompted multi-task pretraining strategy, which further improves model representations by reducing the interference caused by learning the two different pretext tasks. Extensive experiments on five downstream tasks under three large-scale datasets are conducted, demonstrating the superior generalization capacity of PCM$^{\rm 3}$ compared to the state-of-the-art works. Our project is publicly available at: https://jhang2020.github.io/Projects/PCM3/PCM3.html .
PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning

Synthetic image datasets offer unmatched advantages for designing and evaluating deep neural networks: they make it possible to (i) render as many data samples as needed, (ii) precisely control each scene and yield granular ground truth labels (and captions), (iii) precisely control distribution shifts between training and testing to isolate variables of interest for sound experimentation. Despite such promise, the use of synthetic image data is still limited -- and often played down -- mainly due to their lack of realism. Most works therefore rely on datasets of real images, which have often been scraped from public images on the internet, and may have issues with regards to privacy, bias, and copyright, while offering little control over how objects precisely appear. In this work, we present a path to democratize the use of photorealistic synthetic data: we develop a new generation of interactive environments for representation learning research, that offer both controllability and realism. We use the Unreal Engine, a powerful game engine well known in the entertainment industry, to produce PUG (Photorealistic Unreal Graphics) environments and datasets for representation learning. In this paper, we demonstrate the potential of PUG to enable more rigorous evaluations of vision models.
PAIF: Perception-Aware Infrared-Visible Image Fusion for Attack-Tolerant Semantic Segmentation

Infrared and visible image fusion is a powerful technique that combines complementary information from different modalities for downstream semantic perception tasks. Existing learning-based methods show remarkable performance, but are suffering from the inherent vulnerability of adversarial attacks, causing a significant decrease in accuracy. In this work, a perception-aware fusion framework is proposed to promote segmentation robustness in adversarial scenes. We first conduct systematic analyses about the components of image fusion, investigating the correlation with segmentation robustness under adversarial perturbations. Based on these analyses, we propose a harmonized architecture search with a decomposition-based structure to balance standard accuracy and robustness. We also propose an adaptive learning strategy to improve the parameter robustness of image fusion, which can learn effective feature extraction under diverse adversarial perturbations. Thus, the goals of image fusion (\textit{i.e.,} extracting complementary features from source modalities and defending attack) can be realized from the perspectives of architectural and learning strategies. Extensive experimental results demonstrate that our scheme substantially enhances the robustness, with gains of 15.3% mIOU of segmentation in the adversarial scene, compared with advanced competitors. The source codes are available at https://github.com/LiuZhu-CV/PAIF.
Scope Loss for Imbalanced Classification and RL Exploration

We demonstrate equivalence between the reinforcement learning problem and the supervised classification problem. We consequently equate the exploration exploitation trade-off in reinforcement learning to the dataset imbalance problem in supervised classification, and find similarities in how they are addressed. From our analysis of the aforementioned problems we derive a novel loss function for reinforcement learning and supervised classification. Scope Loss, our new loss function, adjusts gradients to prevent performance losses from over-exploitation and dataset imbalances, without the need for any tuning. We test Scope Loss against SOTA loss functions over a basket of benchmark reinforcement learning tasks and a skewed classification dataset, and show that Scope Loss outperforms other loss functions.
SimplyRetrieve: A Private and Lightweight Retrieval-Centric Generative AI Tool

Large Language Model (LLM) based Generative AI systems have seen significant progress in recent years. Integrating a knowledge retrieval architecture allows for seamless integration of private data into publicly available Generative AI systems using pre-trained LLM without requiring additional model fine-tuning. Moreover, Retrieval-Centric Generation (RCG) approach, a promising future research direction that explicitly separates roles of LLMs and retrievers in context interpretation and knowledge memorization, potentially leads to more efficient implementation. SimplyRetrieve is an open-source tool with the goal of providing a localized, lightweight, and user-friendly interface to these sophisticated advancements to the machine learning community. SimplyRetrieve features a GUI and API based RCG platform, assisted by a Private Knowledge Base Constructor and a Retrieval Tuning Module. By leveraging these capabilities, users can explore the potential of RCG for improving generative AI performance while maintaining privacy standards. The tool is available at https://github.com/RCGAI/SimplyRetrieve with an MIT license.
Generalization bound for estimating causal effects from observational network data

Estimating causal effects from observational network data is a significant but challenging problem. Existing works in causal inference for observational network data lack an analysis of the generalization bound, which can theoretically provide support for alleviating the complex confounding bias and practically guide the design of learning objectives in a principled manner. To fill this gap, we derive a generalization bound for causal effect estimation in network scenarios by exploiting 1) the reweighting schema based on joint propensity score and 2) the representation learning schema based on Integral Probability Metric (IPM). We provide two perspectives on the generalization bound in terms of reweighting and representation learning, respectively. Motivated by the analysis of the bound, we propose a weighting regression method based on the joint propensity score augmented with representation learning. Extensive experimental studies on two real-world networks with semi-synthetic data demonstrate the effectiveness of our algorithm.
Quantum Resources in Harrow-Hassidim-Lloyd Algorithm

Quantum algorithms have the ability to reduce runtime for executing tasks beyond the capabilities of classical algorithms. Therefore, identifying the resources responsible for quantum advantages is an interesting endeavour. We prove that nonvanishing quantum correlations, both bipartite and genuine multipartite entanglement, are required for solving nontrivial linear systems of equations in the Harrow-Hassidim-Lloyd (HHL) algorithm. Moreover, we find a nonvanishing l1-norm quantum coherence of the entire system and the register qubit which turns out to be related to the success probability of the algorithm. Quantitative analysis of the quantum resources reveals that while a significant amount of bipartite entanglement is generated in each step and required for this algorithm, multipartite entanglement content is inversely proportional to the performance indicator. In addition, we report that when imperfections chosen from Gaussian distribution are incorporated in controlled rotations, multipartite entanglement increases with the strength of the disorder, albeit error also increases while bipartite entanglement and coherence decreases, confirming the beneficial role of bipartite entanglement and coherence in this algorithm.
InfeRE: Step-by-Step Regex Generation via Chain of Inference

Automatically generating regular expressions (abbrev. regexes) from natural language description (NL2RE) has been an emerging research area. Prior studies treat regex as a linear sequence of tokens and generate the final expressions autoregressively in a single pass. They did not take into account the step-by-step internal text-matching processes behind the final results. This significantly hinders the efficacy and interpretability of regex generation by neural language models. In this paper, we propose a new paradigm called InfeRE, which decomposes the generation of regexes into chains of step-by-step inference. To enhance the robustness, we introduce a self-consistency decoding mechanism that ensembles multiple outputs sampled from different models. We evaluate InfeRE on two publicly available datasets, NL-RX-Turk and KB13, and compare the results with state-of-the-art approaches and the popular tree-based generation approach TRANX. Experimental results show that InfeRE substantially outperforms previous baselines, yielding 16.3% and 14.7% improvement in DFA@5 accuracy on two datasets, respectively. Particularly, InfeRE outperforms the popular tree-based generation approach by 18.1% and 11.3% on both datasets, respectively, in terms of DFA@5 accuracy.
Scalable Performance Analysis for Vision-Language Models

Joint vision-language models have shown great performance over a diverse set of tasks.However, little is known about their limitations, as the high dimensional space learned by these models makes it difficult to identify semantic errors.Recent work has addressed this problem by designing highly controlled probing task benchmarks.Our paper introduces a more scalable solution that relies on already annotated benchmarks.Our method consists of extracting a large set of diverse features from a vision-language benchmark and measuring their correlation with the output of the target model.We confirm previous findings that CLIP behaves like a bag of words model and performs better with nouns and verbs; we also uncover novel insights such as CLIP getting confused by concrete words.Our framework is available at https://github.com/MichiganNLP/Scalable-VLM-Probing and can be used with other multimodal models and benchmarks.
Including Facial Expressions in Contextual Embeddings for Sign Language Generation

State-of-the-art sign language generation frameworks lack expressivity and naturalness which is the result of only focusing on manual signs, neglecting the affective, grammatical, and semantic functions of facial expressions.The purpose of this work is to augment semantic representation of sign language through grounding facial expressions.We study the effect of modeling the relationship between text, gloss, and facial expressions on the performance of the sign generation systems.In particular, we propose a Dual Encoder Transformer able to generate manual signs as well as facial expressions by capturing the similarities and differences found in the text and sign gloss annotation.We take into consideration the role of facial muscle activity to express intensities of manual signs by being the first to employ facial action units in sign language generation.We perform a series of experiments showing that our proposed model improves the quality of automatically generated sign language.
Representation of Lexical Stylistic Features in Language Models’ Embedding Space

The representation space of pretrained Language Models (LMs) encodes rich information about words and their relationships (e.g., similarity, hypernymy, polysemy) as well as abstract semantic notions (e.g., intensity).In this paper, we demonstrate that lexical stylistic notions such as complexity, formality, and figurativeness, can also be identified in this space.We show that it is possible to derive a vector representation for each of these stylistic notions from only a small number of seed pairs.Using these vectors, we can characterize new texts in terms of these dimensions by performing simple calculations in the corresponding embedding space.We conduct experiments on five datasets and find that static embeddings encode these features more accurately at the level of words and phrases, whereas contextualized LMs perform better on sentences.The lower performance of contextualized representations at the word level is partially attributable to the anisotropy of their vector space, which can be corrected to some extent using techniques like standardization. 1
Does Character-level Information Always Improve DRS-based Semantic Parsing?

Even in the era of massive language models, it has been suggested that character-level representations improve the performance of neural models.The state-of-the-art neural semantic parser for Discourse Representation Structures uses character-level representations, improving performance in the four languages (i.e., English, German, Dutch, and Italian) in the Parallel Meaning Bank dataset.However, how and why character-level information improves the parser's performance remains unclear.This study provides an in-depth analysis of performance changes by order of character sequences.In the experiments, we compare F1-scores by shuffling the order and randomizing character sequences after testing the performance of character-level information.Our results indicate that incorporating characterlevel information does not improve the performance in English and German.In addition, we find that the parser is not sensitive to correct character order in Dutch.Nevertheless, performance improvements are observed when using character-level information.
Event Semantic Knowledge in Procedural Text Understanding

The task of entity state tracking aims to automatically analyze procedural texts -texts that describe a step-by-step process (e.g. a baking recipe).Specifically, the goal is to track various states of the entities participating in a given process.Some of the challenges for this NLP task include annotated data scarcity and annotators' reliance on commonsense knowledge to annotate implicit state information.Zhang et al. ( 2021) successfully incorporated commonsense entity-centric knowledge from Con-ceptNet into their BERT-based neurosymbolic architecture.Since English mostly encodes state change information in verbs, we attempted to test whether injecting semantic knowledge of events (retrieved from the state-of-the-art Verb-Net parser) into a neural model can also improve the performance on this task.To achieve this, we adapt the methodology introduced by Zhang et al. ( 2021) for incorporating symbolic entity information from ConceptNet to the incorporation of VerbNet event semantics.We evaluate the performance of our model on the ProPara dataset (Mishra et al., 2018).In addition, we introduce LEXIS, our purely symbolic model for entity state tracking that uses a simple set of case statements, and is informed mostly by linguistic knowledge retrieved from various computational lexical resources.Our approach is inherently domain-agnostic, and our model is explainable and achieves state-ofthe-art results on the Recipes dataset (Bosselut et al., 2017).