Reviewerly/exHarmony huggingface dataset:
k* Distribution: Evaluating the Latent Space of Deep Neural Networks using Local Neighborhood Analysis

Most examinations of neural networks' learned latent spaces typically employ dimensionality reduction techniques such as t-SNE or UMAP. While these methods effectively capture the overall sample distribution in the entire learned latent space, they tend to distort the structure of sample distributions within specific classes in the subset of the latent space. This distortion complicates the task of easily distinguishing classes identifiable by neural networks. In response to this challenge, we introduce the k* Distribution methodology. This approach focuses on capturing the characteristics and structure of sample distributions for individual classes within the subset of the learned latent space using local neighborhood analysis. The key concept is to facilitate easy comparison of different k* distributions, enabling analysis of how various classes are processed by the same neural network. This provides a more profound understanding of existing contemporary visualizations. Our study reveals three distinct distributions of samples within the learned latent space subset: a) Fractured, b) Overlapped, and c) Clustered. We note and demonstrate that the distribution of samples within the network's learned latent space significantly varies depending on the class. Furthermore, we illustrate that our analysis can be applied to explore the latent space of diverse neural network architectures, various layers within neural networks, transformations applied to input samples, and the distribution of training and testing data for neural networks. We anticipate that our approach will facilitate more targeted investigations into neural networks by collectively examining the distribution of different samples within the learned latent space.
Combining inherent knowledge of vision-language models with unsupervised domain adaptation through self-knowledge distillation

Unsupervised domain adaptation (UDA) tries to overcome the tedious work of labeling data by leveraging a labeled source dataset and transferring its knowledge to a similar but different target dataset. On the other hand, current vision-language models exhibit astonishing zero-shot prediction capabilities. In this work, we combine knowledge gained through UDA with the inherent knowledge of vision-language models. In a first step, we generate the zero-shot predictions of the source and target dataset using the vision-language model. Since zero-shot predictions usually exhibit a large entropy, meaning that the class probabilities are rather evenly distributed, we first adjust the distribution to accentuate the winning probabilities. This is done using both source and target data to keep the relative confidence between source and target data. We then employ a conventional DA method, to gain the knowledge from the source dataset, in combination with self-knowledge distillation, to maintain the inherent knowledge of the vision-language model. We further combine our method with a gradual source domain expansion strategy (GSDE) and show that this strategy can also benefit by including zero-shot predictions. We conduct experiments and ablation studies on three benchmarks (OfficeHome, VisDA, and DomainNet) and outperform state-of-the-art methods. We further show in ablation studies the contributions of different parts of our algorithm.
On the adaptation of in-context learners for system identification

In-context system identification aims at constructing meta-models to describe classes of systems, differently from traditional approaches that model single systems. This paradigm facilitates the leveraging of knowledge acquired from observing the behaviour of different, yet related dynamics. This paper discusses the role of meta-model adaptation. Through numerical examples, we demonstrate how meta-model adaptation can enhance predictive performance in three realistic scenarios: tailoring the meta-model to describe a specific system rather than a class; extending the meta-model to capture the behaviour of systems beyond the initial training class; and recalibrating the model for new prediction tasks. Results highlight the effectiveness of meta-model adaptation to achieve a more robust and versatile meta-learning framework for system identification.
MTVG : Multi-text Video Generation with Text-to-Video Models

Recently, video generation has attracted massive attention and yielded noticeable outcomes. Concerning the characteristics of video, multi-text conditioning incorporating sequential events is necessary for next-step video generation. In this work, we propose a novel multi-text video generation~(MTVG) by directly utilizing a pre-trained diffusion-based text-to-video~(T2V) generation model without additional fine-tuning. To generate consecutive video segments, visual consistency generated by distinct prompts is necessary with diverse variations, such as motion and content-related transitions. Our proposed MTVG includes Dynamic Noise and Last Frame Aware Inversion which reinitialize the noise latent to preserve visual coherence between videos of different prompts and prevent repetitive motion or contents. Furthermore, we present Structure Guiding Sampling to maintain the global appearance across the frames in a single video clip, where we leverage iterative latent updates across the preceding frame. Additionally, our Prompt Generator allows for arbitrary format of text conditions consisting of diverse events. As a result, our extensive experiments, including diverse transitions of descriptions, demonstrate that our proposed methods show superior generated outputs in terms of semantically coherent and temporally seamless video.Video examples are available in our project page: https://kuai-lab.github.io/mtvg-page.
VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Models

With recent advancements in Large Multimodal Models (LMMs) across various domains, a novel prompting method called visual referring prompting has emerged, showing significant potential in enhancing human-computer interaction within multimodal systems. This method offers a more natural and flexible approach to human interaction with these systems compared to traditional text descriptions or coordinates. However, the categorization of visual referring prompting remains undefined, and its impact on the performance of LMMs has yet to be formally examined. In this study, we conduct the first comprehensive analysis of LMMs using a variety of visual referring prompting strategies. We introduce a benchmark dataset called VRPTEST, comprising 3 different visual tasks and 2,275 images, spanning diverse combinations of prompt strategies. Using VRPTEST, we conduct a comprehensive evaluation of eight versions of prominent open-source and proprietary foundation models, including two early versions of GPT-4V. We develop an automated assessment framework based on software metamorphic testing techniques to evaluate the accuracy of LMMs without the need for human intervention or manual labeling. We find that the current proprietary models generally outperform the open-source ones, showing an average accuracy improvement of 22.70%; however, there is still potential for improvement. Moreover, our quantitative analysis shows that the choice of prompt strategy significantly affects the accuracy of LMMs, with variations ranging from -17.5% to +7.3%. Further case studies indicate that an appropriate visual referring prompting strategy can improve LMMs' understanding of context and location information, while an unsuitable one might lead to answer rejection. We also provide insights on minimizing the negative impact of visual referring prompting on LMMs.
Synergistic Signals: Exploiting Co-Engagement and Semantic Links via Graph Neural Networks

Given a set of candidate entities (e.g. movie titles), the ability to identify similar entities is a core capability of many recommender systems. Most often this is achieved by collaborative filtering approaches, i.e. if users co-engage with a pair of entities frequently enough, the embeddings should be similar. However, relying on co-engagement data alone can result in lower-quality embeddings for new and unpopular entities. We study this problem in the context recommender systems at Netflix. We observe that there is abundant semantic information such as genre, content maturity level, themes, etc. that complements co-engagement signals and provides interpretability in similarity models. To learn entity similarities from both data sources holistically, we propose a novel graph-based approach called SemanticGNN. SemanticGNN models entities, semantic concepts, collaborative edges, and semantic edges within a large-scale knowledge graph and conducts representation learning over it. Our key technical contributions are twofold: (1) we develop a novel relation-aware attention graph neural network (GNN) to handle the imbalanced distribution of relation types in our graph; (2) to handle web-scale graph data that has millions of nodes and billions of edges, we develop a novel distributed graph training paradigm. The proposed model is successfully deployed within Netflix and empirical experiments indicate it yields up to 35% improvement in performance on similarity judgment tasks.
Comparative study on compact quantum circuits of hybrid quantum-classical algorithms for quantum impurity models

Predicting the properties of strongly correlated materials is a significant challenge in condensed matter theory. The widely used dynamical mean-field theory faces difficulty in solving quantum impurity models numerically. Hybrid quantum--classical algorithms such as variational quantum eigensolver emerge as a potential solution for quantum impurity models. A common challenge in these algorithms is the rapid growth of the number of variational parameters with the number of spin-orbitals in the impurity. In our approach to this problem, we develop compact ansatzes using a combination of two different strategies. First, we employ compact physics-inspired ansatz, $k$-unitary cluster Jastrow ansatz, developed in the field of quantum chemistry. Second, we eliminate largely redundant variational parameters of physics-inspired ansatzes associated with bath sites based on physical intuition. This is based on the fact that a quantum impurity model with a star-like geometry has no direct hopping between bath sites. We benchmark the accuracy of these ansatzes for both ground-state energy and dynamic quantities by solving typical quantum impurity models with/without shot noise. The results suggest that we can maintain the accuracy of ground-state energy while we drop the number of variational parameters associated with bath sites. Furthermore, we demonstrate that a moment expansion, when combined with the proposed ansatzes, can calculate the imaginary-time Green's functions under the influence of shot noise. This study demonstrates the potential for addressing complex impurity models in large-scale quantum simulations with fewer variational parameters without sacrificing accuracy.
Device independent security of quantum key distribution from monogamy-of-entanglement games

We analyse two party non-local games whose predicate requires Alice and Bob to generate matching bits, and their three party extensions where a third player receives all inputs and is required to output a bit that matches that of the original players. We propose a general device independent quantum key distribution protocol for the subset of such non-local games that satisfy a monogamy-of-entanglement property characterised by a gap in the maximum winning probability between the bipartite and tripartite versions of the game. This gap is due to the optimal strategy for two players requiring entanglement, which due to its monogamy property cannot be shared with any additional players. Based solely on the monogamy-of-entanglement property, we provide a simple proof of information theoretic security of our protocol. Lastly, we numerically optimize the finite and asymptotic secret key rates of our protocol using the magic square game as an example, for which we provide a numerical bound on the maximal tripartite quantum winning probability which closely matches the bipartite classical winning probability. Further, we show that our protocol is robust for depolarizing noise up to about $2.2\%$, providing the first such bound for general attacks for magic square based quantum key distribution.
A Study on the Calibration of In-context Learning

Accurate uncertainty quantification is crucial for the safe deployment of machine learning models, and prior research has demonstrated improvements in the calibration of modern language models (LMs). We study in-context learning (ICL), a prevalent method for adapting static LMs through tailored prompts, and examine the balance between performance and calibration across a broad spectrum of natural language understanding and reasoning tasks. Through comprehensive experiments, we observe that, with an increasing number of ICL examples, models initially exhibit increased miscalibration before achieving better calibration and miscalibration tends to arise in low-shot settings. Moreover, we find that methods aimed at improving usability, such as fine-tuning and chain-of-thought (CoT) prompting, can lead to miscalibration and unreliable natural language explanations. Furthermore, we explore recalibration techniques and find that a scaling-binning calibrator can reduce calibration errors consistently.
Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection

Recent data-privacy laws have sparked interest in machine unlearning, which involves removing the effect of specific training samples from a learnt model as if they were never present in the original training dataset. The challenge of machine unlearning is to discard information about the ``forget'' data in the learnt model without altering the knowledge about the remaining dataset and to do so more efficiently than the naive retraining approach. To achieve this, we adopt a projected-gradient based learning method, named as Projected-Gradient Unlearning (PGU), in which the model takes steps in the orthogonal direction to the gradient subspaces deemed unimportant for the retaining dataset, so as to its knowledge is preserved. By utilizing Stochastic Gradient Descent (SGD) to update the model weights, our method can efficiently scale to any model and dataset size. We provide empirically evidence to demonstrate that our unlearning method can produce models that behave similar to models retrained from scratch across various metrics even when the training dataset is no longer accessible. Our code is available at https://github.com/hnanhtuan/projected_gradient_unlearning.
PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play

Learning from unstructured and uncurated data has become the dominant paradigm for generative approaches in language and vision. Such unstructured and unguided behavior data, commonly known as play, is also easier to collect in robotics but much more difficult to learn from due to its inherently multimodal, noisy, and suboptimal nature. In this paper, we study this problem of learning goal-directed skill policies from unstructured play data which is labeled with language in hindsight. Specifically, we leverage advances in diffusion models to learn a multi-task diffusion model to extract robotic skills from play data. Using a conditional denoising diffusion process in the space of states and actions, we can gracefully handle the complexity and multimodality of play data and generate diverse and interesting robot behaviors. To make diffusion models more useful for skill learning, we encourage robotic agents to acquire a vocabulary of skills by introducing discrete bottlenecks into the conditional behavior generation process. In our experiments, we demonstrate the effectiveness of our approach across a wide variety of environments in both simulation and the real world. Results visualizations and videos at https://play-fusion.github.io
Graph Convolutions Enrich the Self-Attention in Transformers!

Transformers, renowned for their self-attention mechanism, have achieved state-of-the-art performance across various tasks in natural language processing, computer vision, time-series modeling, etc. However, one of the challenges with deep Transformer models is the oversmoothing problem, where representations across layers converge to indistinguishable values, leading to significant performance degradation. We interpret the original self-attention as a simple graph filter and redesign it from a graph signal processing (GSP) perspective. We propose graph-filter-based self-attention (GFSA) to learn a general yet effective one, whose complexity, however, is slightly larger than that of the original self-attention mechanism. We demonstrate that GFSA improves the performance of Transformers in various fields, including computer vision, natural language processing, graph pattern classification, speech recognition, and code classification.
Resource-Efficient Quantum Circuits for Molecular Simulations: A Case Study of Umbrella Inversion in Ammonia

We conducted a thorough evaluation of various state-of-the-art strategies to prepare the ground state wavefunction of a system on a quantum computer, specifically within the framework of variational quantum eigensolver (VQE). Despite the advantages of VQE and its variants, the current quantum computational chemistry calculations often provide inaccurate results for larger molecules, mainly due to the polynomial growth in the depth of quantum circuits and the number of two-qubit gates, such as CNOT gates. To alleviate this problem, we aim to design efficient quantum circuits that would outperform the existing ones on the current noisy quantum devices. In this study, we designed a novel quantum circuit that reduces the required circuit depth and number of two-qubit entangling gates by about 60%, while retaining the accuracy of the ground state energies close to the chemical accuracy. Moreover, even in the presence of device noise, these novel shallower circuits yielded substantially low error rates than the existing approaches for predicting the ground state energies of molecules. By considering the umbrella inversion process in ammonia molecule as an example, we demonstrated the advantages of this new approach and estimated the energy barrier for the inversion process.
General teleportation channel in Fermionic Quantum Theory

Quantum Teleportation is a very useful scheme for transferring quantum information. Given that the quantum information is encoded in a state of a system of distinguishable particles, and given that the shared bi-partite entangled state is also that of a system of distinguishable particles, the optimal teleportation fidelity of the shared state is known to be $(F_{max}d+1)/(d+1)$ with $F_{max}$ being the `maximal singlet fraction' of the shared state. In the present work, we address the question of optimal teleportation fidelity given that the quantum information to be teleported is encoded in Fermionic modes while a $2N$-mode state of a system of Fermions (with maximum $2N$ no. of Fermions -- in the second quantization language) is shared between the sender and receiver with each party possessing $N$ modes of the $2N$-mode state. Parity Superselection Rule (PSSR) in Fermionic Quantum Theory (FQT) puts constraint on the allowed set of physical states and operations, and thereby, leads to a different notion of Quantum Teleportation. Due to PSSR, we introduce restricted Clifford twirl operations that constitute the Unitary 2-design in case of FQT, and show that the structure of the canonical form of Fermionic invariant shared state differs from that of the isotropic state -- the corresponding canonical invariant form for teleportation in Standard Quantum Theory (SQT). We provide a lower bound on the optimal teleportation fidelity in FQT and compare the result with teleportation in SQT. Surprisingly, we find that, under separable measurements on a bipartite Fermionic state, input and output states of the Fermionic teleportation channel cannot be distinguished operationally, even if a particular kind of resource state with `maximal singlet fraction' being less than unity is used.
TeMO: Towards Text-Driven 3D Stylization for Multi-Object Meshes

Recent progress in the text-driven 3D stylization of a single object has been considerably promoted by CLIP-based methods. However, the stylization of multi-object 3D scenes is still impeded in that the image-text pairs used for pre-training CLIP mostly consist of an object. Meanwhile, the local details of multiple objects may be susceptible to omission due to the existing supervision manner primarily relying on coarse-grained contrast of image-text pairs. To overcome these challenges, we present a novel framework, dubbed TeMO, to parse multi-object 3D scenes and edit their styles under the contrast supervision at multiple levels. We first propose a Decoupled Graph Attention (DGA) module to distinguishably reinforce the features of 3D surface points. Particularly, a cross-modal graph is constructed to align the object points accurately and noun phrases decoupled from the 3D mesh and textual description. Then, we develop a Cross-Grained Contrast (CGC) supervision system, where a fine-grained loss between the words in the textual description and the randomly rendered images are constructed to complement the coarse-grained loss. Extensive experiments show that our method can synthesize high-quality stylized content and outperform the existing methods over a wide range of multi-object 3D meshes. Our code and results will be made publicly available
Large Language Models for Mathematicians

Large language models (LLMs) such as ChatGPT have received immense interest for their general-purpose language understanding and, in particular, their ability to generate high-quality text or computer code. For many professions, LLMs represent an invaluable tool that can speed up and improve the quality of work. In this note, we discuss to what extent they can aid professional mathematicians. We first provide a mathematical description of the transformer model used in all modern language models. Based on recent studies, we then outline best practices and potential issues and report on the mathematical abilities of language models. Finally, we shed light on the potential of LLMs to change how mathematicians work.
New ternary self-orthogonal codes and related LCD codes from weakly regular plateaued functions

A linear code is said to be self-orthogonal if it is contained in its dual. Self-orthogonal codes are of interest because of their important applications, such as for constructing linear complementary dual (LCD) codes and quantum codes. In this paper, we construct several new families of ternary self-orthogonal codes by employing weakly regular plateaued functions. Their parameters and weight distributions are completely determined. Then we apply these self-orthogonal codes to construct several new families of ternary LCD codes. As a consequence, we obtain many (almost) optimal ternary self-orthogonal codes and LCD codes.