Reviewerly/exHarmony huggingface dataset:
Penetrating Shields: A Systematic Analysis of Memory Corruption Mitigations in the Spectre Era

This paper provides the first systematic analysis of a synergistic threat model encompassing memory corruption vulnerabilities and microarchitectural side-channel vulnerabilities. We study speculative shield bypass attacks that leverage speculative execution attacks to leak secrets that are critical to the security of memory corruption mitigations (i.e., the shields), and then use the leaked secrets to bypass the mitigation mechanisms and successfully conduct memory corruption exploits, such as control-flow hijacking. We start by systematizing a taxonomy of the state-of-the-art memory corruption mitigations focusing on hardware-software co-design solutions. The taxonomy helps us to identify 10 likely vulnerable defense schemes out of 20 schemes that we analyze. Next, we develop a graph-based model to analyze the 10 likely vulnerable defenses and reason about possible countermeasures. Finally, we present three proof-of-concept attacks targeting an already-deployed mitigation mechanism and two state-of-the-art academic proposals.
From Text to Mask: Localizing Entities Using the Attention of Text-to-Image Diffusion Models

Diffusion models have revolted the field of text-to-image generation recently. The unique way of fusing text and image information contributes to their remarkable capability of generating highly text-related images. From another perspective, these generative models imply clues about the precise correlation between words and pixels. In this work, a simple but effective method is proposed to utilize the attention mechanism in the denoising network of text-to-image diffusion models. Without re-training nor inference-time optimization, the semantic grounding of phrases can be attained directly. We evaluate our method on Pascal VOC 2012 and Microsoft COCO 2014 under weakly-supervised semantic segmentation setting and our method achieves superior performance to prior methods. In addition, the acquired word-pixel correlation is found to be generalizable for the learned text embedding of customized generation methods, requiring only a few modifications. To validate our discovery, we introduce a new practical task called "personalized referring image segmentation" with a new dataset. Experiments in various situations demonstrate the advantages of our method compared to strong baselines on this task. In summary, our work reveals a novel way to extract the rich multi-modal knowledge hidden in diffusion models for segmentation.
Complete analysis of a realistic fiber-based quantum repeater scheme

We present a quantum repeater protocol for distributing entanglement over long distances, where each repeater node contains several qubits that can couple to one single-photon emitter. Photons from the emitters perform heralded entanglement generation between qubits in neighboring nodes. The protocol leaves the emitters disentangled from the qubits and photons, thus allowing them to be reused to entangle other qubits. The protocol can therefore be time multiplexed, which increases the rate of generated EPR pairs. Deterministic entanglement swapping and heralded entanglement purification are used to extend the distance of the entanglement and reduce the error of the entangled qubits, respectively. We perform a complete protocol analysis by considering all relevant error sources, such as initialization, two-qubit gate, and qubit measurement errors, as well as the exponential decoherence of the qubits with time. The latter is particularly important since we analyze the protocol performance for a broad range of experimental parameters and obtain secret key rates ranging from $1 \rightarrow 1000$ Hz at a distance of $1000$ km. Our results suggest that it is important to reach a qubit memory coherence time of around one second, and two-qubit gate and measurement errors in the order of $10^{-3}$ to obtain reasonable secret key rates over distances longer than achievable with direct transmission.
Stereo Matching in Time: 100+ FPS Video Stereo Matching for Extended Reality

Real-time Stereo Matching is a cornerstone algorithm for many Extended Reality (XR) applications, such as indoor 3D understanding, video pass-through, and mixed-reality games. Despite significant advancements in deep stereo methods, achieving real-time depth inference with high accuracy on a low-power device remains a major challenge. One of the major difficulties is the lack of high-quality indoor video stereo training datasets captured by head-mounted VR/AR glasses. To address this issue, we introduce a novel video stereo synthetic dataset that comprises photorealistic renderings of various indoor scenes and realistic camera motion captured by a 6-DoF moving VR/AR head-mounted display (HMD). This facilitates the evaluation of existing approaches and promotes further research on indoor augmented reality scenarios. Our newly proposed dataset enables us to develop a novel framework for continuous video-rate stereo matching. As another contribution, our dataset enables us to proposed a new video-based stereo matching approach tailored for XR applications, which achieves real-time inference at an impressive 134fps on a standard desktop computer, or 30fps on a battery-powered HMD. Our key insight is that disparity and contextual information are highly correlated and redundant between consecutive stereo frames. By unrolling an iterative cost aggregation in time (i.e. in the temporal dimension), we are able to distribute and reuse the aggregated features over time. This approach leads to a substantial reduction in computation without sacrificing accuracy. We conducted extensive evaluations and comparisons and demonstrated that our method achieves superior performance compared to the current state-of-the-art, making it a strong contender for real-time stereo matching in VR/AR applications.
Tapping into Permutation Symmetry for Improved Detection of k-Symmetric Extensions

Symmetric extensions are essential in quantum mechanics, providing a lens to investigate the correlations of entangled quantum systems and to address challenges like the quantum marginal problem. Though semi-definite programming (SDP) is a recognized method for handling symmetric extensions, it grapples with computational constraints, especially due to the large real parameters in generalized qudit systems. In this study, we introduce an approach that adeptly leverages permutation symmetry. By fine-tuning the SDP problem for detecting \( k \)-symmetric extensions, our method markedly diminishes the searching space dimensionality and trims the number of parameters essential for positive definiteness tests. This leads to an algorithmic enhancement, reducing the complexity from \( O(d^{2k}) \) to \( O(k^{d^2}) \) in the qudit \( k \)-symmetric extension scenario. Additionally, our approach streamlines the process of verifying the positive definiteness of the results. These advancements pave the way for deeper insights into quantum correlations, highlighting potential avenues for refined research and innovations in quantum information theory.
UQ at #SMM4H 2023: ALEX for Public Health Analysis with Social Media

As social media becomes increasingly popular, more and more activities related to public health emerge. Current techniques for public health analysis involve popular models such as BERT and large language models (LLMs). However, the costs of training in-domain LLMs for public health are especially expensive. Furthermore, such kinds of in-domain datasets from social media are generally imbalanced. To tackle these challenges, the data imbalance issue can be overcome by data augmentation and balanced training. Moreover, the ability of the LLMs can be effectively utilized by prompting the model properly. In this paper, a novel ALEX framework is proposed to improve the performance of public health analysis on social media by adopting an LLMs explanation mechanism. Results show that our ALEX model got the best performance among all submissions in both Task 2 and Task 4 with a high score in Task 1 in Social Media Mining for Health 2023 (SMM4H)[1]. Our code has been released at https:// github.com/YanJiangJerry/ALEX.
Robust online estimation of biophysical neural circuits

The control of neuronal networks, whether biological or neuromorphic, relies on tools for estimating parameters in the presence of model uncertainty. In this work, we explore the robustness of adaptive observers for neuronal estimation. Inspired by biology, we show that decentralization and redundancy help recover the performance of a centralized recursive mean square algorithm in the presence of uncertainty and mismatch on the internal dynamics of the model.
Adaptive Distributed Kernel Ridge Regression: A Feasible Distributed Learning Scheme for Data Silos

Data silos, mainly caused by privacy and interoperability, significantly constrain collaborations among different organizations with similar data for the same purpose. Distributed learning based on divide-and-conquer provides a promising way to settle the data silos, but it suffers from several challenges, including autonomy, privacy guarantees, and the necessity of collaborations. This paper focuses on developing an adaptive distributed kernel ridge regression (AdaDKRR) by taking autonomy in parameter selection, privacy in communicating non-sensitive information, and the necessity of collaborations in performance improvement into account. We provide both solid theoretical verification and comprehensive experiments for AdaDKRR to demonstrate its feasibility and effectiveness. Theoretically, we prove that under some mild conditions, AdaDKRR performs similarly to running the optimal learning algorithms on the whole data, verifying the necessity of collaborations and showing that no other distributed learning scheme can essentially beat AdaDKRR under the same conditions. Numerically, we test AdaDKRR on both toy simulations and two real-world applications to show that AdaDKRR is superior to other existing distributed learning schemes. All these results show that AdaDKRR is a feasible scheme to defend against data silos, which are highly desired in numerous application regions such as intelligent decision-making, pricing forecasting, and performance prediction for products.
Asymmetric Clean Segments-Guided Self-Supervised Learning for Robust Speaker Verification

Contrastive self-supervised learning (CSL) for speaker verification (SV) has drawn increasing interest recently due to its ability to exploit unlabeled data. Performing data augmentation on raw waveforms, such as adding noise or reverberation, plays a pivotal role in achieving promising results in SV. Data augmentation, however, demands meticulous calibration to ensure intact speaker-specific information, which is difficult to achieve without speaker labels. To address this issue, we introduce a novel framework by incorporating clean and augmented segments into the contrastive training pipeline. The clean segments are repurposed to pair with noisy segments to form additional positive and negative pairs. Moreover, the contrastive loss is weighted to increase the difference between the clean and augmented embeddings of different speakers. Experimental results on Voxceleb1 suggest that the proposed framework can achieve a remarkable 19% improvement over the conventional methods, and it surpasses many existing state-of-the-art techniques.
From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting

Selecting the ``right'' amount of information to include in a summary is a difficult task. A good summary should be detailed and entity-centric without being overly dense and hard to follow. To better understand this tradeoff, we solicit increasingly dense GPT-4 summaries with what we refer to as a ``Chain of Density'' (CoD) prompt. Specifically, GPT-4 generates an initial entity-sparse summary before iteratively incorporating missing salient entities without increasing the length. Summaries generated by CoD are more abstractive, exhibit more fusion, and have less of a lead bias than GPT-4 summaries generated by a vanilla prompt. We conduct a human preference study on 100 CNN DailyMail articles and find that that humans prefer GPT-4 summaries that are more dense than those generated by a vanilla prompt and almost as dense as human written summaries. Qualitative analysis supports the notion that there exists a tradeoff between informativeness and readability. 500 annotated CoD summaries, as well as an extra 5,000 unannotated summaries, are freely available on HuggingFace (https://huggingface.co/datasets/griffin/chain_of_density).
Provider Fairness and Beyond-Accuracy Trade-offs in Recommender Systems

Recommender systems, while transformative in online user experiences, have raised concerns over potential provider-side fairness issues. These systems may inadvertently favor popular items, thereby marginalizing less popular ones and compromising provider fairness. While previous research has recognized provider-side fairness issues, the investigation into how these biases affect beyond-accuracy aspects of recommendation systems - such as diversity, novelty, coverage, and serendipity - has been less emphasized. In this paper, we address this gap by introducing a simple yet effective post-processing re-ranking model that prioritizes provider fairness, while simultaneously maintaining user relevance and recommendation quality. We then conduct an in-depth evaluation of the model's impact on various aspects of recommendation quality across multiple datasets. Specifically, we apply the post-processing algorithm to four distinct recommendation models across four varied domain datasets, assessing the improvement in each metric, encompassing both accuracy and beyond-accuracy aspects. This comprehensive analysis allows us to gauge the effectiveness of our approach in mitigating provider biases. Our findings underscore the effectiveness of the adopted method in improving provider fairness and recommendation quality. They also provide valuable insights into the trade-offs involved in achieving fairness in recommender systems, contributing to a more nuanced understanding of this complex issue.
LLMCad: Fast and Scalable On-device Large Language Model Inference

Generative tasks, such as text generation and question answering, hold a crucial position in the realm of mobile applications. Due to their sensitivity to privacy concerns, there is a growing demand for their execution directly on mobile devices. Currently, the execution of these generative tasks heavily depends on Large Language Models (LLMs). Nevertheless, the limited memory capacity of these devices presents a formidable challenge to the scalability of such models. In our research, we introduce LLMCad, an innovative on-device inference engine specifically designed for efficient generative Natural Language Processing (NLP) tasks. The core idea behind LLMCad revolves around model collaboration: a compact LLM, residing in memory, takes charge of generating the most straightforward tokens, while a high-precision LLM steps in to validate these tokens and rectify any identified errors. LLMCad incorporates three novel techniques: (1) Instead of generating candidate tokens in a sequential manner, LLMCad employs the smaller LLM to construct a token tree, encompassing a wider range of plausible token pathways. Subsequently, the larger LLM can efficiently validate all of these pathways simultaneously. (2) It employs a self-adjusting fallback strategy, swiftly initiating the verification process whenever the smaller LLM generates an erroneous token. (3) To ensure a continuous flow of token generation, LLMCad speculatively generates tokens during the verification process by implementing a compute-IO pipeline. Through an extensive series of experiments, LLMCad showcases an impressive token generation speed, achieving rates up to 9.3x faster than existing inference engines.
Predictive Analysis of Rice Pest Distribution in Bali Province Using Backpropagation Neural Network

The distribution of pests in rice plants results in significant losses in production and damage to rice plants for farmers, seen from data on the area of rice borer attacks in the province of Bali in Tabanan district. Therefore, by predicting the distribution of rice pests, we can know the pattern of pest attacks so that we can anticipate them because predicting can provide accuracy and error values through the test results. One of the prediction models is BPNN, where BPNN's advantages for solving complex problems are very suitable for use where large amounts of data are involved and many input/output variables, BPNN is also capable of modeling nonlinear relationships between input and output variables, which may be difficult to capture by this type of predictive model. other. Backpropagation includes supervised learning, which means it can learn from labeled examples and can make accurate predictions on new, unlabeled data. Split data using K-fold cross-validation serves to assess the process performance of an algorithmic method by dividing random data samples and grouping the data as many as K k-fold values.
Encoding Multi-Domain Scientific Papers by Ensembling Multiple CLS Tokens

Many useful tasks on scientific documents, such as topic classification and citation prediction, involve corpora that span multiple scientific domains. Typically, such tasks are accomplished by representing the text with a vector embedding obtained from a Transformer's single CLS token. In this paper, we argue that using multiple CLS tokens could make a Transformer better specialize to multiple scientific domains. We present Multi2SPE: it encourages each of multiple CLS tokens to learn diverse ways of aggregating token embeddings, then sums them up together to create a single vector representation. We also propose our new multi-domain benchmark, Multi-SciDocs, to test scientific paper vector encoders under multi-domain settings. We show that Multi2SPE reduces error by up to 25 percent in multi-domain citation prediction, while requiring only a negligible amount of computation in addition to one BERT forward pass.
Identifying Single-Input Linear System Dynamics from Reachable Sets

This paper is concerned with identifying linear system dynamics without the knowledge of individual system trajectories, but from the knowledge of the system's reachable sets observed at different times. Motivated by a scenario where the reachable sets are known from partially transparent manufacturer specifications or observations of the collective behavior of adversarial agents, we aim to utilize such sets to determine the unknown system's dynamics. This paper has two contributions. Firstly, we show that the sequence of the system's reachable sets can be used to uniquely determine the system's dynamics for asymmetric input sets under some generic assumptions, regardless of the system's dimensions. We also prove the same property holds up to a sign change for two-dimensional systems where the input set is symmetric around zero. Secondly, we present an algorithm to determine these dynamics. We apply and verify the developed theory and algorithms on an unknown band-pass filter circuit solely provided the unknown system's reachable sets over a finite observation period.
Seeing-Eye Quadruped Navigation with Force Responsive Locomotion Control

Seeing-eye robots are very useful tools for guiding visually impaired people, potentially producing a huge societal impact given the low availability and high cost of real guide dogs. Although a few seeing-eye robot systems have already been demonstrated, none considered external tugs from humans, which frequently occur in a real guide dog setting. In this paper, we simultaneously train a locomotion controller that is robust to external tugging forces via Reinforcement Learning (RL), and an external force estimator via supervised learning. The controller ensures stable walking, and the force estimator enables the robot to respond to the external forces from the human. These forces are used to guide the robot to the global goal, which is unknown to the robot, while the robot guides the human around nearby obstacles via a local planner. Experimental results in simulation and on hardware show that our controller is robust to external forces, and our seeing-eye system can accurately detect force direction. We demonstrate our full seeing-eye robot system on a real quadruped robot with a blindfolded human. The video can be seen at our project page: https://bu-air-lab.github.io/guide_dog/
Robust Representation Learning for Privacy-Preserving Machine Learning: A Multi-Objective Autoencoder Approach

Several domains increasingly rely on machine learning in their applications. The resulting heavy dependence on data has led to the emergence of various laws and regulations around data ethics and privacy and growing awareness of the need for privacy-preserving machine learning (ppML). Current ppML techniques utilize methods that are either purely based on cryptography, such as homomorphic encryption, or that introduce noise into the input, such as differential privacy. The main criticism given to those techniques is the fact that they either are too slow or they trade off a model s performance for improved confidentiality. To address this performance reduction, we aim to leverage robust representation learning as a way of encoding our data while optimizing the privacy-utility trade-off. Our method centers on training autoencoders in a multi-objective manner and then concatenating the latent and learned features from the encoding part as the encoded form of our data. Such a deep learning-powered encoding can then safely be sent to a third party for intensive training and hyperparameter tuning. With our proposed framework, we can share our data and use third party tools without being under the threat of revealing its original form. We empirically validate our results on unimodal and multimodal settings, the latter following a vertical splitting system and show improved performance over state-of-the-art.
Effect of Electron-Phonon Interactions on Three-Level QD-based Spaser: Linear and Quadratic Potentials

In this article, a spaser (surface plasmon amplification by stimulated emission of radiation) system consisting of a metal nanoparticle surrounded by a large number of quantum dots (QDs) is studied. Usually, the effect of electron-phonon interaction is neglected in the spaser related literature. But some gain media, attributed by the large Raman scattering cross-section, exhibit stronger electron-phonon interaction. No such study has been performed for a QD-based spaser. Hence, it warrants investigation of the same in spaser system. In the present work, we investigate the effects of electron-phonon interaction on a three-level QD-based spaser. We consider two types of interaction potentials, linear and quadratic, and analyze their effects individually. First, we focus on the linear electron-phonon interaction that perturbs the electrons present in the excited state. This yields a periodic steady-state number of localized surface plasmon (LSP). The accompanying analytic solution reveals that the population inversion of the gain medium depends on the linear potential strength (Frohlich constant) but does not affect the threshold of spaser considerably for the given numerical parameters. In addition to the LSP, phonons may be generated during this process, the temporal dynamics of which are also detailed. Initially, the number of phonons exhibit decaying periodic oscillations, whose amplitude depends on the strength of the electron-phonon interaction. Under continuous pumping, at later times, the number of phonons reaches a steady-state value, which may find application in realization of continuous phonon nanolasers. Further, the effect of the quadratic potential is studied phenomenologically by increasing the excited-state decay rate. This results in a large number of LSP and an intense spaser spectrum.