Miracle-dz/UniFAR-FTUs huggingface dataset:
MUST: An Effective and Scalable Framework for Multimodal Search of Target Modality

We investigate the problem of multimodal search of target modality, where the task involves enhancing a query in a specific target modality by integrating information from auxiliary modalities. The goal is to retrieve relevant objects whose contents in the target modality match the specified multimodal query. The paper first introduces two baseline approaches that integrate techniques from the Database, Information Retrieval, and Computer Vision communities. These baselines either merge the results of separate vector searches for each modality or perform a single-channel vector search by fusing all modalities. However, both baselines have limitations in terms of efficiency and accuracy as they fail to adequately consider the varying importance of fusing information across modalities. To overcome these limitations, the paper proposes a novel framework, called MUST. Our framework employs a hybrid fusion mechanism, combining different modalities at multiple stages. Notably, we leverage vector weight learning to determine the importance of each modality, thereby enhancing the accuracy of joint similarity measurement. Additionally, the proposed framework utilizes a fused proximity graph index, enabling efficient joint search for multimodal queries. MUST offers several other advantageous properties, including pluggable design to integrate any advanced embedding techniques, user flexibility to customize weight preferences, and modularized index construction. Extensive experiments on real-world datasets demonstrate the superiority of MUST over the baselines in terms of both search accuracy and efficiency. Our framework achieves over 10x faster search times while attaining an average of 93% higher accuracy. Furthermore, MUST exhibits scalability to datasets containing more than 10 million data elements.
Contrastive Multiview Subspace Clustering of Hyperspectral Images Based on Graph Convolutional Networks

High-dimensional and complex spectral structures make the clustering of hyperspectral images (HSIs) a challenging task. Subspace clustering is an effective approach for addressing this problem. However, current subspace clustering algorithms are primarily designed for a single view and do not fully exploit the spatial or textural feature information in HSI. In this study, contrastive multiview subspace clustering of HSI was proposed based on graph convolutional networks. Pixel neighbor textural and spatial–spectral information was sent to construct two graph convolutional subspaces to learn their affinity matrices. To maximize the interaction between different views, a contrastive learning algorithm was introduced to promote the consistency of positive samples and assist the model in extracting robust features. An attention-based fusion module was used to adaptively integrate these affinity matrices, constructing a more discriminative affinity matrix. The model was evaluated using four popular HSI datasets: Indian Pines, Pavia University, Houston, and Xu Zhou. It achieved overall accuracies of 97.61%, 96.69%, 87.21%, and 97.65%, respectively, and significantly outperformed state-of-the-art clustering methods. In conclusion, the proposed model effectively improves the clustering accuracy of HSI. Our implementation is available at https://github.com/GuanRX/CMSCGC.
Effects of rainbow gravity on charge-free scalar Bosons in Bonnor-Melvin s universe with cosmological constant

In this paper, we explore the relativistic quantum motion of spin-zero scalar particles influenced by rainbow gravity's in the background of a magnetic solution. Our focus is on the Bonnor-Melvin magnetic space-time, a four-dimensional solution featuring a non-zero cosmological constant. To analyze this scenario, we solve the Klein-Gordon equation using two sets of rainbow functions: (i) $f(\chi)=\frac{1}{(1-\beta\,\chi)}$, $h(\chi)=1$ and (ii) $f(\chi)=\frac{1}{(1-\beta\,\chi)}=h(\chi)$, where $0<\chi(=\frac{|E|}{E_p}) \leq 1$ with $E_p$ being Planck's energy. The resulting solutions provide the relativistic energy profiles of the scalar particles. Furthermore, we study the quantum oscillator fields through the Klein-Gordon oscillator within the same Bonnor-Melvin magnetic space-time background. Employing $f(\chi)=\frac{1}{(1-\beta\,\chi)}$, $h(\chi)=1$ as the rainbow function, we obtain the eigenvalue solutions for the quantum oscillator fields. Notably, we demonstrate that the relativistic energy profile of scalar particles and the oscillator fields are influenced by the topology of the geometry and the cosmological constant both are connected with the magnetic field strength. Additionally, we highlight the impact of the rainbow parameter on the energy profiles.
Optimization for the Metaverse over Mobile Edge Computing with Play to Earn

The concept of the Metaverse has garnered growing interest from both academic and industry circles. The decentralization of both the integrity and security of digital items has spurred the popularity of play-to-earn (P2E) games, where players are entitled to earn and own digital assets which they may trade for physical-world currencies. However, these computationally-intensive games are hardly playable on resource-limited mobile devices and the computational tasks have to be offloaded to an edge server. Through mobile edge computing (MEC), users can upload data to the Metaverse Service Provider (MSP) edge servers for computing. Nevertheless, there is a trade-off between user-perceived in-game latency and user visual experience. The downlink transmission of lower-resolution videos lowers user-perceived latency while lowering the visual fidelity and consequently, earnings of users. In this paper, we design a method to enhance the Metaverse-based mobile augmented reality (MAR) in-game user experience. Specifically, we formulate and solve a multi-objective optimization problem. Given the inherent NP-hardness of the problem, we present a low-complexity algorithm to address it, mitigating the trade-off between delay and earnings. The experiment results show that our method can effectively balance the user-perceived latency and profitability, thus improving the performance of Metaverse-based MAR systems.
Analytical Insights into Constant-Roll Condition: Extending the Paradigm to Non-Canonical Models

In this work, we explore the prospect of generalizing the constant-roll condition in canonical inflationary model to non-canonical models. To find a natural generalization, we focus on three manifestations of this condition and construct constant-roll models corresponding to each manifestation. These models are not equivalent but reduce to the familiar constant-roll model in canonical limit. To showcase the applicability of our generalized mechanism, we examine a specific class of non-canonical models, which can be viewed as extensions of k/G inflation. In these models sound speed is constant. We conduct a comparative study, and with an analytical examination of the model, specify instances when our constant-roll conditions yield dissimilar outcomes and when they exhibit analogies. We also apply our findings to scrutinize another kinetically driven inflationary model with varying sound speed. We demonstrate that each of our constant-roll conditions leads to a unique set of solutions. Afterward, we construct a four-stage constant-roll kinetically driven inflation that complies with CMB constraints, it sustains for a sufficiently long period of time, and finally gracefully exits. In this model the spectrum of curvature perturbations is enhanced in a brief phase of non-slow-roll inflationary evolution. Employing numerical methods, we analyse this scenario to elucidate how altering the constant-roll condition impacts the power spectrum and the model's dynamics.
InteractDiffusion: Interaction Control in Text-to-Image Diffusion Models

Large-scale text-to-image (T2I) diffusion models have showcased incredible capabilities in generating coherent images based on textual descriptions, enabling vast applications in content generation. While recent advancements have introduced control over factors such as object localization, posture, and image contours, a crucial gap remains in our ability to control the interactions between objects in the generated content. Well-controlling interactions in generated images could yield meaningful applications, such as creating realistic scenes with interacting characters. In this work, we study the problems of conditioning T2I diffusion models with Human-Object Interaction (HOI) information, consisting of a triplet label (person, action, object) and corresponding bounding boxes. We propose a pluggable interaction control model, called InteractDiffusion that extends existing pre-trained T2I diffusion models to enable them being better conditioned on interactions. Specifically, we tokenize the HOI information and learn their relationships via interaction embeddings. A conditioning self-attention layer is trained to map HOI tokens to visual tokens, thereby conditioning the visual tokens better in existing T2I diffusion models. Our model attains the ability to control the interaction and location on existing T2I diffusion models, which outperforms existing baselines by a large margin in HOI detection score, as well as fidelity in FID and KID. Project page: https://jiuntian.github. io/interactdiffusion.
First measurement of the Weyl potential evolution from the Year 3 Dark Energy Survey data: Localising the $\sigma_8$ tension

We present the first measurement of the Weyl potential at four redshifts bins using data from the first three years of observations of the Dark Energy Survey (DES). The Weyl potential, which is the sum of the spatial and temporal distortions of the Universe's geometry, provides a direct way of testing the theory of gravity and the validity of the $\Lambda$CDM model. We find that the measured Weyl potential is 2.3$\sigma$, respectively 3.1$\sigma$, below the $\Lambda$CDM predictions in the two lowest redshift bins. We show that these low values of the Weyl potential are at the origin of the $\sigma_8$ tension between Cosmic Microwave Background (CMB) measurements and weak lensing measurements. Interestingly, we find that the tension remains if no information from the CMB is used. DES data on their own prefer a high value of the primordial fluctuations, followed by a slow evolution of the Weyl potential. A remarkable feature of our method is that the measurements of the Weyl potential are model-independent and can therefore be confronted with any theory of gravity, allowing efficient tests of models beyond General Relativity.
Hypergraph-Guided Disentangled Spectrum Transformer Networks for Near-Infrared Facial Expression Recognition

With the strong robusticity on illumination variations, near-infrared (NIR) can be an effective and essential complement to visible (VIS) facial expression recognition in low lighting or complete darkness conditions. However, facial expression recognition (FER) from NIR images presents a more challenging problem than traditional FER due to the limitations imposed by the data scale and the difficulty of extracting discriminative features from incomplete visible lighting contents. In this paper, we give the first attempt at deep NIR facial expression recognition and propose a novel method called near-infrared facial expression transformer (NFER-Former). Specifically, to make full use of the abundant label information in the field of VIS, we introduce a Self-Attention Orthogonal Decomposition mechanism that disentangles the expression information and spectrum information from the input image, so that the expression features can be extracted without the interference of spectrum variation. We also propose a Hypergraph-Guided Feature Embedding method that models some key facial behaviors and learns the structure of the complex correlations between them, thereby alleviating the interference of inter-class similarity. Additionally, we construct a large NIR-VIS Facial Expression dataset that includes 360 subjects to better validate the efficiency of NFER-Former. Extensive experiments and ablation studies show that NFER-Former significantly improves the performance of NIR FER and achieves state-of-the-art results on the only two available NIR FER datasets, Oulu-CASIA and Large-HFE.
Automotive Radar Sensing with Sparse Linear Arrays Using One-Bit Hankel Matrix Completion

The design of sparse linear arrays has proven instrumental in the implementation of cost-effective and efficient automotive radar systems for high-resolution imaging. This paper investigates the impact of coarse quantization on measurements obtained from such arrays. To recover azimuth angles from quantized measurements, we leverage the low-rank properties of the constructed Hankel matrix. In particular, by addressing the one-bit Hankel matrix completion problem through a developed singular value thresholding algorithm, our proposed approach accurately estimates the azimuth angles of interest. We provide comprehensive insights into recovery performance and the required number of one-bit samples. The effectiveness of our proposed scheme is underscored by numerical results, demonstrating successful reconstruction using only one-bit data.
PCRDiffusion: Diffusion Probabilistic Models for Point Cloud Registration

We propose a new framework that formulates point cloud registration as a denoising diffusion process from noisy transformation to object transformation. During training stage, object transformation diffuses from ground-truth transformation to random distribution, and the model learns to reverse this noising process. In sampling stage, the model refines randomly generated transformation to the output result in a progressive way. We derive the variational bound in closed form for training and provide implementations of the model. Our work provides the following crucial findings: (i) In contrast to most existing methods, our framework, Diffusion Probabilistic Models for Point Cloud Registration (PCRDiffusion) does not require repeatedly update source point cloud to refine the predicted transformation. (ii) Point cloud registration, one of the representative discriminative tasks, can be solved by a generative way and the unified probabilistic formulation. Finally, we discuss and provide an outlook on the application of diffusion model in different scenarios for point cloud registration. Experimental results demonstrate that our model achieves competitive performance in point cloud registration. In correspondence-free and correspondence-based scenarios, PCRDifussion can both achieve exceeding 50\% performance improvements.
An In-Silico Study on Integrated Mechanisms of Mechano-Electric Coupling in Ischaemic Arrhythmogenesis

Heterogeneous mechanical dyskinesis during acute myocardial ischaemia is thought to contribute to arrhythmogenic alterations to cardiac electrophysiology. Various forms of mechano-electric coupling (MEC) mechanisms have been suggested to contribute to these changes, with two primary mechanisms being: (1) myofilament-dependent calcium release events, and (2) the activation of stretch-activated currents (SACs). In this computational investigation, we assessed the collective impact of these processes on mechanically-induced alternans that create an arrhythmogenic substrate during acute ischaemia. To appraise the potential involvement of MEC in ischaemia-induced arrhythmias, we developed a coupled model of ventricular myocyte electrophysiology and contraction including SACs and stretch-dependent calcium buffering and release. The model, reflecting observed electrophysiological changes during ischaemia, was exposed to a series of stretch protocols that replicated both physiological and pathological mechanical conditions. Pathologically realistic myofiber stretch variations revealed calcium sensitivity changes dependent on myofilament, leading to alterations in cytosolic calcium concentrations. Under calcium overload conditions, these changes resulted in electrical alternans. The study implies that strain impacts cellular electrophysiology through myofilament calcium release and SAC opening in ventricular mechano-electrical models, parameterised to available data. This supports experimental evidence suggesting that both calcium-driven instability via MEC and SAC-induced effects contribute to electrical alternans in acute ischaemia.
mir_ref: A Representation Evaluation Framework for Music Information Retrieval Tasks

Music Information Retrieval (MIR) research is increasingly leveraging representation learning to obtain more compact, powerful music audio representations for various downstream MIR tasks. However, current representation evaluation methods are fragmented due to discrepancies in audio and label preprocessing, downstream model and metric implementations, data availability, and computational resources, often leading to inconsistent and limited results. In this work, we introduce mir_ref, an MIR Representation Evaluation Framework focused on seamless, transparent, local-first experiment orchestration to support representation development. It features implementations of a variety of components such as MIR datasets, tasks, embedding models, and tools for result analysis and visualization, while facilitating the implementation of custom components. To demonstrate its utility, we use it to conduct an extensive evaluation of several embedding models across various tasks and datasets, including evaluating their robustness to various audio perturbations and the ease of extracting relevant information from them.
EipFormer: Emphasizing Instance Positions in 3D Instance Segmentation

3D instance segmentation plays a crucial role in comprehending 3D scenes. Despite recent advancements in this field, existing approaches exhibit certain limitations. These methods often rely on fixed instance positions obtained from sampled representative points in vast 3D point clouds, using center prediction or farthest point sampling. However, these selected positions may deviate from actual instance centers, posing challenges in precisely grouping instances. Moreover, the common practice of grouping candidate instances from a single type of coordinates introduces difficulties in identifying neighboring instances or incorporating edge points. To tackle these issues, we present a novel Transformer-based architecture, EipFormer, which comprises progressive aggregation and dual position embedding. The progressive aggregation mechanism leverages instance positions to refine instance proposals. It enhances the initial instance positions through weighted farthest point sampling and further refines the instance positions and proposals using aggregation averaging and center matching. Additionally, dual position embedding superposes the original and centralized position embeddings, thereby enhancing the model performance in distinguishing adjacent instances. Extensive experiments on popular datasets demonstrate that EipFormer achieves superior or comparable performance compared to state-of-the-art approaches.
X2-Softmax: Margin Adaptive Loss Function for Face Recognition

Learning the discriminative features of different faces is an important task in face recognition. By extracting face features in neural networks, it becomes easy to measure the similarity of different face images, which makes face recognition possible. To enhance the neural network's face feature separability, incorporating an angular margin during training is common practice. State-of-the-art loss functions CosFace and ArcFace apply fixed margins between weights of classes to enhance the inter-class separation of face features. Since the distribution of samples in the training set is imbalanced, similarities between different identities are unequal. Therefore, using an inappropriately fixed angular margin may lead to the problem that the model is difficult to converge or the face features are not discriminative enough. It is more in line with our intuition that the margins are angular adaptive, which could increase with the angles between classes growing. In this paper, we propose a new angular margin loss named X2-Softmax. X2-Softmax loss has adaptive angular margins, which provide the margin that increases with the angle between different classes growing. The angular adaptive margin ensures model flexibility and effectively improves the effect of face recognition. We have trained the neural network with X2-Softmax loss on the MS1Mv3 dataset and tested it on several evaluation benchmarks to demonstrate the effectiveness and superiority of our loss function.
Deformation and motion of giant unilamellar vesicles loaded with gold nanoparticles driven by induced charge electro-osmotic flow

A vesicle is a spherical structure composed of a phospholipid bilayer that is used as a container for chemicals, both it in vivo and it in vitro systems. In both cases, the vesicles can be passively moved using external molecular motors or flows. The active motion of the vesicles can potentially expand their applications in microfluid devices. In this study, we created giant unilamellar vesicles (GUVs) that loads dodecanethiol-functionalized gold nanoparticles (AuNPs) using natural swelling method. An external alternating current (AC) electric field was applied to the sample to drive the system. A flow was confirmed with dense optical flow method around GUVs, even in the absence of AuNPs. The quadratic dependence of the flow on applied elecric fields confirms that the flow is due to induced charge electro-osmotic (ICEO) mechanism. Furthermore, the GUVs containing AuNPs moved and deformed significantly under external AC electric fields compared with those without AuNPs. We also confirmed that the translational speed of GUVs was positively correlated with the volume ratio of AuNPs. These experimental results suggest that the motion and deformation of GUVs were cause by ICEO flow, which was unbalanced owing to the presence of localized AuNPs on the membrane.
Deep Photonic Reservoir Computer for Speech Recognition

Speech recognition is a critical task in the field of artificial intelligence and has witnessed remarkable advancements thanks to large and complex neural networks, whose training process typically requires massive amounts of labeled data and computationally intensive operations. An alternative paradigm, reservoir computing, is energy efficient and is well adapted to implementation in physical substrates, but exhibits limitations in performance when compared to more resource-intensive machine learning algorithms. In this work we address this challenge by investigating different architectures of interconnected reservoirs, all falling under the umbrella of deep reservoir computing. We propose a photonic-based deep reservoir computer and evaluate its effectiveness on different speech recognition tasks. We show specific design choices that aim to simplify the practical implementation of a reservoir computer while simultaneously achieving high-speed processing of high-dimensional audio signals. Overall, with the present work we hope to help the advancement of low-power and high-performance neuromorphic hardware.