Reviewerly/exHarmony huggingface dataset:
WePerson: Learning a Generalized Re-identification Model from All-weather Virtual Data

The aim of person re-identification (Re-ID) is retrieving a person of interest across multiple non-overlapping cameras. Re-ID has gained significantly increased advancement in recent years. However, real data annotation is costly and model generalization ability is hindered by the lack of large-scale and diverse data. To address this problem, we propose a Weather Person pipeline that can generate a synthesized Re-ID dataset with different weather, scenes, and natural lighting conditions automatically. The pipeline is built on the top of a game engine which contains a digital city, weather and lighting simulation system, and various character models with manifold dressing. To train a generalizable Re-ID model from the large-scale virtual WePerson dataset, we design an adaptive sample selection strategy to close the domain gap and avoid redundancy. We also design an informative sampling method for a mini-batch sampler to accelerate the learning process. In addition, an efficient training method is introduced by adopting instance normalization to capture identity invariant components from various appearances. We evaluate our pipeline using direct transfer on 3 widely-used real-world benchmarks, achieving competitive performance without any real-world image training. This dataset starts the attempt to evaluate diverse environmental factors in a controllable virtual engine, which provides important guidance for future generalizable Re-ID model design. Notably, we improve the current state-of-the-art accuracy from 38.5% to 46.4% on the challenging MSMT17 dataset. Dataset and code are available at https://github.com/lihe404/WePerson https://github.com/lihe404/WePerson.
Phone-to-Audio Alignment without Text: A Semi-Supervised Approach

The task of phone-to-audio alignment has many applications in speech research. Here we introduce two Wav2Vec2-based models for both text-dependent and text-independent phone-to-audio alignment. The proposed Wav2Vec2-FS, a semi-supervised model, directly learns phone-to-audio alignment through contrastive learning and a forward sum loss, and can be coupled with a pretrained phone recognizer to achieve text-independent alignment. The other model, Wav2Vec2-FC, is a frame classification model trained on forced aligned labels that can both perform forced alignment and text-independent segmentation. Evaluation results suggest that both proposed methods, even when transcriptions are not available, generate highly close results to existing forced alignment tools. Our work presents a neural pipeline of fully automated phone-to-audio alignment. Code and pretrained models are available at https://github.com/lingjzhu/charsiu.
Multiple Object Tracking by Trajectory Map Regression with Temporal Priors Embedding

Prevailing Multiple Object Tracking (MOT) works following the Tracking-by-Detection (TBD) paradigm pay most attention to either object detection in a first step or data association in a second step. In this paper, we approach the MOT problem from a different perspective by directly obtaining the embedded spatial-temporal information of trajectories from raw video data. For the purpose we propose a joint trajectory locating and attributes encoding framework for real-time, on-line MOT. We firstly introduce a trajectory attribute representation scheme designed for each tracked target (instead of object) where the extracted Trajectory Map (TM) encodes the spatial-temporal attributes of a trajectory across a window of consecutive video frames. Next we present a Temporal Priors Embedding (TPE) methodology to infer these attributes with a logical reasoning strategy based on long-term feature dynamics. The proposed MOT framework projects multiple attributes of tracked targets, e.g., presence, enter/exit, location, scale, motion, etc. into a continuous TM to perform one-shot regression for real-time MOT. Experimental results show that, our proposed video-based method runs at 33 FPS and is more accurate and robust as compared to the detection-based tracking methods and a few other State-of-the- Art (SOTA) approaches on MOT16/17/20 benchmarks.
Qubit-efficient encoding scheme for quantum simulations of electronic structure

Simulating electronic structure on a quantum computer requires encoding of fermionic systems onto qubits. Common encoding methods transform a fermionic system of $N$ spin-orbitals into an $N$-qubit system, but many of the fermionic configurations do not respect the required conditions and symmetries of the system so the qubit Hilbert space in this case may have unphysical states and thus can not be fully utilized. We propose a generalized qubit-efficient encoding (QEE) scheme that requires the qubit number to be only logarithmic in the number of configurations that satisfy the required conditions and symmetries. For the case of considering only the particle-conserving and singlet configurations, we reduce the qubit count to an upper bound of $\mathcal O(m\log_2N)$, where $m$ is the number of particles. This QEE scheme is demonstrated on an H$_2$ molecule in the 6-31G basis set and a LiH molecule in the STO-3G basis set using fewer qubits than the common encoding methods. We calculate the ground-state energy surfaces using a variational quantum eigensolver algorithm with a hardware-efficient ansatz circuit. We choose to use a hardware-efficient ansatz since most of the Hilbert space in our scheme is spanned by desired configurations so a heuristic search for an eigenstate is sensible. The simulations are performed on IBM Quantum machines and the Qiskit simulator with a noise model implemented from a IBM Quantum machine. Using the methods of measurement error mitigation and error-free linear extrapolation, we demonstrate that most of the distributions of the extrapolated energies using our QEE scheme agree with the exact results obtained by Hamiltonian diagonalization in the given basis sets within chemical accuracy. Our proposed scheme and results show the feasibility of quantum simulations for larger molecular systems in the noisy intermediate-scale quantum (NISQ) era.
Yes, "Attention Is All You Need", for Exemplar based Colorization

Conventional exemplar based image colorization tends to transfer colors from reference image only to grayscale image based on the semantic correspondence between them. But their practical capabilities are limited when semantic correspondence can hardly be found. To overcome this issue, additional information, such as colors from the database is normally introduced. However, it's a great challenge to consider color information from reference image and database simultaneously because there lacks a unified framework to model different color information and the multi-modal ambiguity in database cannot be removed easily. Also, it is difficult to fuse different color information effectively. Thus, a general attention based colorization framework is proposed in this work, where the color histogram of reference image is adopted as a prior to eliminate the ambiguity in database. Moreover, a sparse loss is designed to guarantee the success of information fusion. Both qualitative and quantitative experimental results show that the proposed approach achieves better colorization performance compared with the state-of-the-art methods on public databases with different quality metrics.
Remember and Reuse

Existing blind image quality assessment (BIQA) methods have made great progress in various task-specific applications, including the synthetic, authentic, or over-enhanced distortion evaluations. However, limited by the static model and once-for-all learning strategy, they failed to perform the cross-task evaluations in many practical applications, where diverse evaluation criteria and distortion types are constantly emerging. To address this issue, in this paper, we propose a dynamic Remember and Reuse (R&R) network, which efficiently performs the cross-task BIQA based on a novel relevance-aware incremental learning strategy. Given multiple evaluation tasks across different distortion types or databases, our R&R network sequentially updates the parameters for every task one by one. After each update step, part of task-specific parameters is settled, which ensures R&R Remembers their dedicated evaluation preferences. The remaining parameters are pruned for the dynamic usage of the subsequent tasks. To further exploit the correlation between different tasks, we feed the training data of a new task to previously settled parameters. Better prediction accuracy is considered as higher task relevance and vice versa. Then, we selectively Reuse parts of previously settled parameters, whose proportion is adaptively determined by the task relevance. Extensive experiments show that the proposed method efficiently achieves the cross-task BIQA without catastrophic forgetting, and significantly outperforms many state-of-the-art methods. Code is available at https://github.com/maruiperfect/R-R-Net.
Conversational Group Detection with Graph Neural Networks

We study conversational group detection in varied social scenes using a message-passing Graph Neural Network (GNN) in combination with the Dominant Sets clustering algorithm. Our approach first describes a scene as an interaction graph, where nodes encode individual features and edges encode pairwise relationship data. Then, it uses a GNN to predict pairwise affinity values that represent the likelihood of two people interacting together, and computes non-overlapping group assignments based on these affinities. We evaluate the proposed approach on the Cocktail Party and MatchNMingle datasets. Our results suggest that using GNNs to leverage both individual and relationship features when computing groups is beneficial, especially when more features are available for each individual.
A unified deep sparse graph attention network for scene graph generation

Scene graph generation (SGG) plays an important role in deep understanding of the visual scene. Despite the empirical success of traditional methods in many applications, they still have several challenges in the high computational complexity of dense graph and the inaccurate pruning of sparse graph. To tackle these problems, we propose a novel deep sparse graph attention network to mine the rich contextual clues and simultaneously preserve the statistical co-occurrence knowledge of SGG. Specifically, our Relationship Measurement Network (RelMN) is adapted to first classify all object pairs in dense graph as the foreground and background categories to filter the false relationships and then construct a sparse graph efficiently. Meanwhile, we design a novel feature aggregation and update method via graphical message passing to jointly learn the node and edge features for object recognition and relationship classification in the graph attention network. Extensive experimental results on the large scale VG and VRD datasets demonstrate our proposed method outperforms several state-of-the-art approaches.
A novel equilibrium optimizer based on levy flight and iterative cosine operator for engineering optimization problems

Abstract Equilibrium optimizer (EO) is a novel optimization algorithm with high exploration and exploitation capabilities. The capabilities of EO are impacted by the generation rate, exponential term, and equilibrium pool, wherein the initial two depend on the turnover rate. The performance of the EO on different optimization functions can be improved by updating these factors. This paper designs a modified equilibrium optimizer (MEO) by incorporating three changes in the EO: (i) replacement of the equilibrium pool with an iterative cosine operator (ICO) that gradually reduces diversification to intensification as the iterations progress; (ii) implementation of levy flight to update the concentration of particle that improves exploration capabilities; and (iii) the random vector with uniform distribution of turnover rate is replaced by heavy‐tailed non‐uniform levy distribution to improve exploration capability of the algorithm. Here, the other parameters on which MEO depends are selected by performing the sensitivity analysis. The capabilities of MEO have been analysed by comparing it with 19 algorithms, including the EO algorithm, 12 state‐of‐art algorithms, and 6 hybrid/improved algorithms on 62 functions, that is, 23 benchmark functions, 10 CEC‐06‐2019, 29 CEC‐2017 functions. The non‐parametric Friedman test and Kruskal–Wallis test validate that MEO outperforms the other algorithms due to its balanced exploration and exploitation abilities. MEO robustness has been validated by the diversity analysis along with the scalability test. MEO is also evaluated on five practical engineering problems to showcase its significance.
Snowy: Recommending Utterances for Conversational Visual Analysis

Natural language interfaces (NLIs) have become a prevalent medium for conducting visual data analysis, enabling people with varying levels of analytic experience to ask questions of and interact with their data. While there have been notable improvements with respect to language understanding capabilities in these systems, fundamental user experience and interaction challenges including the lack of analytic guidance (i.e., knowing what aspects of the data to consider) and discoverability of natural language input (i.e., knowing how to phrase input utterances) persist. To address these challenges, we investigate utterance recommendations that contextually provide analytic guidance by suggesting data features (e.g., attributes, values, trends) while implicitly making users aware of the types of phrasings that an NLI supports. We present Snowy, a prototype system that generates and recommends utterances for visual analysis based on a combination of data interestingness metrics and language pragmatics. Through a preliminary user study, we found that utterance recommendations in Snowy support conversational visual analysis by guiding the participants’ analytic workflows and making them aware of the system’s language interpretation capabilities. Based on the feedback and observations from the study, we discuss potential implications and considerations for incorporating recommendations in future NLIs for visual analysis.
Recommender Systems And Digital Storytelling To Enhance Tourism Experience In Cultural Heritage Sites

The world of Cultural Heritage finds significant ad-vantages from fusion with new technologies. For example, the use of Recommender Systems to analyze contextual information and the Digital Storytelling technique allows improving the experience of users who get in touch with artistic and cultural heritage. This paper aims to describe a new approach that suggests cultural-touristic paths exploit recommendation techniques and propose multimedia content. Moreover, the proposed approach aims to provide recommendations when ratings are unknown, using a novel approach that takes advantage of the user and item profile knowledge. The proposed approach has been tested through an application prototype. The test involved standard and expert users of the University of Salerno with promising results.
Neighbor-Vote

As cameras are increasingly deployed in new application domains such as autonomous driving, performing 3D object detection on monocular images becomes an important task for visual scene understanding. Recent advances on monocular 3D object detection mainly rely on the "pseudo-LiDAR'' generation, which performs monocular depth estimation and lifts the 2D pixels to pseudo 3D points. However, depth estimation from monocular images, due to its poor accuracy, leads to inevitable position shift of pseudo-LiDAR points within the object. Therefore, the predicted bounding boxes may suffer from inaccurate location and deformed shape. In this paper, we present a novel neighbor-voting method that incorporates neighbor predictions to ameliorate object detection from severely deformed pseudo-LiDAR point clouds. Specifically, each feature point around the object forms their own predictions, and then the "consensus'' is achieved through voting. In this way, we can effectively combine the neighbors' predictions with local prediction and achieve more accurate 3D detection. To further enlarge the difference between the foreground region of interest (ROI) pseudo-LiDAR points and the background points, we also encode the ROI prediction scores of 2D foreground pixels into the corresponding pseudo-LiDAR points. We conduct extensive experiments on the KITTI benchmark to validate the merits of our proposed method. Our results on the bird's eye view detection outperform the state-of-the-art performance, especially for the "hard" level detection. The code is available at https://github.com/cxmomo/Neighbor-Vote.
Dynamic Binary Neural Network by Learning Channel-Wise Thresholds

Binary neural networks (BNNs) constrain weights and activations to +1 or -1 with limited storage and computational cost, which is hardware-friendly for portable devices. Recently, BNNs have achieved remarkable progress and been adopted into various fields. However, the performance of BNNs is sensitive to activation distribution. The existing BNNs utilized the Sign function with predefined or learned static thresholds to binarize activations. This process limits representation capacity of BNNs since different samples may adapt to unequal thresholds. To address this problem, we propose a dynamic BNN (DyBNN) incorporating dynamic learnable channel-wise thresholds of Sign function and shift parameters of PReLU. The method aggregates the global information into the hyper function and effectively increases the feature expression ability. The experimental results prove that our method is an effective and straightforward way to reduce information loss and enhance performance of BNNs. The DyBNN based on two backbones of ReActNet (MobileNetV1 and ResNet18) achieve 71.2% and 67.4% top1-accuracy on ImageNet dataset, outperforming baselines by a large margin (i.e., 1.8% and 1.5% respectively).
Augmented reality instructions for construction toys enabled by accurate model registration and realistic object/hand occlusions

BRICKxAR is a novel augmented reality (AR) instruction method for construction toys such as LEGO®. With BRICKxAR, physical LEGO construction is guided by virtual bricks. Compared with the state of the art, accuracy of the virtual-physical model alignment is significantly improved through a new design of marker-based registration, which can achieve an average error less than 1 mm throughout the model. Realistic object occlusion is accomplished to reveal the true spatial relationship between physical and virtual bricks. LEGO players' hand detection and occlusion are realized to visualize the correct spatial relationship between real hands and virtual bricks, and allow virtual bricks to be "grasped" by real hands. The major finding of the research is that the integration of these features makes AR instructions possible for small parts assembly, validated through a working AR prototype for constructing LEGO Arc de Triomphe and quantitative measures of the accuracies of registration and occlusions. In addition, a heuristic evaluation of BRICKxAR's features has led to findings that the present method could advance AR instructions in terms of enhancing part visibility, match between mental models and visualization, alignment of physical and virtual parts in perspective views and spatial transformations, tangible user interface, consolidated structural diagrams, virtual cutaway views, among other benefits for guiding construction.The online version contains supplementary material available at 10.1007/s10055-021-00582-7.
Domain Generalization via Feature Variation Decorrelation

Domain generalization aims to learn a model that generalizes to unseen target domains from multiple source domains. Various approaches have been proposed to address this problem by adversarial learning, meta-learning, and data augmentation. However, those methods have no guarantee for target domain generalization. Motivated by an observation that the class-irrelevant information of sample in the form of semantic variation would lead to negative transfer, we propose to linearly disentangle the variation out of sample in feature space and impose a novel class decorrelation regularization on the feature variation. By doing so, the model would focus on the high-level categorical concept for model prediction while ignoring the misleading clue from other variations (including domain changes). As a result, we achieve state-of-the-art performances over all of widely used domain generalization benchmarks, namely PACS, VLCS, Office-Home, and Digits-DG with large margins. Further analysis reveals our method could learn a better domain-invariant representation, and decorrelated feature variation could successfully capture semantic meaning.
T-count and T-depth of any multi-qubit unitary

While implementing a quantum algorithm it is crucial to reduce the quantum resources, in order to obtain the desired computational advantage. For most fault-tolerant quantum error-correcting codes the cost of implementing the non-Clifford gate is the highest among all the gates in a universal fault-tolerant gate set. In this paper we design provable algorithm to determine T-count of any $n$-qubit ($n\geq 1$) unitary $W$ of size $2^n\times 2^n$, over the Clifford+T gate set. The space and time complexity of our algorithm are $O\left(2^{2n}\right)$ and $O\left(2^{2n\mathcal{T}_{\epsilon}(W)+4n}\right)$ respectively. $\mathcal{T}_{\epsilon}(W)$ ($\epsilon$-T-count) is the (minimum possible) T-count of an exactly implementable unitary $U$ i.e. $\mathcal{T}(U)$, such that $d(U,W)\leq\epsilon$ and $\mathcal{T}(U)\leq\mathcal{T}(U')$ where $U'$ is any exactly implementable unitary with $d(U',W)\leq\epsilon$. $d(.,.)$ is the global phase invariant distance. Our algorithm can also be used to determine the (minimum possible) T-depth of any multi-qubit unitary and the complexity has exponential dependence on $n$ and $\epsilon$-T-depth. This is the first algorithm that gives T-count or T-depth of any multi-qubit ($n\geq 1$) unitary. For small enough $\epsilon$, we can synthesize the T-count and T-depth-optimal circuits. Our results can be used to determine the minimum count (or depth) of non-Clifford gates required to implement any multi-qubit unitary with a universal gate set consisting of Clifford and non-Clifford gates like Clifford+CS, Clifford+V, etc. To the best of our knowledge, there were no such optimal-synthesis algorithm for arbitrary multi-qubit unitaries in any universal gate set.