Reviewerly/exHarmony huggingface dataset:
Information Extraction from Pill Bottle Images via Text Stitching

Extracting information from a sequence of images of a pill bottle captured through a phone camera can allow customers to record useful entities such as drug name, dosage, instructions etc. This information can enable customers to track their medication adherence and interact with a pharmacy to schedule refills though phone applications. However, partial images of pill bottles being text-heavy pose several challenges during image stitching. This is due to the limitations of existing image stitching algorithms, which do not consider text features present in the image. In this work, we propose an end-to-end framework for recognising entities from a sequence of images of pill bottles by using a combination of image and text features. The features are used to generate a text panorama and apply named entity recognition for information extraction. The results obtained on a dataset of pill bottle images show that the accuracy of our proposed framework significantly improves the performance of information extraction from the image sets where state-of-the-art panorama stitching method fails.
Interpretation of Neural Networks is Susceptible to Universal Adversarial Perturbations

Interpreting neural network classifiers using gradient-based saliency maps has been extensively studied in the deep learning literature. While the existing algorithms manage to achieve satisfactory performance in application to standard image recognition datasets, recent works demonstrate the vulnerability of widely-used gradient-based interpretation schemes to norm-bounded perturbations adversarially designed for every individual input sample. However, such adversarial perturbations are commonly designed using the knowledge of an input sample, and hence perform sub-optimally in application to an unknown or constantly changing data point. In this paper, we show the existence of a Universal Perturbation for Interpretation (UPI) for standard image datasets, which can alter a gradient-based feature map of neural networks over a significant fraction of test samples. To design such a UPI, we propose a gradient-based optimization method as well as a principal component analysis (PCA)-based approach to compute a UPI which can effectively alter a neural network’s gradient-based interpretation on different samples. We support the proposed UPI approaches by presenting several numerical results of their successful applications to standard image datasets.
SPADE: Self-Supervised Pretraining for Acoustic Disentanglement

Self-supervised representation learning approaches have grown in popularity due to the ability to train models on large amounts of unlabeled data and have demonstrated success in diverse fields such as natural language processing, computer vision, and speech. Previous self-supervised work in the speech domain has disentangled multiple attributes of speech such as linguistic content, speaker identity, and rhythm. In this work, we introduce a self-supervised approach to disentangle room acoustics from speech and use the acoustic representation on the downstream task of device arbitration. Our results demonstrate that our proposed approach significantly improves performance over a baseline when labeled training data is scarce, indicating that our pretraining scheme learns to encode room acoustic information while remaining invariant to other attributes of the speech signal.
Small-Footprint Slimmable Networks for Keyword Spotting

In this work, we present Slimmable Neural Networks applied to the problem of small-footprint keyword spotting. We show that slimmable neural networks allow us to create super-nets from Convolutional Neural Networks and Transformers, from which sub-networks of different sizes can be extracted. We demonstrate the usefulness of these models on in-house voice assistant data and Google Speech Commands, and focus our efforts on models for the on-device use case, limiting ourselves to less than 250k parameters. We show that slimmable models can match (and in some cases, outperform) models trained from scratch. Slimmable neural networks are therefore a class of models particularly useful when the same functionality is to be replicated at different memory and compute budgets, with different accuracy requirements.
Improving Noisy Student Training on Non-Target Domain Data for Automatic Speech Recognition

Noisy Student Training (NST) has recently demonstrated extremely strong performance in Automatic Speech Recognition (ASR). In this paper, we propose a data selection strategy named LM Filter to improve the performance of NST on non-target domain data in ASR tasks. Hypotheses with and without a Language Model are generated and the CER differences between them are utilized as a filter threshold. Results reveal that significant improvements of 10.4% compared with no data filtering baselines. We can achieve 3.31% CER in AISHELL-1 test set, which is best result from our knowledge without any other supervised data. We also perform evaluations on the supervised 1000 hour AISHELL-2 dataset and competitive results of 4.73% CER can be achieved.
TRICL: Triplet Continual Learning

A class-incremental learning agent learns online with a neverending stream of data in one training epoch. In this setting, the agent suffers from severe catastrophic forgetting due to the absence of data from the observed classes after learning data from new classes. Besides, the prototypes rapidly become outdated as the agent adapts to new data sequentially, and the previous example embeddings spread out in an unforeseen way, which exacerbates forgetting (i.e., concept drift). Based on this observation, we propose a replay-based method, called TriCL, which gathers the embeddings near the prototype from the same class and separates the embeddings from the different class prototypes. TriCL leverages an improved triplet loss without extra arranged input data triplets. To facilitate rapid convergence between the same class samples, we design a memory update algorithm for decreasing the variance among the buffered samples from the same class. Furthermore, we make a prototype compensation strategy for preventing drift. Compared to the state-of-the-art benchmarks, the experiments demonstrate that our proposed method presents improved performance in the online class-incremental learning setting.
Efficient Uncertainty Estimation with Gaussian Process for Reliable Dialog Response Retrieval

Deep neural networks have achieved remarkable performance in retrieval-based dialogue systems, but they are shown to be ill calibrated. Though basic calibration methods like Monte Carlo Dropout and Ensemble can calibrate well, these methods are time-consuming in the training or inference stages. To tackle these challenges, we propose an efficient uncertainty calibration framework GPF-BERT for BERT-based conversational search, which employs a Gaussian Process layer and the focal loss on top of the BERT architecture to achieve a high-quality neural ranker. Extensive experiments are conducted to verify the effectiveness of our method. In comparison with basic calibration methods, GPF-BERT achieves the lowest empirical calibration error (ECE) in three in-domain datasets and the distributional shift tasks, while yielding the highest R <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">10</inf> @1 and MAP performance on most cases. In terms of time consumption, our GPF-BERT has an 8× speedup.
SemGeo: Semantic Keywords for Cross-View Image Geo-Localization

Image geo-localization refers to determining location of images using visual information. In cross-view image geo-localization, location of images are identified by matching them against a database of geo-tagged aerial imagery. The methods based on this approach rely solely on color statistics to resolve view point changes. This has seemed to work well for large Field of View (FoV) images, but leads to poor results in low FoV setups. Images captured from standard cameras have small FoVs, these devices cannot capture images with FoV larger than 180°. Exploring information other than color statistics is important to enable geo-localization for them. We present SemGeo, a ground to aerial cross-view image geo-localization framework that considers semantic keywords from images while geo-localizing them. Experiments on standard benchmarks demonstrates that the proposed approach shows impressive gains on FoV constrained image geo-localization improving top-1% image recall rate on the CVUSA data-set from 89.18% to 94.40% and from 85.60% to 91.86% on the CVACT benchmark for 90°FoV.
Tree-Like Interaction Learning for Bundle Recommendation

Bundle recommendation suggests a set of items to users against their complex needs, where user-bundle interaction learning is key. It is observed that Gromov’s δ-hyperbolicity of the interaction graph in bundle recommendation is smaller (lower is more hyperbolic) than those in traditional item recommendation when measuring a graph’s tree likeness. However, state-of-the-art bundle recommendation methods learn to embed the entities (user, bundle, item) of tree-like interaction graph in Euclidean space, which could cause severe distortion problems. We argue hyperbolic space provides a promising way to get accurate entity embeddings, with this paper proposing a novel bundle recommendation model. The model learns user preferences via hyperbolic graph convolution, aiming at decreasing the distortion of bundle graph node embeddings. Extensive empirical experiments conducted on two real-world datasets confirm that our model achieves promising performance compared to baseline methods representing state-of-the-art bundle recommendation methods.
Distortion-Aware Convolutional Neural Network-Based Interpolation Filter for AVS3

Motion compensation is a key technology in video coding for removing the temporal redundancy between video frames. Considering the incompatibility between traditional interpolation filters and diversified video content, the inter prediction method still has considerable room for improvement. This paper proposed a distortion-aware convolutional neural network-based interpolation filter (DA-NNIF) to further improve the interpolation prediction accuracy of sub-pixels with one model. Distortion parameters are introduced into the proposed network to reflect the quantization noise of reference frames. The experimental result shows that the proposed method achieves on average 1.47 % BD-rate reduction on Y component for ClassB, ClassC and ClassD sequences under the random access configuration of AVS3.
BHE-DARTS: Bilevel Optimization Based on Hypergradient Estimation for Differentiable Architecture Search

In this paper, we propose a stochastic bilevel optimization approach based on a hypergradient estimator, called BHE- DARTS, as a remedy for this issue that it is easy to search for locally optimal structures rather than globally optimal ones in Differentiable Architecture Search (DARTS) bilevel optimization model. To be specific, we apply a stochastic gradient for updating the lower level variable ω and design a hypergradient estimator, which is built by the Jacobian- and Hessianvector product, to assist in updating the upper level variable α. This operation can more fully apply the gradient information to escape the trap of local optimal in the NAS bilevel model. Compared to state-of-the-art DARTS methods, experimental studies have shown the competitive performance of the proposed BHE-DARTS in the DARTS search space (CIFAR-100: a test accuracy rate of 82.69 % ) and NAS-Bench-201 search space (ImageNet16-120: a test accuracy rate of 42.44%).
Improving Prosody for Cross-Speaker Style Transfer by Semi-Supervised Style Extractor and Hierarchical Modeling in Speech Synthesis

Cross-speaker style transfer in speech synthesis aims at transferring a style from source speaker to synthesized speech of a target speaker’s timbre. In most previous methods, the synthesized fine-grained prosody features often represent the source speaker’s average style, similar to the one-to-many problem(i.e., multiple prosody variations correspond to the same text). In response to this problem, a strength-controlled semi-supervised style extractor is proposed to disentangle the style from content and timbre, improving the representation and interpretability of the global style embedding, which can alleviate the one-to-many mapping and data imbalance problems in prosody prediction. A hierarchical prosody predictor is proposed to improve prosody modeling. We find that better style transfer can be achieved by using the source speaker’s prosody features that are easily predicted. Additionally, a speaker-transfer-wise cycle consistency loss is proposed to assist the model in learning unseen style-timbre combinations during the training phase. Experimental results show that the method outperforms the baseline. We provide a website with audio samples <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Wordreg: Mitigating the Gap between Training and Inference with Worst-Case Drop Regularization

Dropout has emerged as one of the most frequently used techniques for training deep neural networks (DNNs). Although effective, the sampled sub-model by random dropout during training is inconsistent with the full model (without dropout) during inference. To mitigate this undesirable gap, we propose WordReg, a simple yet effective regularization built on dropout that enforces the consistency between the outputs of different sub-models sampled by dropout. Specifically, WordReg first obtains the worst-case dropout by maximizing the divergence between the outputs with two sub-models with different random dropouts. And then, it encourages the agreements between the outputs of the two sub-models with worstcase divergence. Extensive experiments on diverse DNNs and tasks reveal that WordReg can achieve notable and consistent improvements over non-regularized models and yields some state-of-the-art results. Theoretically, we verify that WordReg can reduce the gap between training and inference.
Retrieval-Based Natural 3D Human Motion Generation

It is challenging to generate 3D human motions automatically from text. In an ideal scenario, the generated motions should explore the text-grounded motion space while accurately depicting the content in the prescribed text descriptions. Text2length and text2motion training have been used to address this problem in previous research. There is, however, a lack of knowledge about the relationship between motion length and text. In this work, context-aware retrieval-based approaches are proposed for predicting motion lengths and generating proper 3D motions (C-MO). Specifically, we train a context-aware encoder-decoder model that uses the previous output of the decoder or the embedding vector of a ground truth motion as context so that the model becomes increasingly aware of previous alignments. Retrieving the most similar motions from the training set is based on the trained model given a text. Finally, we use the retrieval motion to guide the probability distribution for the final generated motions. Our method combines the advantages of both information retrieval and neural machine translation. C-MO is evaluated on a large-scale dataset, KIT, and its experimental results show that it achieves great improvements over the state-of-the-art.
Global Localisation in Continuous Magnetic Vector Fields Using Gaussian Processes

Localisation is one of the key capabilities for autonomous robots with sensors. Magnetic sensors to perceive the environment, although less explored, are an alternative modality to aid localisation. This paper proposes the use of continuous vector fields provided by a Gaussian Process (GP) with a divergence-free kernel that follows the magnetic flux to localise a mobile robot moving in a 2D space. The environment is pre-mapped in 3D producing a magnetic vector field with a GP that can be queried at any location. By means of a particle filter for Monte Carlo localisation, the mobile robot can be globally localised in the environment. We validate our approach using simulations and experimental results. Complex simulated environments using ANSYS are exploited to show our approach outperforms commonly used kernels.
On Designing A 3d Imaging Summer Project For Ontario’s High School Students During Covid-19 Pandemic

During the Covid-19 pandemic, like the vast majority of countries in the world, Canada was under government-mandated lockdown, creating unprecedented challenges for the higher education system. This has exacerbated the problem of gender and ethnic inequalities in the STEM field due to the sudden disappearance of in-person communication and communities that had supported minority groups. To provide emergency support and reduce the known gender / ethnic gap, at York University in Toronto we designed a 3D imaging project for Ontario’s high school (HS) students, as part of an annual summer outreach program in the Lassonde School of Engineering. The project aims to create an equitable opportunity for HS students, providing a comprehensive introduction to image processing through experiential learning. We document our design methodology and experiences in the project, as well as feedback and evaluations from participants at all levels. We believe such documentation is valuable to promote gender- and ethnic-balanced education in image processing and the broader STEM field in the future, in an increasingly unpredictable environment due to climate change.
Toroidal Probabilistic Spherical Discriminant Analysis

In speaker recognition, where speech segments are mapped to embeddings on the unit hypersphere, two scoring back-ends are commonly used, namely cosine scoring and PLDA. We have recently proposed PSDA, an analog to PLDA that uses Von Mises-Fisher distributions instead of Gaussians. In this paper, we present toroidal PSDA (T-PSDA). It extends PSDA with the ability to model within and between-speaker variabilities in toroidal submanifolds of the hypersphere. Like PLDA and PSDA, the model allows closed-form scoring and closed-form EM updates for training. On VoxCeleb, we find T-PSDA accu-racy on par with cosine scoring, while PLDA accuracy is inferior. On NIST SRE’21 we find that T-PSDA gives large accuracy gains compared to both cosine scoring and PLDA. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
A Comprehensive Comparison of Projections in Omnidirectional Super-Resolution

Super-Resolution (SR) has gained increasing research attention over the past few years. With the development of Deep Neural Networks (DNNs), many super-resolution methods based on DNNs have been proposed. Although most of these methods are aimed at ordinary frames, there are few works on super-resolution of omnidirectional frames. In these works, omnidirectional frames are projected from the 3D sphere to a 2D plane by Equi-Rectangular Projection (ERP). Although ERP has been widely used for projection, it has severe projection distortion near poles. Current DNN-based SR methods use 2D convolution modules, which is more suitable for the regular grid. In this paper, we find that different projection methods have great impact on the performance of DNNs. To study this problem, a comprehensive comparison of projections in omnidirectional super-resolution is conducted. We compare the SR results of different projection methods. Experimental results show that Equi-Angular cube map projection (EAC), which has minimal distortion, achieves the best result in terms of WS-PSNR compared with other projections. Code and data will be released.
An Isotropy Analysis for Self-Supervised Acoustic Unit Embeddings on the Zero Resource Speech Challenge 2021 Framework

In recent years, self-supervised representation learning has gained much attention for its proven advantages in many downstream tasks. Consequently, various self-supervised representation learning methods have been developed. However, few studies have investigated the resulting embedding space or analyzed why any particular approach performs better than any other. Here, we are interested in investigating the geometry in terms of the isotropy of embedding spaces learned by self-supervised speech representation, which can influence the performance in discriminating acoustic units on the Zero Resource Speech Challenge 2021 (ZR2021) framework. Most top systems from the published ZR2021 results are based on the contrastive predictive coding (CPC) technique. Here, we propose using hidden-unit BERT (HuBERT) self-supervised representation learning, and we provide detailed analyses and comparisons of their isotropies of embedding space, which might influence performance. Furthermore, we use simple yet effective feature fusion techniques to combine both models’ strengths, leading to the ability to reduce the ABX error rate and outperform top models in the ZR2021 dev-other dataset.
Shadow Removal of Text Document Images Using Background Estimation and Adaptive Text Enhancement

This paper proposes a simple yet effective method to re-move shadows from text document images. It mainly includes several parts. Firstly, we propose a text elimination-based background extraction strategy to estimate shadow map. It indicates the shadow regions accurately and helps to predict global background. Secondly, a binarization-based text ex-traction algorithm is designed to obtain texts from document image. By fusing texts and global background, a preparatory shadow-free image can be obtained. Thirdly, we propose an adaptive text contrast enhancement strategy to generate shadow-free results with comfortable visual perception across shadow and non-shadow regions. Quantitative and visual results performed on open datasets indicate that the proposed method can generate clear shadow-free images from text document images. Our code will be publicly available soon.
Scalable Weight Reparametrization for Efficient Transfer Learning

This paper proposes a novel, efficient transfer learning method, called Scalable Weight Reparametrization (SWR) that is efficient and effective for multiple downstream tasks. Efficient transfer learning involves utilizing a pre-trained model trained on a larger dataset and repurposing it for downstream tasks with the aim of maximizing the reuse of the pre-trained model. However, previous works have led to an increase in updated parameters and task-specific modules, resulting in more computations, especially for tiny models. Additionally, there has been no practical consideration for controlling the number of updated parameters. To address these issues, we suggest learning a policy network that can decide where to reparametrize the pre-trained model, while adhering to a given constraint for the number of updated parameters. The policy network is only used during the transfer learning process and not afterward. As a result, our approach attains state-of-the-art performance in a proposed multi-lingual keyword spotting and a standard benchmark, ImageNet-to-Sketch, while requiring zero additional computations and significantly fewer additional parameters.