Reviewerly/exHarmony huggingface dataset:
Inorganic geochemistry of saline substrates associated with coastal turf, Otago Peninsula, New Zealand

ABSTRACTABSTRACTHalophytic plant communities have developed on exposed coastal landforms to form low-stature turf in several locations around New Zealand. We quantify the inorganic geochemistry, mineralogy and simulated ephemeral surface water composition of saline soil-free bare ground at a well-preserved site at Maori Head on Otago Peninsula, where halophytes have colonised from adjacent coastal turf. The community at this site is dominated by Leptinella dioica, Samolus repens, Disphyma australe and Selliera radicans. Maori Head substrates are dominated by clay-altered basalt colluvium and show a general trend from high electrical conductivity (EC > 10 mS/cm) and weakly acid pH (< 6) on bare ground to low EC (< 1 mS cm−1) and higher pH (> 7) in turf and adjacent pasture soils. High salinities on dry substrates and in simulated ephemeral surface waters result from surface evaporation of airborne sea spray and contain halite (NaCl) with some gypsum (CaSO4.2H2O). Chemical interactions with the substrate are minor. The high surface salinities were maintained despite a major winter rain event with strong onshore winds. However, surface salinity is frequently leached by rain and shallow groundwater, whereas inland saline sites (pH 8–11) have substantial salt accumulations driven by lower rainfall, high evaporation and chemical interaction with Otago Schist-derived substrates.KEYWORDS: Salthalitehalophyteelectrical conductivitypHbasaltschistevaporation AcknowledgementsLogistical support and data archiving by the Queen Elizabeth II National Trust were instrumental in developing this research project. Access to the Maori Head site was generously provided by the landowners. Discussions with Clement Lagrue (Department of Conservation) and Scott Jarvie (Otago Regional Council) assisted in developing ideas about links between inland and coastal saline sites. Constructive comments from two journal referees and the handling editor substantially improved the manuscript.Disclosure statementNo potential conflict of interest was reported by the author(s).Data availabilityAll data relevant to this study are included within this paper and within cited references.Additional informationFundingFunding for this research was provided by the University of Otago.
Uncertainty quantification for deep neural networks: An empirical comparison and usage guidelines

Summary Deep neural networks (DNN) are increasingly used as components of larger software systems that need to process complex data, such as images, written texts, audio/video signals. DNN predictions cannot be assumed to be always correct for several reasons, amongst which the huge input space that is dealt with, the ambiguity of some inputs data, as well as the intrinsic properties of learning algorithms, which can provide only statistical warranties. Hence, developers have to cope with some residual error probability. An architectural pattern commonly adopted to manage failure prone components is the supervisor , an additional component that can estimate the reliability of the predictions made by untrusted (e.g., DNN) components and can activate an automated healing procedure when these are likely to fail, ensuring that the deep learning‐based system (DLS) does not cause damages, despite its main functionality being suspended. In this paper, we consider DLS that implement a supervisor by means of uncertainty estimation. After overviewing the main approaches to uncertainty estimation and discussing their pros and cons, we motivate the need for a specific empirical assessment method that can deal with the experimental setting in which supervisors are used, where accuracy of the DNN matters only as long as the supervisor lets the DLS continue to operate. Then we present a large empirical study conducted to compare the alternative approaches to uncertainty estimation. We distilled a set of guidelines for developers that are useful to incorporate a supervisor based on uncertainty monitoring into a DLS.
Dual-branche attention network for super-resolution of remote sensing images

Remote sensing (RS) images are considered to be reflections of the real world. However, RS images often suffered from low resolution, making further research difficult to follow. Although super resolution (SR) techniques based on deep learning have achieved considerable breakthroughs, they show limited performance when dealing with low-quality RS images with complicated backgrounds; for instance, the SR results tend to loss details and have undesired structural distortion. Thus, this paper proposes an innovative dual-branch attention network (DBAN) to produce sufficient details and preserve clear structural information for SR results of RS images. It consists of two components: a feature extraction branch and a high-frequency information learning branch. The features extraction branch, formed as a densely residual structure, combines a series of dual attention blocks that are designed to exploit valid features from different dimensions, and then all these multi-scale features are reused through a global concatenation. The high-frequency information extraction branch, incorporating noise removing units (NRU) and high-frequency attention units (HFU), is responsible for producing the high-frequency features without noise, which enables DBAN to handle the problem of structural distortion. Meanwhile, a composite loss function based on a Laplacian pyramid is proposed to maximize the structural similarity between reconstruction results and real high-resolution RS images. The proposed network is efficient and lightweight because of its strong and effective attention to feature learning. Experimental results on three open-source RS image datasets and the JiLin-01 dataset demonstrate the effectiveness of our DBAN where higher accuracy over state-of-the-art methods in super-resolving complicated images was achieved.
Investigating In-Situ Personal Health Data Queries on Smartwatches

Smartwatches enable not only the continuous collection of but also ubiquitous access to personal health data. However, exploring this data in-situ on a smartwatch is often reserved for singular and generic metrics, without the capacity for further insight. To address our limited knowledge surrounding smartwatch data exploration needs, we collect and characterize desired personal health data queries from smartwatch users. We conducted a week-long study (N = 18), providing participants with an application for recording responses that contain their query and current activity related information, throughout their daily lives. From the responses, we curated a dataset of 205 natural language queries. Upon analysis, we highlight a new preemptive and proactive data insight category, an activity-based lens for data exploration, and see the desired use of a smartwatch for data exploration throughout daily life. To aid in future research and the development of smartwatch health applications, we contribute the dataset and discuss implications of our findings.
Structural Identity Representation Learning for Blockchain-Enabled Metaverse Based on Complex Network Analysis

The metaverse and its underlying blockchain technology have attracted extensive attention in the past few years. How to mine, process, and analyze the tremendous data generated by the metaverse systems has posed a number of challenges. Aiming to address them, we mainly focus on modeling and understanding the blockchain transaction network from a structural identity perspective, which represents the entire network structure and reveals the relations among multiple entities. In this article, we analyze three metaverse-related systems: non-fungible token (NFT), Ethereum (ETH), and Bitcoin (BTC) from the structural-identity perspective. First, we conduct the complex network analysis of the metaverse network and obtain several new insights (i.e., power-law degree distribution, disconnection, disassortativity, preferential attachment, and non-rich-club effect). Secondly, based on such findings, we propose a novel representation learning method named structure-to-vector with random pace (SVRP) for learning both the latent representation and structural identity of the network. Thirdly, we conduct node classification and link prediction tasks with the integration of graph neural networks (GNNs). Empirical results on three real-world datasets demonstrate that our proposed SVRP outperforms other existing methods in multiple tasks. In particular, our SVRP achieves the highest node classification accuracy (Acc) (99.3 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\%$</tex-math> </inline-formula> ) and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$F$</tex-math> </inline-formula> 1-score (96.7 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\%$</tex-math> </inline-formula> ) while only requiring original non-attributed graphs.
A Levenshtein distance-based method for word segmentation in corpus augmentation of geoscience texts

Formulae display:?Mathematical formulae have been encoded as MathML and are displayed in this HTML version using MathJax in order to improve their display. Uncheck the box to turn MathJax off. This feature requires Javascript. Click on a formula to zoom.
Creating Effective Marketing Messages Through Moderately Surprising Syntax

Language is critical to the effectiveness of marketing messages. Achieving a desired outcome requires arranging words to formulate a message (i.e., syntax), but this task is not trivial. The authors study the role of syntax in marketing communications by focusing on syntactic surprise (i.e., how unexpected the syntax of a message is). They introduce a measure that captures syntactic surprise, establishes its internal and external validity, and tests its effectiveness for marketing messages. In a series of studies that include field data and randomized field experiments from contexts such as donations, advertising, and product reviews, the authors show that a message's syntactic surprise is related to its effectiveness. This relationship follows an inverted U-shape, such that medium-syntactic-surprise messages are the most effective. The authors then conduct experiments on Facebook and Instagram to demonstrate how these findings can be used to write effective marketing messages. In collaboration with two independent companies, they show that ads for products, services, or jobs that are written with medium syntactic surprise result in higher click-through rates than ads written with low or high syntactic surprise.
Non-Autoregressive End-to-End Approaches for Joint Automatic Speech Recognition and Spoken Language Understanding

This paper presents the use of non-autoregressive (NAR) approaches for joint automatic speech recognition (ASR) and spoken language understanding (SLU) tasks. The proposed NAR systems employ a Conformer encoder that applies connectionist temporal classification (CTC) to transcribe the speech utterance into raw ASR hypotheses, which are further refined with a bidirectional encoder representations from Transformers (BERT)-like decoder. In the meantime, the intent and slot labels of the utterance are predicted simultaneously using the same decoder. Both Mask-CTC and self-conditioned CTC (SC-CTC) approaches are explored for this study. Experiments conducted on the SLURP dataset show that the proposed SC-Mask-CTC NAR system achieves 3.7% and 3.2% absolute gains in SLU metrics and a competitive level of ASR accuracy, when compared to a Conformer-Transformer based autoregressive (AR) model. Additionally, the NAR systems achieve 6× faster decoding speed than the AR baseline.
HMM vs. CTC for Automatic Speech Recognition: Comparison Based on Full-Sum Training from Scratch

In this work, we compare from-scratch sequence-level cross-entropy (full-sum) training of Hidden Markov Model (HMM) and Connectionist Temporal Classification (CTC) topologies for automatic speech recognition (ASR). Besides accuracy, we further analyze their capability for generating high-quality time alignment between the speech signal and the transcription, which can be crucial for many subsequent applications. Moreover, we propose several methods to improve convergence of from-scratch full-sum training by addressing the alignment modeling issue. Systematic comparison is conducted on both Switchboard and LibriSpeech corpora across CTC, posterior HMM with and w/o transition probabilities, and standard hybrid HMM. We also provide a detailed analysis of both Viterbi forced-alignment and Baum-Welch full-sum occupation probabilities.
Styletts-VC: One-Shot Voice Conversion by Knowledge Transfer From Style-Based TTS Models

One-shot voice conversion (VC) aims to convert speech from any source speaker to an arbitrary target speaker with only a few seconds of reference speech from the target speaker. This relies heavily on disentangling the speaker's identity and speech content, a task that still remains challenging. Here, we propose a novel approach to learning disentangled speech representation by transfer learning from style- based text-to-speech (TTS) models. With cycle consistent and adversarial training, the style-based TTS models can perform transcription-guided one-shot VC with high fidelity and similarity. By learning an additional mel-spectrogram encoder through a teacher-student knowledge transfer and novel data augmentation scheme, our approach results in disentangled speech representation without needing the input text. The subjective evaluation shows that our approach can significantly outperform the previous state-of-the-art one-shot voice conversion models in both naturalness and similarity.
On the Use of Modality-Specific Large-Scale Pre-Trained Encoders for Multimodal Sentiment Analysis

This paper investigates the effectiveness and implementation of modality-specific large-scale pre-trained encoders for multimodal sentiment analysis (MSA). Although the effectiveness of pre-trained encoders in various fields has been reported, conventional MSA methods employ them for only linguistic modality, and their application has not been investigated. This paper compares the features yielded by large-scale pre-trained encoders with conventional heuristic features. One each of the largest pre-trained encoders publicly available for each modality are used; CLIP-ViT, WavLM, and BERT for visual, acoustic, and linguistic modalities, respectively. Experiments on two datasets reveal that methods with domain-specific pre-trained encoders attain better performance than those with conventional features in both unimodal and multimodal scenarios. We also find it better to use the outputs of the intermediate layers of the encoders than those of the output layer. The codes are available at https://github.com/ando-hub/MSA_Pretrain.
Investigating Active-Learning-Based Training Data Selection for Speech Spoofing Countermeasure

Training a speech spoofing countermeasure (CM) that generalizes to various unseen test data is challenging. Methods such as data augmentation and self-supervised learning can help, but the imperfect CM performance still calls for additional strategies. This paper investigates CM training using active learning (AL) to select useful training data from a large pool set, which is an unexplored area for speech anti-spoofing. Existing AL methods are compared to select useful data from a large pool set. A new AL method is also proposed that actively removes useless data from a pool. Experiments demonstrate that an energy-score-based AL method and the proposed data-removing method outperformed our strong baseline, and the relative reduction in detection error rates was higher than 40% on multiple test sets. Furthermore, compared with a top-line method that blindly used the whole pool set for training, the two AL-based CMs used less training data and achieved better or similar performance.
Response Timing Estimation for Spoken Dialog Systems Based on Syntactic Completeness Prediction

Appropriate response timing is very important for achieving smooth dialog progression. Conventionally, prosodic, temporal and linguistic features have been used to determine timing. In addition to the conventional parameters, we propose to utilize the syntactic completeness after a certain time, which represents whether the other party is about to finish speaking. We generate the next token sequence from intermediate speech recognition results using a language model and obtain the probability of the end of utterance appearing <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$K$</tex> tokens ahead, where <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$K$</tex> varies from 1 to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$M$</tex> . We obtain an <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$M$</tex> -dimensional vector, which we denote as estimates of syntactic completeness (ESC). We evaluated this method on a simulated dialog database of a restaurant information center. The results confirmed that considering ESC improves the performance of response timing estimation, especially the accuracy in quick responses, compared with the method using only conventional features.
PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representations

While self-supervised speech representation learning (SSL) models serve a variety of downstream tasks, these models have been observed to overfit to the domain from which the unlabeled data originates. To alleviate this issue, we propose PADA (Pruning Assisted Domain Adaptation). Before performing the target-domain ASR fine-tuning, we discover the redundant weights from pre-trained wav2vec 2.0 models through various pruning strategies. We investigate the effect of Task-Agnostic and Task-Aware pruning and propose a new pruning paradigm called, Cross-Domain Task-Aware Pruning (CD-TAW). CD-TAW obtains the initial pruning mask from a well fine-tuned out-of-domain (OOD) model, thereby making use of the readily available fine-tuned models from the web. The proposed CD-TAW method achieves up to 20.6% relative WER improvement over our baseline when fine-tuned on a 2-hour subset of Switchboard data without language model (LM) decoding.
On the Utility of Self-Supervised Models for Prosody-Related Tasks

Self-Supervised Learning (SSL) from speech data has produced models that have achieved remarkable performance in many tasks, and that are known to implicitly represent many aspects of information latently present in speech signals. However, relatively little is known about the suitability of such models for prosody-related tasks or the extent to which they encode prosodic information. We present a new evaluation framework, “SUPERB-prosody,” consisting of three prosody-related downstream tasks and two pseudo tasks. We find that 13 of the 15 SSL models outperformed the baseline on all the prosody-related tasks. We also show good performance on two pseudo tasks: prosody reconstruction and future prosody prediction. We further analyze the layerwise contributions of the SSL models. Overall we conclude that SSL speech models are highly effective for prosody-related tasks. We release our code <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> https://github.com/JSALT-2022-SSL/superb-prosody for the community to support further investigation of SSL models' utility for prosody.
Monotonic Segmental Attention for Automatic Speech Recognition

We introduce a novel segmental-attention model for automatic speech recognition. We restrict the decoder attention to segments to avoid quadratic runtime of global attention, better generalize to long sequences, and eventually enable streaming. We directly compare global-attention and different segmental-attention modeling variants. We develop and compare two separate time-synchronous decoders, one specifically taking the segmental nature into account, yielding further improvements. Using time-synchronous decoding for segmental models is novel and a step towards streaming applications. Our experiments show the importance of a length model to predict the segment boundaries. The final best segmental-attention model using segmental decoding performs better than global-attention, in contrast to other monotonic attention approaches in the literature. Further, we observe that the segmental model generalizes much better to long sequences of up to several minutes.
Peppanet: Effective Mispronunciation Detection and Diagnosis Leveraging Phonetic, Phonological, and Acoustic Cues

Mispronunciation detection and diagnosis (MDD) aims to detect erroneous pronunciation segments in an L2 learner's articulation and subsequently provide informative diagnostic feedback. Most existing neural methods follow a dictation-based modeling paradigm that finds out pronunciation errors and returns diagnostic feedback at the same time by aligning the recognized phone sequence uttered by an L2 learner to the corresponding canonical phone sequence of a given text prompt. However, the main downside of these methods is that the dictation process and alignment process are mostly made independent of each other. In view of this, we present a novel end-to-end neural method, dubbed PeppaNet, building on a unified structure that can jointly model the dictation process and the alignment process. The model of our method learns to directly predict the pronunciation correctness of each canonical phone of the text prompt and in turn provides its corresponding diagnostic feedback. In contrast to the conventional dictation-based methods that rely mainly on a free-phone recognition process, PeppaNet makes good use of an effective selective gating mechanism to simultaneously incorporate phonetic, phonological and acoustic cues to generate corrections that are more proper and phonetically related to the canonical pronunciations. Extensive sets of experiments conducted on the L2-ARCTIC benchmark dataset seem to show the merits of our proposed method in comparison to some recent top-of-the-line methods.
An Investigation of Monotonic Transducers for Large-Scale Automatic Speech Recognition

The two most popular loss functions for streaming end-to-end automatic speech recognition (ASR) are RNN-Transducer (RNN-T) and connectionist temporal classification (CTC). Between these two loss types we can classify the monotonic RNN-T (MonoRNN-T) and the recently proposed CTC-like Transducer (CTC-T). Monotonic transducers have a few advantages. First, RNN-T can suffer from runaway hallucination, where a model keeps emitting non-blank symbols without advancing in time. Secondly, monotonic transducers consume exactly one model score per time step and are therefore more compatible with traditional FST-based ASR decoders. However, the MonoRNN-T so far has been found to have worse accuracy than RNN-T. It does not have to be that way: By regularizing the training via joint LAS training or parameter initialization from RNN-T, both MonoRNN-T and CTC-T perform as well or better than RNN-T. This is demonstrated for LibriSpeech and for a large-scale in-house data set.