Reviewerly/exHarmony huggingface dataset:
Understanding Online Harassment and Safety Concerns of Marginalized LGBTQ+ Populations on Social Media in Bangladesh

This note explores how various technology-mediated negative experiences and safety concerns of non-Western LGBTQ+ users, particularly from Bangladesh, hinder their continuing online interactions and self-presentation practices. Based on face-to-face and Skype semi-structured interviews (n=31), our initial results report that along with facing life-threatening harassing experiences online, Bangladeshi LGBTQ+ users also struggle with audience management and perceived privacy affordances that critically restrict their identity exploration and overall online participation, often forcing them to adopt fake/pseudo-identity online. These findings advocate for better design implications on safer social media participation, especially for LGBTQ+ users from non-Western contexts, and call for more attention to inclusive technologies.
Cross-Domain Learning with Normalizing Flow

Cross-domain learning aims to transfer knowledge learned from one or more datasets to other datasets in different domains, so that less data will be required for learning in new tasks and datasets. One big challenge in cross-domain learning is to effectively synergize the knowledge learning between domains. In this paper, we propose a new solution to address this challenge using normalizing flow, named as DomainFlow, which works as a learned mapping to establish knowledge sharing between source and target domains. The learned flow encourages the posterior distributions in multi-domain learning to be better aligned, leading to better performance in the target domain tasks. We conduct extensive experiments on three representative cross-domain learning tasks: unsupervised domain adaptation, domain generalization and zero-shot sketch-based image retrieval, which demonstrates that with DomainFlow, the overall performance on these diverse tasks can all be improved.
Contrastive Domain Adaptation Via Delimitation Discriminator

Unsupervised domain adaptation aims to transfer the knowledge learned from the labeled source domain to the unlabeled target domain, thereby improving the classification performance of the target domain. Recent methods use contrastive learning to optimize this task, however, these methods only focus on contrastive learning for aspects of domain alignment, which do not actively serve the classification task, resulting in suboptimal solution. To this end, we proposed contrastive domain adaptation via delimitation discriminator (CDVD), which addresses the inconsistency problem of optimizing contrastive learning and classification tasks. We introduce a delimitation discriminator to maximize the output difference between two types of random data-augmented samples of the target domain to detect data-augmented samples that are not conducive to classification, and then minimize the difference through the feature generator to generate effective data-augmented features (good for classification). Our extensive experiments on several public datasets show that CDVD has good adaptability and effectiveness.
Learning From Single-Expert Annotated Labels for Automatic Sleep Staging

Existing automatic sleep staging algorithms rely on accurately labeled data. However, due to the subjectivity of sleep experts, accurate labels must be obtained through joint labeling by multiple experts, which results in high time and labor costs. In this work, we treat labels mislabeled by a single expert as noisy labels and first propose SE-ASS, an automatic sleep staging learning framework based on single-expert annotated data. Since multiple models tend to produce inconsistent predictions for instances with incorrect labels during training, we use two networks with the same structure but different initializations and regularize them with a prediction consistency loss to prevent overfitting to noisy labels. Furthermore, we use a contrastive loss between models to enhance the exploration of feature representations without relying on potentially noisy labels. Our results on two publicly available datasets show that SE-ASS can effectively improve the performance of automatic sleep staging models trained on single-expert annotated datasets.
Topology Uncertainty Modeling For Imbalanced Node Classification on Graphs

Most existing graph neural networks work under a class-balanced assumption, while ignoring class-imbalanced scenarios that widely exist in real-world graphs. Although there are many methods in other fields that can alleviate this issue, they do not consider the special topology of the non-Euclidean graph. Hence, we propose Graph Topology Uncertainty (GraphTU), a novel probabilistic class-imbalanced solution specifically for graphs. Firstly, an invisible "uncertain gap" between under-represented minorities in training set and authentic minorities in unseen set is modeled by estimating statistical variances in topology. We extend the training distribution for minorities by sampling in this gap through a non-parametric way. Moreover, a gradient-guided mask is introduced to prevent biased statistics. Extensive experiments demonstrate the superior performance of GraphTU.
Massively Multilingual ASR on 70 Languages: Tokenization, Architecture, and Generalization Capabilities

End-to-end multilingual ASR has become more appealing because of several reasons such as simplifying the training and deployment process and positive performance transfer from high-resource to low-resource languages. However, scaling up the number of languages, total hours, and number of unique tokens is not a trivial task. This paper explores large-scale multilingual ASR models on 70 languages. We inspect two architectures: (1) Shared embedding and output and (2) Multiple embedding and output model. In the shared model experiments, we show the importance of tokenization strategy across different languages. Later, we use our optimal tokenization strategy to train multiple embedding and output model to further improve our result. Our multilingual ASR achieves 13.9%-15.6% average WER relative improvement compared to monolingual models. We show that our multilingual ASR generalizes well on an unseen dataset and domain, achieving 9.5% and 7.5% WER on Multilingual Librispeech (MLS) with zero-shot and finetuning, respectively.
Learning to Build Reasoning Chains by Reliable Path Retrieval

Question answering (QA) systems have long pursued the ability to reason over explicit knowledge credibly. Recent work has incorporated knowledge into fine-grained sentences and constructed natural language database (NLDB) task, and conducts complex QA with explicit reasoning chains. Existing models focus on retrieving evidence by combining multiple modules or discretely. However, these models ignore utilizing path information (e.g. sentence order), which is proven to be important for evidence retrievers. In this work, we propose a ReliAble Path-retrieval (RAP) to generate varying length evidence chains iteratively. It comprehensively models reasoning chains and introduces loss from two views. The experimental results show that our model demonstrates state-of-the-art performance on both evidence chain retrieval and question-answering tasks. Additional experiments on sequential supervised and sequential unsupervised retrieval fully indicate the significance of RAP.
Post-Trained Language Model Adaptive to Extractive Summarization of Long Spoken Documents

The General Meeting Understanding and Generation Challenge challenge track 2 in ICASSP 2023 Signal Processing Grand Challenge aims at extractive summarization of meeting transcripts. The main characteristic of this challenge is abridged as long spoken documents that are extremely difficult to manipulate with pre-trained language models. In this paper, we propose a post-trained DeBERTA which does not only adapt to spoken language but also manages long documents. According to the experimental results, the proposed method shows a 6% higher test score than the challenge baseline model.
AV-TAD: Audio-Visual Temporal Action Detection With Transformer

As an important and challenging task in video understanding, Temporal Action Detection (TAD) has been deeply studied in recent years. However, current works mainly tackle this task with visual information, while neglecting to explore the potential of the audio modality. To address this challenge, in this paper, we propose a simple yet effective AudioVisual Temporal Action Detection Transformer named AV- TAD, which performs early fusion on audio and visual modalities in an end-to-end fashion. On top of it, a novel query formulation is introduced by directly adopting temporal segment coordinates as queries in Transformer decoder, thus allowing us to perform dynamic segment update layer-by-layer. To the best of our knowledge, this is the first attempt to investigate both audio and video feature with a multi-modal Transformer in TAD task. Extensive experiments on THUMOS14 dataset demonstrate that our proposed AV-TAD can outperform the previous methods by a clear margin.
Exploring Instance Relation for Decentralized Multi-Source Domain Adaptation

Multi-source domain adaptation aims to transfer knowledge from multiple labeled source domains to an unlabeled target domain and reduce the domain shift. Considering data privacy and storage cost, the data from different domains are isolated, which leads to the difficulty of domain adaptation. To reduce the domain shift on the decentralized source domains and target domain, we propose an instance relation consistency method for decentralized multi-source domain adaptation. Specifically, we utilize the models from other domains as bridges to conduct domain adaptation. We impose inter-domain instance relation consistency on the isolated source and target domain to transfer the semantic relation knowledge across different domain models. Meanwhile, we exploit intra-domain instance relation consistency to learn the intrinsic semantic relation across different data views. Experiments on three benchmarks indicate the effectiveness of our method for decentralized multi-source domain adaptation.
Investigation into Phone-Based Subword Units for Multilingual End-to-End Speech Recognition

Multilingual automatic speech recognition (ASR) models with phones as modeling units have have improved greatly in low-resource and similar-language scenarios, which benefits from shared representation across languages. Meanwhile, subwords have demonstrated their effectiveness for monolingual end-to-end recognition systems. In this paper, we investigate the use of phone-based sub-words, specifically Byte Pair Encoding (BPE), as modeling units for multilingual end-to-end speech recognition. To explore the possibilities of phone-based BPE (PBPE) for multilingual ASR, we first use three types of multilingual BPE training methods for similar low-resource languages in Central Asia. Then, by adding three high-resource European languages to the experiments, we analyze language sharing degree in similar and low-resource scenarios. Finally, we propose a method to adjust the bigram statistics in the BPE algorithm and show that the PBPE representation leads to accuracy improvements in multilingual scenarios. The experiments show that PBPE outperforms phone, character and character-based BPE as output representation units. Particularly, the best PBPE model in multilingual experiments achieves a 25% relative improvement on a low-resource language compared to a character-based BPE system.
Tracking Objects and Activities with Attention for Temporal Sentence Grounding

Temporal sentence grounding (TSG) aims to localize the temporal segment which is semantically aligned with a natural language query in an untrimmed video. Most existing methods extract frame-grained features or object-grained features by 3D ConvNet or detection network under a conventional TSG framework, failing to capture the subtle differences between frames or to model the spatio-temporal behavior of core persons/objects. In this paper, we introduce a new perspective to address the TSG task by tracking pivotal objects and activities to learn more fine-grained spatio-temporal behaviors. Specifically, we propose a novel Temporal Sentence Tracking Network (TSTNet), which contains (A) a Cross-modal Targets Generator to generate multi-modal templates and search space, filtering objects and activities, and (B) a Temporal Sentence Tracker to track multi-modal targets for modeling the targets’ behavior and to predict query-related segment. Extensive experiments and comparisons with state-of-the-arts are conducted on challenging benchmarks: Charades-STA and TACoS. And our TSTNet achieves the leading performance with a considerable real-time speed.
Two-Stage Video De-Raining with Spatio-Temporal Fusion and Illumination-Invariant Detail Preservation

Video de-raining is an important yet highly challenging task in the field of computer vision. Though numerous video de-raining methods are developed with encouraging performance, two major challenges for video de-raining are still unsatisfactorily solved and need to be further investigated as follows: 1) how to sufficiently explore the useful spatio-temporal information from adjacent rainy frames to facilitate the rain removal, and 2) how to well preserve background details even in a video with illumination variance. Regarding the above challenges, this paper specifically develops a new two-stage video de-raining method, which cleverly integrates two typical modules that are beneficial for the video de-raining task, namely Spatio-Temporal Fusion (STF) module and Illumination-Invariant Detail Preservation (IIDP) module. The STF module is designed to fuse the spatio-temporal information from successive frames effectively, while the IIDP module is developed to deliver the enhanced features from the first stage sub-network to the second stage sub-network to preserve clear edge details of objects. Experimental results demonstrate the superiority of our proposed method over previous state-of-the-arts. The code will be publicly available at https://github.com/mapleTan1113/TSVDN.
Online Edge Flow Prediction Over Expanding Simplicial Complexes

Simplicial convolutional filters can process signals defined over levels of a simplicial complex such as nodes, edges, triangles, and so on with applications in e.g., flow prediction in transportation or financial networks. However, the underlying topology expands over time in a way that new edges and triangles form. For example, in a transportation network, a new connection between two locations is newly built, or in a currency exchange market, two currencies can be exchanged without an intermediate currency that can be understood as a new edge between them. To handle the streaming nature of data, we propose an online prediction for edge flows which generalizes to other higher-order simplicial signals. This is achieved by updating the filter coefficients via an online gradient descent with a provable sub-linear regret relative to the simplicial filter optimized over the whole sequence of edge flows. The update of the filter coefficients associated with the lower and upper Hodge Laplacians can be uncoupled in general. We test the online edge flow prediction on an expanding synthetic simplicial complex and a coauthorship complex showing a close performance to the offline counterpart.
Machine Learning Based Currency Classification System

There are 39 million blind people and 284 million people who are visually impaired worldwide, according to a World Health Organization estimate. In today's fast-paced environment, blind people are actively involved in learning new skills that will enhance their quality of life. As social beings, people frequently deal with money throughout their lives. Due to the incapability to recognize things, blind people are having a lot of trouble in dealing with money. In order to assist them in recognizing the currency we ought to bring a mobile application with a voice based output that reads cash. The classification of various cash notes in this application is done using machine learning techniques. To increase the model's accuracy, we have used data augmentation techniques and Teachable Machine to train the model. To execute ML models on numerous devices, we have used TensorFlow Lite library.
Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation

Direct speech-to-speech translation (S2ST) is an attractive research topic with many advantages compared to cascaded S2ST. However, direct S2ST suffers from the data scarcity problem because the corpora from the speech of the source language to the speech of the target language are very rare. To address this issue, we propose in this paper a Speech2S model, which is jointly pre-trained with unpaired speech and bilingual text data for direct speech-to-speech translation tasks. By effectively leveraging the paired text data, Speech2S is capable of modeling the cross-lingual speech conversion from source to target language. We verify the performance of the proposed Speech2S on Europarl-ST and VoxPopuli datasets. Experimental results demonstrate that Speech2S gets an improvement of about 5 BLEU scores compared to encoder-only pre-training models, and achieves a competitive or even better performance than existing state-of-the-art models <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Learning an insertion region for advertisement embedding on planes

Advertisement embedding aims to insert an advertisement into an image. The main task is to determine the most suitable insertion region on the planes for the advertisement to fit into. In this paper, we propose to realize practical advertisement embedding by learning the proper insertion region. As we assume the advertisement is a square in the 3D space, the insertion plane can be represented by its four corner points which correspond to the four corners of the advertisement. The common plane supervision for the plane detection in PlaneRCNN is an unconstrained shape of the 2D mask, which is not sufficient to label the square in the world coordinate. To solve this problem, we build a new dataset with the label of the most suitable insertion region. We map all of the points in the mask back to the world coordinates and compute a maximum square for each plane to determine the correct insertion region. With the proposed dataset, we design a network to regress the insertion region directly. The network is trained end-to-end with the original Mask R-CNN to resolve the ambiguity of point regression by multi-task learning loss. Our method is evaluated through quantitative and qualitative experiments, and the results demonstrate the impressive performance of our method on the proposed dataset.
Filter pruning by quantifying feature similarity and entropy of feature maps

Filter pruning can effectively reduce the time cost and computing resources of convolutional neural networks (CNNs), and is well applied to lightweight edge devices. However, most of the current pruning methods focus on the inherent properties of the filters themselves to prune the network, and pay less attention to the connection between the filters and the feature maps. Feature similarity (FSIM) utilizes the fact that the human visual system is more sensitive to the underlying features of the images to more accurately assess image quality. We discover that FSIM is also suitable for evaluating feature maps of CNNs. In addition, the information richness in the feature maps reflects the degree of importance of the filters. Based on the above research, we propose to quantify the importance of feature maps with FSIM and two-dimensional entropy (2D Entropy) indicator to further guide filter pruning (FSIM-E). The FSIM-E is executed on CIFAR-10 and ILSVRC-2012 to demonstrate that FSIM-E can effectively compress and accelerate the network model. For example, for ResNet-110 on CIFAR-10, FSIM-E prunes 71.1% of the FLOPs and 66.5% of the parameters, while improving the accuracy by 0.1%. With ResNet-50, FSIM-E can achieve 57.2% pruning rate of FLOPs and 53.1% pruning rate of parameters on ILSVRC-2012 with loss of only 0.42% of Top-5 accuracy.
Bandit Interpretability of Deep Models via Confidence Selection

Interpretability of black-box deep models is yet challenging because existing model-agnostic methods mainly locally explain the behavior of the classifier by learning a linear proxy around the instance being predicted. The explanation can be faithful locally, but may not be accurate globally. In this paper, we for the first time formulate the interpretation of classifiers as a bandit problem and introduce a Bandit Interpretation method via Confidence Selection (BICS). We statistically impose disturbances on different arms (image regions) and examine non-linear changes of the model’s output to fairly select important regions via Upper Confidence Bounds (UCB). Unlike previous model-agnostic methods that directly occlude super-pixels, our method softly applies perturbations at a pixel level and thus can fully explore more regions with multiple granularities, leading to a more precise and robust interpretation. Quantitative and qualitative experimental results demonstrate that our approach provides reasonable and precise explanations for various image recognition tasks on different models.
Bayesian network parameter learning using fuzzy constraints

Prior knowledge effectively mitigates low modeling accuracy when insufficient data exist. This idea has been confirmed in many fields, especially the transformation of prior knowledge into constraints, widely employed in Bayesian network (BN) parameter learning. The role of constraints in parameter learning is the focus of this study. If parameter learning results are blindly biased toward constraints, underfitting occurs, whereas the effectiveness of the constraint intervention is weak and can not mitigate overfitting by insufficient data. Therefore, fuzzy theory is introduced in parameter learning. The fuzzy membership function is applied to measure the interference effects of constraints and improve the interpretability and accuracy of the constraint usage. For the maximum a posteriori method, hyperparameters are utilized as virtual samples to realize the intervention of parameter learning. This study proposes a fuzzy maximum a posteriori (FMAP) method, in which the hyperparameter is adjusted to a suitable value using a fuzzy membership function. For the data extension method, a proper extension function is critical to the quality of the extended data. Therefore, this paper proposes a fuzzy bootstrap (FB) method that uses a fuzzy membership function to determine the distribution of the expansion parameters. The two algorithms presented in this paper are verified on 12 standard networks. The experimental results show that the proposed methods effectively improve the accuracy of parameter learning.