Reviewerly/exHarmony huggingface dataset:
Weather image classification based on generative adversarial network and transfer learning

In view of the low recognition accuracy of traditional weather recognition methods and the serious imbalance in the number of weather images in various categories in the weather image data set, a weather image classification algorithm based on generative adversarial network and transfer learning is proposed to solve the above problems. The proposed method mainly includes two parts: data equalization based on generative adversarial network and image classification based on transfer learning. This paper uses generative adversarial network to amplify the data of a few categories of weather images, so as to obtain a relatively balanced weather image data set.The method of transfer learning is used to fine-tune the model to realize the classification of weather images. The experimental results show that the method proposed in this paper is better than the traditional method, effectively solving the problem of low model classification accuracy caused by the imbalance of training samples, and realizing the recognition and classification of four types of weather images: sunny, foggy, rainy, and snowy.
Reversible steganographic method of hiding information based on image interpolation

When information is exchanged through open communication networks, there is a possibility of third-party interception. Various methods of data protection have been developed and applied to eliminate this flaw. In this work, a task of developing a new reversible steganographic method of concealing information based on interpolation of an image is set and solved. The developed algorithm has a higher payload of secret information while preserving the high visual quality of the stego image. Results of the pilot studies confirm this and are presented in this article.
ISHIGAKI Retrieval System Using 3D Shape Matching and Combinatorial Optimization

Abstract In April 2016, a massive earthquake with a magnitude of 7.3 struck Kumamoto region, Japan, causing major devastation. One of the structures that were damaged in Kumamoto was Kumamoto Castle, a cultural asset of great significance in Japan. The stone retaining wall “ishigaki” that formed the foundation of the castle collapsed, and the superstructure was destroyed. The number of stones is estimated to be more than 70,000, and restoration work is anticipated to take more than 20 years. Since each of the stones is an important cultural asset, the broken stone structure needed to be restored to its original state in order not to lose its cultural value forever. In addition, the fallen stones need to be returned to their original positions in the ishigakis. In similar cases, non-automatic visual verification was used. However, for Kumamoto Castle, this would have been impossible because a large number of stones were displaced as a result of the collapse. The purpose of this project is to provide support for the restoration work by matching the stones that fell down after the collapse with those before the collapse using information technology, such as computer vision and optimization technologies. Specifically, we captured photographic images of the stones before and after the collapse to match them. The technical contributions of this study are as follows: (a) To estimate the scale and surface orientation of the stones, we exploit 3D model construction from the images. (b) To solve the jigsaw-puzzle-like problem of reassembling the stone fragments, we exploit the combination of a customized iterative closest point (ICP) algorithm for shape position matching and an assignment algorithm to find the best pairs of stones before and after the collapse by using the matching degree obtained from ICP. Here, only the 2D shape of the stones before the collapse can be used due to the small number of photos available. In contrast, a detailed 3D shape can be obtained from the stones after their collapse. We matched these asymmetric data in 2D and 3D to enable a comprehensive reconstruction. (c) We developed a user-friendly graphical user interface system that was used by actual masons without special knowledge. The developed system was used to match the ishigaki of a turret, Iidamaru. As a result, we succeeded in identifying 337 stones, or approximately 90% of the 370 images. These results are expected to be useful for and were used as a blueprint during actual restoration work.
Towards Witness Encryption Without Multilinear Maps

Current proposals of extractable witness encryption are based on multilinear maps. In this paper, we propose a new construction without. We propose the notion of hidden group with hashing and make an extractable witness encryption from it. We show that the construction is secure in a generic model. We propose a concrete construction based on RSA-related problems. Namely, we use an extension of the knowledge-of-exponent assumption and the order problem. Our construction allows to encrypt for an instance of the subset sum problem (actually, a multidimensional variant of it) for which short solutions to the homogeneous equation are hard to find. Alas, we do not propose any reduction from a known NP-complete problem.
High-fidelity facial reconstruction from a single photo using photo-realistic rendering

We propose a fully automated method for realistic 3D face reconstruction from a single frontal photo that produces a high-resolution head mesh and a diffuse map. The photo is input to a convolutional neural network that estimates the weights of a morphable model to produce an initial head shape that is further adjusted through landmark-guided deformation. Two key features of the method are: 1) the network is exclusively trained on synthetic photos that are photo-realistic enough to learn real shape predictive features, making it unnecessary to train with real facial photos and corresponding 3D scans; 2) the landmarking statistical errors are incorporated in the reconstruction for optimal accuracy. While the method is based on a limited amount of real data, we show that it robustly and quickly performs plausible face reconstructions from real photos.
Research on the processing method of multi-source heterogeneous data in the intelligent agriculture cloud platform

Abstract With the development of big data and blockchain technology, a large amount of multi-source heterogeneous data has been accumulated in the agricultural field by before, during and after production. Agricultural information service systems are often targeted at specific regions, specific applications and specific data resources. Due to the lack of effective analysis and refining, the conversion efficiency of data resources into useful information is too low, resulting in contradiction between the continuous enrichment of agricultural data resources and the relative lack of agricultural information services. Therefore, in view of the multi-source heterogeneous characteristics of agricultural data and the specific business needs of different agricultural scenarios, the intelligent processing method of agricultural data is analysed, and a heuristic algorithm based on K-Means limited clustering number is proposed to judge the accuracy of abnormal data processing. By inputting test sample data for testing, the algorithm has improved accuracy by nearly 30% compared to traditional K-Means.
Visualization and Interpretation of Latent Spaces for Controlling Expressive Speech Synthesis through Audio Analysis

The field of Text-to-Speech has experienced huge improvements last years benefiting from deep learning techniques. Producing realistic speech becomes possible now. As a consequence, the research on the control of the expressiveness, allowing to generate speech in different styles or manners, has attracted increasing attention lately. Systems able to control style have been developed and show impressive results. However the control parameters often consist of latent variables and remain complex to interpret. In this paper, we analyze and compare different latent spaces and obtain an interpretation of their influence on expressive speech. This will enable the possibility to build controllable speech synthesis systems with an understandable behaviour.
[Multimodality-based super-resolution reconstruction for routine brain magnetic resonance images].

To propose a multi-modality-based super-resolution synthesis model for reconstruction of routine brain magnetic resonance images (MRI) with a low resolution and a high thickness into high-resolution images.Based on real paired low-high resolution MRI data (2D T1, 2D T2 FLAIR and 3D T1), a structure-constrained image mapping network was used to extract important features from the images with different modalities including the whole T1 and subcortical regions of T2 FLAIR to reconstruct T1 images with higher resolutions. The gray scale intensity and structural similarities between the super-resolution images and high-resolution images were used to enhance the reconstruction performance. We used the anatomical information acquired from segment maps of the super-resolution T1 image and the ground truth by a segmentation tool as a significant constraint for adaptive learning of the intrinsic tissue structure characteristics of the brain to improve the reconstruction performance of the model.Our method showed the performance on the testing dataset than other methods with an average PSNR of 33.11 and SSIM of 0.996. The anatomical structure of the brain including the sulcus, gyrus, and subcortex were all reconstructed clearly using the proposed method, which also greatly enhanced the precision of MSCSR for brain volume measurement.The proposed MSCSR model shows excellent performance for reconstructing super-resolution brain MR images based on the information of brain tissue structure and multimodality MR images.
SCaLa: Supervised Contrastive Learning for End-to-End Speech Recognition

End-to-end Automatic Speech Recognition (ASR) models are usually trained to optimize the loss of the whole token sequence, while neglecting explicit phonemic-granularity supervision. This could result in recognition errors due to similar-phoneme confusion or phoneme reduction. To alleviate this problem, we propose a novel framework based on Supervised Contrastive Learning (SCaLa) to enhance phonemic representation learning for end-to-end ASR systems. Specifically, we extend the self-supervised Masked Contrastive Predictive Coding (MCPC) to a fully-supervised setting, where the supervision is applied in the following way. First, SCaLa masks variable-length encoder features according to phoneme boundaries given phoneme forced-alignment extracted from a pre-trained acoustic model; it then predicts the masked features via contrastive learning. The forced-alignment can provide phoneme labels to mitigate the noise introduced by positive-negative pairs in self-supervised MCPC. Experiments on reading and spontaneous speech datasets show that our proposed approach achieves 2.8 and 1.4 points Character Error Rate (CER) absolute reductions compared to the baseline, respectively.
IntentVizor: Towards Generic Query Guided Interactive Video Summarization

The target of automatic video summarization is to create a short skim of the original long video while preserving the major content/events. There is a growing interest in the integration of user queries into video summarization or query-driven video summarization. This video summarization method predicts a concise synopsis of the original video based on the user query, which is commonly represented by the input text. However, two inherent problems exist in this query-driven way. First, the text query might not be enough to describe the exact and diverse needs of the user. Second, the user cannot edit once the summaries are produced, while we assume the needs of the user should be subtle and need to be adjusted interactively. To solve these two problems, we propose IntentVizor, an interactive video summarization framework guided by generic multi-modality queries. The input query that describes the user's needs are not limited to text but also the video snippets. We further represent these multi-modality finer-grained queries as user `intent', which is interpretable, interactable, editable, and can better quantify the user's needs. In this paper, we use a set of the proposed intents to represent the user query and design a new interactive visual analytic interface. Users can interactively control and adjust these mixed-initiative intents to obtain a more satisfying summary through the interface. Also, to improve the summarization quality via video understanding, a novel Granularity-Scalable Ego-Graph Convolutional Networks (GSE-GCN) is proposed. We conduct our experiments on two benchmark datasets. Comparisons with the state-of-the-art methods verify the effectiveness of the proposed framework. Code and dataset are available at https://github.com/jnzs1836/intent-vizor.
Securing Wireless Payment-Channel Networks With Minimum Lock Time Windows

Payment-channel networks (PCN) enhance the impact of cryptocurrencies by providing a fast and consensus-free solution to the scalability problems of traditional blockchain protocols. However, PCNs often rely on powerful nodes with high availability, large storage capacity, and strong computational power, which hinders their adoption in mobile environments. In this paper, we consider a PCN architecture that extends the functionalities of traditional PCNs to wireless resource-constrained devices. We address the token theft problem, a vulnerability that is critical on wireless PCNs, and propose a countermeasure based on minimum time windows that lock tokens whenever a user disconnects. We evaluate our proposal with real data from Bitcoin’s Lightning Network and 3G/4G mobile broadband networks. The results show that the countermeasure is most effective when devices present high availability and that there is a security-efficiency trade-off when connectivity is low.
The FeatureCloud AI Store for Federated Learning in Biomedicine and Beyond

Machine Learning (ML) and Artificial Intelligence (AI) have shown promising results in many areas and are driven by the increasing amount of available data. However, this data is often distributed across different institutions and cannot be shared due to privacy concerns. Privacy-preserving methods, such as Federated Learning (FL), allow for training ML models without sharing sensitive data, but their implementation is time-consuming and requires advanced programming skills. Here, we present the FeatureCloud AI Store for FL as an all-in-one platform for biomedical research and other applications. It removes large parts of this complexity for developers and end-users by providing an extensible AI Store with a collection of ready-to-use apps. We show that the federated apps produce similar results to centralized ML, scale well for a typical number of collaborators and can be combined with Secure Multiparty Computation (SMPC), thereby making FL algorithms safely and easily applicable in biomedical and clinical environments.
Robust Feature Disentanglement in Imaging Data via Joint Invariant Variational Autoencoders: from Cards to Atoms

Recent advances in imaging from celestial objects in astronomy visualized via optical and radio telescopes to atoms and molecules resolved via electron and probe microscopes are generating immense volumes of imaging data, containing information about the structure of the universe from atomic to astronomic levels. The classical deep convolutional neural network architectures traditionally perform poorly on the data sets having a significant orientational disorder, that is, having multiple copies of the same or similar object in arbitrary orientation in the image plane. Similarly, while clustering methods are well suited for classification into discrete classes and manifold learning and variational autoencoders methods can disentangle representations of the data, the combined problem is ill-suited to a classical non-supervised learning paradigm. Here we introduce a joint rotationally (and translationally) invariant variational autoencoder (j-trVAE) that is ideally suited to the solution of such a problem. The performance of this method is validated on several synthetic data sets and extended to high-resolution imaging data of electron and scanning probe microscopy. We show that latent space behaviors directly comport to the known physics of ferroelectric materials and quantum systems. We further note that the engineering of the latent space structure via imposed topological structure or directed graph relationship allows for applications in topological discovery and causal physical learning.
A Review of Speaker Diarization: Recent Advances with Deep Learning

Speaker diarization is a task to label audio or video recordings with classes that correspond to speaker identity, or in short, a task to identify "who spoke when". In the early years, speaker diarization algorithms were developed for speech recognition on multispeaker audio recordings to enable speaker adaptive processing. These algorithms also gained their own value as a standalone application over time to provide speaker-specific metainformation for downstream tasks such as audio retrieval. More recently, with the emergence of deep learning technology, which has driven revolutionary changes in research and practices across speech application domains, rapid advancements have been made for speaker diarization. In this paper, we review not only the historical development of speaker diarization technology but also the recent advancements in neural speaker diarization approaches. Furthermore, we discuss how speaker diarization systems have been integrated with speech recognition applications and how the recent surge of deep learning is leading the way of jointly modeling these two components to be complementary to each other. By considering such exciting technical trends, we believe that this paper is a valuable contribution to the community to provide a survey work by consolidating the recent developments with neural methods and thus facilitating further progress toward a more efficient speaker diarization.
Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks

There are two approaches for pairwise sentence scoring: Cross-encoders, which perform full-attention over the input pair, and Bi-encoders, which map each input independently to a dense vector space. While cross-encoders often achieve higher performance, they are too slow for many practical use cases. Bi-encoders, on the other hand, require substantial training data and fine-tuning over the target task to achieve competitive performance. We present a simple yet efficient data augmentation strategy called Augmented SBERT, where we use the cross-encoder to label a larger set of input pairs to augment the training data for the bi-encoder. We show that, in this process, selecting the sentence pairs is non-trivial and crucial for the success of the method. We evaluate our approach on multiple tasks (in-domain) as well as on a domain adaptation task. Augmented SBERT achieves an improvement of up to 6 points for in-domain and of up to 37 points for domain adaptation tasks compared to the original bi-encoder performance.
OpenEDS2020: Open Eyes Dataset

We present the second edition of OpenEDS dataset, OpenEDS2020, a novel dataset of eye-image sequences captured at a frame rate of 100 Hz under controlled illumination, using a virtual-reality head-mounted display mounted with two synchronized eye-facing cameras. The dataset, which is anonymized to remove any personally identifiable information on participants, consists of 80 participants of varied appearance performing several gaze-elicited tasks, and is divided in two subsets: 1) Gaze Prediction Dataset, with up to 66,560 sequences containing 550,400 eye-images and respective gaze vectors, created to foster research in spatio-temporal gaze estimation and prediction approaches; and 2) Eye Segmentation Dataset, consisting of 200 sequences sampled at 5 Hz, with up to 29,500 images, of which 5% contain a semantic segmentation label, devised to encourage the use of temporal information to propagate labels to contiguous frames. Baseline experiments have been evaluated on OpenEDS2020, one for each task, with average angular error of 5.37 degrees when performing gaze prediction on 1 to 5 frames into the future, and a mean intersection over union score of 84.1% for semantic segmentation. As its predecessor, OpenEDS dataset, we anticipate that this new dataset will continue creating opportunities to researchers in eye tracking, machine learning and computer vision communities, to advance the state of the art for virtual reality applications. The dataset is available for download upon request at http://research.fb.com/programs/openeds-2020-challenge/.
Fact or Fiction: Verifying Scientific Claims

We introduce scientific claim verification, a new task to select abstracts from the research literature containing evidence that SUPPORTS or REFUTES a given scientific claim, and to identify rationales justifying each decision. To study this task, we construct SciFact, a dataset of 1.4K expert-written scientific claims paired with evidence-containing abstracts annotated with labels and rationales. We develop baseline models for SciFact, and demonstrate that simple domain adaptation techniques substantially improve performance compared to models trained on Wikipedia or political news. We show that our system is able to verify claims related to COVID-19 by identifying evidence from the CORD-19 corpus. Our experiments indicate that SciFact will provide a challenging testbed for the development of new systems designed to retrieve and reason over corpora containing specialized domain knowledge. Data and code for this new task are publicly available at https://github.com/allenai/scifact. A leaderboard and COVID-19 fact-checking demo are available at https://scifact.apps.allenai.org.
CASIA at SemEval-2022 Task 11: Chinese Named Entity Recognition for Complex and Ambiguous Entities

This paper describes our approach to develop a complex named entity recognition system in SemEval 2022 Task 11: MultiCoNER Multilingual Complex Named Entity Recognition,Track 9 -Chinese.In this task, we need to identify the entity boundaries and category labels for the six identified categories of CW, LOC, PER, GRP, CORP, and PORD.The task focuses on detecting semantically ambiguous and complex entities in short and low-context settings.We constructed a hybrid system based on Roberta-large model with three training mechanisms and a series of data augmentation.Three training mechanisms include adversarial training, Child-Tuning training, and continued pre-training.The core idea of the hybrid system is to improve the performance of the model in complex environments by introducing more domain knowledge through data augmentation and continuing pre-training domain adaptation of the model.Our proposed method in this paper achieves a macro-F1 of 0.797 on the final test set, ranking second.
Improving Contextual Representation with Gloss Regularized Pre-training

Though achieving impressive results on many NLP tasks, the BERT-like masked language models (MLM) encounter the discrepancy between pre-training and inference.In light of this gap, we investigate the contextual representation of pre-training and inference from the perspective of word probability distribution.We discover that BERT risks neglecting the contextual word similarity in pre-training.To tackle this issue, we propose an auxiliary gloss regularizer module to BERT pre-training (GR-BERT), to enhance word semantic similarity.By predicting masked words and aligning contextual embeddings to corresponding glosses simultaneously, the word similarity can be explicitly modeled.We design two architectures for GR-BERT and evaluate our model in downstream tasks.Experimental results show that the gloss regularizer benefits BERT in wordlevel and sentence-level semantic representation.The GR-BERT achieves new state-of-theart in lexical substitution task and greatly promotes BERT sentence representation in both unsupervised and supervised STS tasks.