Reviewerly/exHarmony huggingface dataset:
Investigating Learning Dynamics of BERT Fine-Tuning.

The recently introduced pre-trained language model BERT advances the state-of-the-art on many NLP tasks through the fine-tuning approach, but few studies investigate how the fine-tuning process improves the model performance on downstream tasks. In this paper, we inspect the learning dynamics of BERT fine-tuning with two indicators. We use JS divergence to detect the change of the attention mode and use SVCCA distance to examine the change to the feature extraction mode during BERT fine-tuning. We conclude that BERT fine-tuning mainly changes the attention mode of the last layers and modifies the feature extraction mode of the intermediate and last layers. Moreover, we analyze the consistency of BERT fine-tuning between different random seeds and different datasets. In summary, we provide a distinctive understanding of the learning dynamics of BERT fine-tuning, which sheds some light on improving the fine-tuning results.
Viewmaker Networks: Learning Views for Unsupervised Representation Learning

Many recent methods for unsupervised representation learning involve training models to be invariant to different views, or augmented versions of an input. However, designing these views requires considerable human expertise and experimentation, hindering widespread adoption of unsupervised representation learning methods across domains and modalities. To address this, we propose viewmaker networks: generative models which learn to produce input-dependent views for contrastive learning. We train these networks jointly with the main network to produce adversarial lp perturbations for an input, which yields challenging yet faithful views without extensive human tuning. Our learned views enable comparable transfer accuracy to the the well-studied SimCLR augmentations when applied on CIFAR-10, while significantly outperforming baseline augmentations in speech (+9% absolute) and IMU sensor (+17% absolute) domains. We also show how viewmaker views can be combined with SimCLR views to improve robustness to common image corruptions. Our method provides a roadmap for reducing the amount of expertise and effort needed for unsupervised learning, potentially extending its benefits to a much wider set of domains.
MetaNorm: Learning to Normalize Few-Shot Batches Across Domains

Batch normalization plays a crucial role when training deep neural networks. However, batch statistics become unstable with small batch sizes and are unreliable in the presence of distribution shifts. We propose MetaNorm, a simple yet effective meta-learning normalization. It tackles the aforementioned issues in a unified way by leveraging the meta-learning setting and learns to infer adaptive statistics for batch normalization. MetaNorm is generic, flexible and model-agnostic, making it a simple plug-and-play module that is seamlessly embedded into existing meta-learning approaches. It can be efficiently implemented by lightweight hypernetworks with low computational cost. We verify its effectiveness by extensive evaluation on representative tasks suffering from the small batch and domain shift problems: few-shot learning and domain generalization. We further introduce an even more challenging setting: few-shot domain generalization. Results demonstrate that MetaNorm consistently achieves better, or at least competitive, accuracy compared to existing batch normalization methods.
Development of Reading Materials with Coastal Culture Tapanuli Central for School Literation Movement in Basic School Pandan District

The purpose of this study was to determine: (1) The results of the design of reading materials with coastal culture for literacy sources, (2) The results of expert validation on the design of reading materials containing coastal culture for literacy sources, and ( 3) The results of testing of reading materials with coastal culture. for literacy sources in Class V Elementary School, Pandan, Tapanuli Tengah. This type of research is the type of research development or Research &amp; Development. The development model in this study will be made with a few modifications adapted to the Borg and Gall development procedure. Development research is generally divided into 3 (three) stages, namely the problem identification stage, the design stage, and the dissemination stage. The researcher attempted to adapt the development model to Borg and Gall's development procedures. As subjects in this study are all State and Private Elementary Schools in Pandan, Tapanuli Tengah. As the object of this research, developing reading materials based on the culture of the coastal communities of North Tapanuli Tengah Regency in supporting the School Literacy Movement. The results showed: (1) reading material with Tapanuli Tengah coastal culture is valid for use in the implementation of the school literacy movement in SD Pandan District.
Translating Similar Languages: Role of Mutual Intelligibility in Multilingual Transformers.

In this work we investigate different approaches to translate between similar languages despite low resource limitations. This work is done as the participation of the UBC NLP research group in the WMT 2019 Similar Languages Translation Shared Task. We participated in all language pairs and performed various experiments. We used a transformer architecture for all the models and used back-translation for one of the language pairs. We explore both bilingual and multi-lingual approaches. We describe the pre-processing, training, translation and results for each model. We also investigate the role of mutual intelligibility in model performance.
Tencent AI Lab Machine Translation Systems for WMT20 Chat Translation Task

This paper describes the Tencent AI Lab’s submission of the WMT 2020 shared task on chat translation in English-German. Our neural machine translation (NMT) systems are built on sentence-level, document-level, non-autoregressive (NAT) and pretrained models. We integrate a number of advanced techniques into our systems, including data selection, back/forward translation, larger batch learning, model ensemble, finetuning as well as system combination. Specifically, we proposed a hybrid data selection method to select high-quality and in-domain sentences from out-of-domain data. To better capture the source contexts, we exploit to augment NAT models with evolved cross-attention. Furthermore, we explore to transfer general knowledge from four different pre-training language models to the downstream translation task. In general, we present extensive experimental results for this new translation task. Among all the participants, our German-to-English primary system is ranked the second in terms of BLEU scores.
Evaluation of Blockchain-based Data Sharing Acceptance among Intelligence Community

Intelligence data are among the critical elements used as a reference for risk-assessment and decision-making regarding national security.The intelligence data are shared among intelligence agencies in the intelligence community in improving the efficiency of their services.Centralised data with central authority is highly exposed to being an easy target of attackers.Leaked or unauthorised access of the intelligence data to a non-intelligence community will bring severe effect to a state.Blockchain as immutable and high-security technology is capable of providing cryptographic data in a decentralised environment and potentially can be applied for data sharing among the intelligence community.However, the acceptance and readiness of users on blockchain usage in the intelligence community are yet to be systematically studied.Considering the statement, this paper proposed an evaluation method to study the acceptance of blockchain technology by integrating a reliable acceptance model for blockchain technology implementation in the intelligence community.The acceptance model consisted of constructs from the Technology Acceptance Model 3 (TAM 3) and Technology Readiness Index 2 (TRI 2) and was analysed using partial least squares structural equation modelling (PLS-SEM).In this study, the result indicates that TAM 3 and TRI 2.0 integration model could contribute to determining the acceptance level in developing blockchain-based intelligence data sharing for the intelligence community.
Functional differentiations in evolutionary reservoir computing networks

We propose an extended reservoir computer that shows the functional differentiation of neurons. The reservoir computer is developed to enable changing of the internal reservoir using evolutionary dynamics, and we call it an evolutionary reservoir computer. To develop neuronal units to show specificity, depending on the input information, the internal dynamics should be controlled to produce contracting dynamics after expanding dynamics. Expanding dynamics magnifies the difference of input information, while contracting dynamics contributes to forming clusters of input information, thereby producing multiple attractors. The simultaneous appearance of both dynamics indicates the existence of chaos. In contrast, sequential appearance of these dynamics during finite time intervals may induce functional differentiations. In this paper, we show how specific neuronal units are yielded in the evolutionary reservoir computer.
Simulating self-learning in photorefractive optical reservoir computers

Photorefractive materials exhibit an interesting plasticity under the influence of an optical field. By extending the finite-difference time-domain method to include the photorefractive effect, we explore how this property can be exploited in the context of neuromorphic computing for telecom applications. By first priming the photorefractive material with a random bit stream, the material reorganizes itself to better recognize simple patterns in the stream. We demonstrate this by simulating a typical reservoir computing setup, which gets a significant performance boost on performing the XOR on two consecutive bits in the stream after this initial priming step.
Variationally Inferred Sampling through a Refined Bound

In this work, a framework to boost the efficiency of Bayesian inference in probabilistic models is introduced by embedding a Markov chain sampler within a variational posterior approximation. We call this framework “refined variational approximation”. Its strengths are its ease of implementation and the automatic tuning of sampler parameters, leading to a faster mixing time through automatic differentiation. Several strategies to approximate evidence lower bound (ELBO) computation are also introduced. Its efficient performance is showcased experimentally using state-space models for time-series data, a variational encoder for density estimation and a conditional variational autoencoder as a deep Bayes classifier.
Employee’s Attitude and Organizational Sustainability Performance: An Evidence From Jordan’s Banking Sector

This study aims to investigate the role of employee’s attitudes towards implementing and expanding the sustainability policy and the sustainability performance in terms of economic, social, and environmental dimensions. The study model and hypotheses were developed based on a comprehensive literature review. The data were collected from 84 employees working in Jordan’s banking sector through a questionnaire. The results revealed that there is a significant and positive relationship between employee’s attitudes towards implementing and expanding the sustainability policy and the triple bottom lines of sustainability performance. This study promotes the idea of focusing on employee’s attitudes in the banking sector and others sectors in order to enhance sustainability performance, especially that sustainability still does not have the highest priority in a country with an emerging economy like Jordan. The results of the study have significant implications and open many avenues for future research.
Data Anonymisation, Outlier Detection and Fighting Overfitting with Restricted Boltzmann Machines

We propose a novel approach to the anonymisation of datasets through non-parametric learning of the underlying multivariate distribution of dataset features and generation of the new synthetic samples from the learned distribution. The main objective is to ensure equal (or better) performance of the classifiers and regressors trained on synthetic datasets in comparison with the same classifiers and regressors trained on the original data. The ability to generate unlimited number of synthetic data samples from the learned distribution can be a remedy in fighting overtting when dealing with small original datasets. When the synthetic data generator is trained as an autoencoder with the bottleneck information compression structure we can also expect to see a reduced number of outliers in the generated datasets, thus further improving the generalization capabilities of the classifiers trained on synthetic data. We achieve these objectives with the help of the Restricted Boltzmann Machine, a special type of generative neural network that possesses all the required properties of a powerful data anonymiser.
Sequence Image Interpolation via Separable Convolution Network

Remote-sensing time-series data are significant for global environmental change research and a better understanding of the Earth. However, remote-sensing acquisitions often provide sparse time series due to sensor resolution limitations and environmental factors, such as cloud noise for optical data. Image interpolation is the method that is often used to deal with this issue. This paper considers the deep learning method to learn the complex mapping of an interpolated intermediate image from predecessor and successor images, called separable convolution network for sequence image interpolation. The separable convolution network uses a separable 1D convolution kernel instead of 2D kernels to capture the spatial characteristics of input sequence images and then is trained end-to-end using sequence images. Our experiments, which were performed with unmanned aerial vehicle (UAV) and Landsat-8 datasets, show that the method is effective to produce high-quality time-series interpolated images, and the data-driven deep model can better simulate complex and diverse nonlinear image data information.
Analisis Media Pembelajaran Flash Card Untuk Anak Usia Dini

Early Childhood Education is one of the educational institutions which is now a concern among the world of education. So that it has become a necessity for observers and education policy makers to pay attention and think of what strategies are best to be implemented in the implementation of learning in PAUD institutions. Learning strategies are defined as each activity, whether procedures, steps, methods or media chosen in order to provide convenience, facilities, and other assistance to students in achieving instructional goals. The learning strategies in early childhood always prioritize aspects of the activities of playing, singing (having fun), and working in the sense of activities. Playing, singing and doing activities are three characteristics of PAUD. Any aspect of education should be covered with active play, singing, and activities or work. These three things will hone brain intelligence, emotional intelligence, and physical skills that are carried out cheerfully, freely, and without burden. In the learning strategy contains various alternatives that must be considered to be chosen in the framework of teaching planning. To implement a certain strategy requires a set of teaching media. One of the learning media that can be chosen is flash card learning media.
The biglasso Package: A Memory- and Computation-Efficient Solver for Lasso Model Fitting with Big Data in R

Penalized regression models such as the lasso have been extensively applied to analyzing high-dimensional data sets. However, due to memory limitations, existing R packages like glmnet and ncvreg are not capable of fitting lasso-type models for ultrahigh-dimensional, multi-gigabyte data sets that are increasingly seen in many areas such as genetics, genomics, biomedical imaging, and high-frequency finance. In this research, we implement an R package called biglasso that tackles this challenge. biglasso utilizes memory-mapped files to store the massive data on the disk, only reading data into memory when necessary during model fitting, and is thus able to handle out-of-core computation seamlessly. Moreover, it's equipped with newly proposed, more efficient feature screening rules, which substantially accelerate the computation. Benchmarking experiments show that our biglasso package, as compared to existing popular ones like glmnet, is much more memory- and computation-efficient. We further analyze a 31 GB real data set on a laptop with only 16 GB RAM to demonstrate the out-of-core computation capability of biglasso in analyzing massive data sets that cannot be accommodated by existing R packages.
Graph and Natural Language Processing Based Recommendation System for Choosing Machine Learning Algorithms

Machine Learning is a subset of Artificial Intelligence(AI). It provides the systems with the ability to learn automatically and perform independently without being programmed. When concentrating on the machine learning project choosing an algorithm is one of the most important steps. Machine learning algorithms are the math and logic that adjust the training of the model and the performance. When it comes to a machine learning project the choosing a correct algorithm for implementing the model is a big task. If the developer doesn't choose the suitable and most efficient algorithm for the program the accuracy of the program can be decreased. When we look at these machine learning algorithms there is no solution or no approach that fits for all. Several factors affect when choosing a machine learning algorithm such as the number of data, type of the problem ..etc. Most of the time a developer chooses the machine learning algorithm using his prior experience or analyzing several past similar projects. However, when it comes to a beginner it can be a tough experience. Most of the time beginners try with several algorithmic approaches to implement their model without any understanding. However, to avoid this there is no proper solution. Therefore here in this research, proposed an NLP and graph analytics-based approach for recommending a machine learning algorithm for a project. The summary of the proposed solution is here it analyzes past algorithms used in machine learning projects which are stored in a graph using NLP based keyword analysis and recommend the most suitable algorithmic approach. When the user inputs his project idea by using the Natural Language processing it generates keywords for the project description. Thereafter the system analyzes the graph which stores past machine learning projects and used technologies to find the most suitable algorithmic approach to the users' project. Moreover, it shows how the proposed algorithms were used in past similar projects. Therefore by using this system the developers can get a clear idea of the algorithm approach that they need to choose.
A supervised scheme for aspect extraction in sentiment analysis using the hybrid feature set of word dependency relations and lemmas

Due to the massive progression of the Web, people post their reviews for any product, movies and places they visit on social media. The reviews available on social media are helpful to customers as well as the product owners to evaluate their products based on different reviews. Analyzing structured data is easy as compared to unstructured data. The reviews are available in an unstructured format. Aspect-Based Sentiment Analysis mines the aspects of a product from the reviews and further determines sentiment for each aspect. In this work, two methods for aspect extraction are proposed. The datasets used for this work are SemEval restaurant review dataset, Yelp and Kaggle datasets. In the first method a multivariate filter-based approach for feature selection is proposed. This method support to select significant features and reduces redundancy among selected features. It shows improvement in F 1-score compared to a method that uses only relevant features selected using Term Frequency weight. In another method, selective dependency relations are used to extract features. This is done using Stanford NLP parser. The results gained using features extracted by selective dependency rules are better as compared to features extracted by using all dependency rules. In the hybrid approach, both lemma features and selective dependency relation based features are extracted. Using the hybrid feature set, 94.78% accuracy and 85.24% F 1-score is achieved in the aspect category prediction task.
Facial Image Segmentation Based on Gabor Filter

As an important part of face recognition, facial image segmentation has become a focus of human feature detection. In this paper, the AdaBoost algorithm and the Gabor texture analysis algorithm are used to segment an image containing multiple faces, which effectively reduces the false detection rate of facial image segmentation. In facial image segmentation, the image containing face information is first analyzed for texture using the Gabor algorithm, and appropriate thresholds are set with different thresholds of skin-like areas, where skin-like areas in the image’s background information are removed. Then, the AdaBoost algorithm is used to detect face regions, and finally, the detected face regions are segmented. Experiments show that this method can quickly and accurately segment faces in an image and effectively reduce the rate of missed and false detections.
The Digital Transformation Effects in Distance Education in Light of the Epidemics (COVID-19) in Egypt

On the relatively rare occasions when disaster forces schools and universities to close for a prolonged period of time, e-learning has helped fill the gap in instruction In this paper, we study the role of digital transformation in e-learning systems in light of the global conditions resulting from the epidemics (COVID-19) in Egypt Therefore, we focus on the importance of distance education at several factors, trying to assess the staff's response and students to new education methods and assess the distance education experience in Egyptian universities Where the study set that various staff excited to utilize this method, unlike some of the students, who did not to accept because of their knowledge lacking © 2021 NSP Natural Sciences Publishing Cor
Distance-Based Regularisation of Deep Networks for Fine-Tuning

We investigate approaches to regularisation during fine-tuning of deep neural networks. First we provide a neural network generalisation bound based on Rademacher complexity that uses the distance the weights have moved from their initial values. This bound has no direct dependence on the number of weights and compares favourably to other bounds when applied to convolutional networks. Our bound is highly relevant for fine-tuning, because providing a network with a good initialisation based on transfer learning means that learning can modify the weights less, and hence achieve tighter generalisation. Inspired by this, we develop a simple yet effective fine-tuning algorithm that constrains the hypothesis class to a small sphere centred on the initial pre-trained weights, thus obtaining provably better generalisation performance than conventional transfer learning. Empirical evaluation shows that our algorithm works well, corroborating our theoretical results. It outperforms both state of the art fine-tuning competitors, and penalty-based alternatives that we show do not directly constrain the radius of the search space.