Query: Self-training with Weak Supervision

title: Self Training with Weak Supervision
abstract: State of the art deep neural networks require large scale labeled training data that is often expensive to obtain or not available for many tasks. Weak supervision in the form of domain specific rules has been shown to be useful in such settings to automatically generate weakly labeled training data. However, learning with weak rules is challenging due to their inherent heuristic and noisy nature. An additional challenge is rule coverage and overlap, where prior work on weak supervision only considers instances that are covered by weak rules, thus leaving valuable unlabeled data behind. In this work, we develop a weak supervision framework (ASTRA) that leverages all the available data for a given task. To this end, we leverage task specific unlabeled data through self training with a model (student) that considers contextualized representations and predicts pseudo labels for instances that may not be covered by weak rules. We further develop a rule attention network (teacher) that learns how to aggregate student pseudo labels with weak rule labels, conditioned on their fidelity and the underlying context of an instance. Finally, we construct a semi supervised learning objective for end to end training with unlabeled data, domain specific rules, and a small amount of labeled data. Extensive experiments on six benchmark datasets for text classification demonstrate the effectiveness of our approach with significant improvements over state of the art baselines.

title: Fine Tuning Pre trained Language Model with Weak Supervision: A Contrastive Regularized Self Training Approach
abstract: Fine tuned pre trained language models (LMs) have achieved enormous success in many natural language processing (NLP) tasks, but they still require excessive labeled data in the fine tuning stage. We study the problem of fine tuning pre trained LMs using only weak supervision, without any labeled data. This problem is challenging because the high capacity of LMs makes them prone to overfitting the noisy labels generated by weak supervision. To address this problem, we develop a contrastive self training framework, COSINE, to enable fine tuning LMs with weak supervision. Underpinned by contrastive regularization and confidence based reweighting, our framework gradually improves model fitting while effectively suppressing error propagation. Experiments on sequence, token, and sentence pair classification tasks show that our model outperforms the strongest baseline by large margins and achieves competitive performance with fully supervised fine tuning methods. Our implementation is available on https:/github.com/yueyu1030/COSINE.

title: Co training an Unsupervised Constituency Parser with Weak Supervision
abstract: We introduce a method for unsupervised parsing that relies on bootstrapping classifiers to identify if a node dominates a specific span in a sentence. There are two types of classifiers, an inside classifier that acts on a span, and an outside classifier that acts on everything outside of a given span. Through self training and co training with the two classifiers, we show that the interplay between them helps improve the accuracy of both, and as a result, effectively parse. A seed bootstrapping technique prepares the data to train these classifiers. Our analyses further validate that such an approach in conjunction with weak supervision using prior branching knowledge of a known language (left/right branching) and minimal heuristics injects strong inductive bias into the parser, achieving 63.1 F1 on the English (PTB) test set. In addition, we show the effectiveness of our architecture by evaluating on treebanks for Chinese (CTB) and Japanese (KTB) and achieve new state of the art

title: Mixed Supervision Learning for Whole Slide Image Classification
abstract: Weak supervision learning on classification labels has demonstrated high performance in various tasks. When a few pixel level fine annotations are also affordable, it is natural to leverage both of the pixel level (e.g. segmentation) and image level (e.g. classification) annotation to further improve the performance. In computational pathology, however, such weak or mixed supervision learning is still a challenging task, since the high resolution of whole slide images makes it unattainable to perform endto end training of classification models. An alternative approach is to analyze such data by patch base model training, i.e. using self supervised learning to generate pixellevel pseudo labels for patches. However, such methods usually have model drifting issues, i.e. hard to converge, because the noise accumulates during the self training process. To handle those problems, we propose a mixed supervision learning framework for super high resolution images to effectively utilize their various labels (e.g. sufficient image level coarse annotations and a few pixel level fine labels) During the patch training stage, this framework can make use of coarse image level labels to refine selfsupervised learning and generate high quality pixel level pseudo labels. A comprehensive strategy is proposed to suppress pixel level false positives and false negatives. Three real world datasets with very large number of images (i.e. more than 10,000 whole slide images) and various types of labels are used to evaluate the effectiveness of mixed supervision learning. We reduced the false positive rate by around one third compared to state of the art while retaining 100% sensitivity, in the task of image level classification.

title: Weakly Supervised Hierarchical Text Classification
abstract: Hierarchical text classification, which aims to classify text documents into a given hierarchy, is an important task in many real world applications. Recently, deep neural models are gaining increasing popularity for text classification due to their expressive power and minimum requirement for feature engineering. However, applying deep neural networks for hierarchical text classification remains challenging, because they heavily rely on a large amount of training data and meanwhile cannot easily determine appropriate levels of documents in the hierarchical setting. In this paper, we propose a weakly supervised neural method for hierarchical text classification. Our method does not require a large amount of training data but requires only easy to provide weak supervision signals such as a few class related documents or keywords. Our method effectively leverages such weak supervision signals to generate pseudo documents for model pre training, and then performs self training on real unlabeled data to iteratively refine the model. During the training process, our model features a hierarchical neural structure, which mimics the given hierarchy and is capable of determining the proper levels for documents with a blocking mechanism. Experiments on three datasets from different domains demonstrate the efficacy of our method compared with a comprehensive set of baselines.

title: Learning Dual Retrieval Module for Semi supervised Relation Extraction
abstract: Relation extraction is an important task in structuring content of text data, and becomes especially challenging when learning with weak supervision where only a limited number of labeled sentences are given and a large number of unlabeled sentences are available. Most existing work exploits unlabeled data based on the ideas of self training (i.e. bootstrapping a model) and self ensembling (e.g. ensembling multiple model variants) However, these methods either suffer from the issue of semantic drift, or do not fully capture the problem characteristics of relation extraction. In this paper, we leverage a key insight that retrieving sentences expressing a relation is a dual task of predicting the relation label for a given sentence two tasks are complementary to each other and can be optimized jointly for mutual enhancement. To model this intuition, we propose DualRE, a principled framework that introduces a retrieval module which is jointly trained with the original relation prediction module. In this way, high quality samples selected by the retrieval module from unlabeled data can be used to improve the prediction module, and vice versa. Experimental results1 on two public datasets as well as case studies demonstrate the effectiveness of the DualRE approach.

title: Weakly Supervised Neural Text Classification
abstract: Deep neural networks are gaining increasing popularity for the classic text classification task, due to their strong expressive power and less requirement for feature engineering. Despite such attractiveness, neural text classification models suffer from the lack of training data in many real world applications. Although many semi supervised and weakly supervised text classification models exist, they cannot be easily applied to deep neural models and meanwhile support limited supervision types. In this paper, we propose a weakly supervised method that addresses the lack of training data in neural text classification. Our method consists of two modules: (1) a pseudo document generator that leverages seed information to generate pseudo labeled documents for model pre training, and (2) a self training module that bootstraps on real unlabeled data for model refinement. Our method has the flexibility to handle different types of weak supervision and can be easily integrated into existing deep neural models for text classification. We have performed extensive experiments on three real world datasets from different domains. The results demonstrate that our proposed method achieves inspiring performance without requiring excessive training data and outperforms baseline methods significantly.

title: Analisis kompetensi pedagogik guru PAI dalam
abstract: Pedagogical competence has a very important role in implementing the curriculum. Managing pedagogical competencies is a common problem in realizing learning objectives. The purpose of this study is to describe the pedagogical competence of PAI teachers, the efforts and constraints of PAI teachers in implementing curriculum in five state high schools in Aceh. This research uses descriptive qualitative research methods, data collection techniques through, observation, indepth interviews, and documentation. The results showed that the pedagogical competence of PAI teachers is well managed which is realized by the preparation of learning plans, syllabus, annual programs, and semester programs. The competency development effort is carried out through training, self development, supervision, motivation, and teacher training activities. While the constraints of curriculum implementation are lack of motivation, weak mastery of technology and information, lack of socialization of curriculum changes and evaluation of learning, limited learning resources, minimal curriculum structure allocation, and low student input.

title: VOCABULARY ALIGNMENT METHOD BASED ON ADVERSARIAL LEARNING
abstract: Tibetan is a low resource language. In order to alleviate the shortage of parallel corpus between Tibetan and Chinese, this paper uses two monolingual corpora and a small number of seed dictionaries to learn the semi supervised method with seed dictionaries and self supervised adversarial training method through the similarity calculation of word clusters in different embedded spaces and puts forward an improved self supervised adversarial learning method of Tibetan and Chinese monolingual data alignment only. The experimental results are as follows. First, the experimental results of Tibetan syllables Chinese characters are not good, which reflects the weak semantic correlation between Tibetan syllables and Chinese characters; second, the seed dictionary of semi supervised method made before 10 predicted word accuracy of 66.5 (Tibetan Chinese) and 74.8 (Chinese Tibetan) results, to improve the self supervision methods in both language directions have reached 53.5 accuracy.

title: A Aelf supervised Tibetan chinese Vocabulary Alignment Method Based On Adversarial Learning
abstract: Tibetan is a low resource language. In order to alleviate the shortage of parallel corpus between Tibetan and Chinese, this paper uses two monolingual corpora and a small number of seed dictionaries to learn the semi supervised method with seed dictionaries and self supervised adversarial training method through the similarity calculation of word clusters in different embedded spaces and puts forward an improved selfsupervised adversarial learning method of Tibetan and Chinese monolingual data alignment only. The experimental results are as follows. First, the experimental results of Tibetan syllables Chinese characters are not good, which reflects the weak semantic correlation between Tibetan syllables and Chinese characters; second, the seed dictionary of semi supervised method made before 10 predicted word accuracy of 66.5 (Tibetan Chinese) and 74.8 (Chinese Tibetan) results, to improve the self supervision methods in both language directions have reached 53.5 accuracy.