Query: Transformer and BERT

Candidate papers:

abstract: High performance gate drive power supply (GDPS) plays a crucial role in ensuring the reliability and safety of the gate driver for power semiconductor devices. This article focuses on the design of a high voltage insulated GDPS for the 10 kV silicon carbide MOSFET in medium voltage (MV) application. Design considerations, including insulation scheme, high voltage insulated transformer design, and load voltage regulation scheme, are proposed. In addition, the performance of the secondary side regulated (SSR) GDPS and that of the primary side regulated (PSR) GDPS are compared for several aspects, including interwinding capacitance, load voltage regulation rate, conversion efficiency, and hardware complexity. Finally, an SSR GDPS and a PSR GDPS, with an insulation voltage of 20 kV, are built in the lab. The test results demonstrate that the PSR GDPS is more preferable because of lower interwinding capacitance, lower load voltage regulation rate, higher conversion efficiency, and simpler control circuit.
author_names: Li Zhang

Shiqi Ji

Shida Gu

Xingxuan Huang

James Palmer

William Giewont

Leon M Tolbert
title: Design Considerations for High Voltage Insulated Gate Drive Power Supply for 10 kV SiC MOSFET Applied in Medium Voltage Converter
venue: IEEE Transactions on Industrial Electronics
year: 2021

abstract: In this article, we introduce a fractional N all digital phase locked loop (ADPLL) architecture based on a single LC tank, featuring an ultra wide tuning range (TR) and optimized for ultra low area in 10 nm FinFET CMOS. Underpinned by excellent switches in the FinFET technology, a high turn on/off capacitance ratio of LC tank switched capacitors, in addition to an adjustable magnetic coupling technique, yields almost an octave TR from 10.8 to 19.3GHz. A new method to compensate for the tracking bank resolution can maintain its quantization noise level over this wide TR. A new scheme is adopted to overcome the metastability resolution problem in a fractional N ADPLL operation. A low complexity TDC gain estimator reduces the digital core area by progressive averaging and time division multiplexing. Among the published fractional N PLLs with an area smaller than 0.1mm2, this work achieves an rms jitter of 725fs in an internal fractional N mode of ADPLL's phase detector (2.7 4.825GHz) yielding the best overall jitter figure of merit (FOM) of 232dB. This topology features small area (0.034mm2) wide TR (56.5% and good supply noise rejection (1.8%/V) resulting in FOMs with normalized TR (FOMT) of 247dB, and normalized TR and area (FOMTA) of 262dB.
author_names: Min-Shueh Yuan

Chia-Chun Liao

Chih-Hsien Chang

Yu-Tso Lin

Tsung-Hsien Tsai

Tien-Chien Huang

Hsien-Yuan Liao

Chung-Ting Lu

Hung-Yi Kuo

Augusto Ronchini Ximenes

Robert Bogdan Staszewski
title: A Compact Transformer Based Fractional N ADPLL in 10 nm FinFET CMOS
venue: IEEE Transactions on Circuits and Systems I: Regular Papers
year: 2021

abstract: Social web contains a large amount of information with user sentiment and opinions across different fields. For example, drugs.com provides users' textual review and numeric ratings of drugs. However, text reviews may not always be consistent with the numeric ratings. In this project, we built different classification models to classify user ratings of drugs with their textual review. Multiple supervised machine learning models including Random Forest and Naive Bayesian classifiers were built with drug reviews using TF IDF features as input. Also, transformer based neural network models including BERT, BioBERT, RoBERTa, XLNet, ELECTRA, and ALBERT were built for classification using the raw text as input. Overall, BioBERT model outperformed the other models with the overall accuracy of 87% This research demonstrated that transformer based classification models can be used to classify drug reviews and identify reviews that are inconsistent with the ratings.
author_names: A A Shiju

Z He
title: Classifying Drug Ratings Using User Reviews with Transformer Based Language Models
venue: medRxiv
year: 2021

abstract: Transformer based neural networks have heavily impacted the field of natural language processing, outperforming most previous state of the art models. However, well known models such as BERT, RoBERTa, and GPT 2 require a huge compute budget to create a high quality contextualised representations. In this paper, we study several efficient pre training objectives for Transformersbased models. By testing these objectives on different tasks, we determine which of the ELECTRA model's new features is the most relevant: (i) Transformers pre training can be improved when the input is not altered with artificial symbols, e.g. masked tokens; and (ii) loss functions computed using the whole output reduce training time. (iii) Additionally, we study efficient models composed of two blocks: a discriminator and a simple generator (inspired by the ELECTRA architecture) Our generator is based on a much simpler statistical approach, which minimally increases the computational cost. Our experiments show that it is possible to efficiently train BERT like models using a discriminative approach as in ELECTRA but without a complex generator. Finally, we show that ELECTRA largely benefits from a deep hyper parameter search.
author_names: Luca Di Liello

Matteo Gabburo

Alessandro Moschitti
title: Efficient pre training objectives for Transformers
venue: ArXiv
year: 2021

abstract: In this work, we show the process of building a large scale training set from digital and digitized collections at a national library. The resulting Bidirectional Encoder Representations from Transformers (BERT) based language model for Norwegian outperforms multilingual BERT (mBERT) models in several token and sequence classification tasks for both Norwegian Bokmal and Norwegian Nynorsk. Our model also improves the mBERT performance for other languages present in the corpus such as English, Swedish, and Danish. For languages not included in the corpus, the weights degrade moderately while keeping strong multilingual properties. Therefore, we show that building high quality models within a memory institution using somewhat noisy optical character recognition (OCR) content is feasible, and we hope to pave the way for other memory institutions to follow.
author_names: Per Egil Kummervold

Javier de la Rosa

Freddy Wetjen

Svein Arne Brygfjeld
title: Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model
venue: NODALIDA
year: 2021

abstract: We probe pre trained transformer language models for bridging inference. We first investigate individual attention heads in BERT and observe that attention heads at higher layers prominently focus on bridging relations in comparison with the lower and middle layers, also, few specific attention heads concentrate consistently on bridging. More importantly, we consider language models as a whole in our second approach where bridging anaphora resolution is formulated as a masked token prediction task (Of Cloze test) Our formulation produces optimistic results without any fine tuning, which indicates that pre trained language models substantially capture bridging inference. Our further investigation shows that the distance between anaphor antecedent and the context provided to language models play an important role in the inference.
author_names: Onkar Arun Pandit

Yufang Hou
title: Probing for Bridging Inference in Transformer Language Models
venue: NAACL
year: 2021

abstract: Major depressive disorder (MDD) is a prevalent psychiatric disorder that is associated with significant healthcare burden worldwide. Phenotyping of MDD can help early diagnosis and consequently may have significant advantages in patient management. In prior research MDD phenotypes have been extracted from structured Electronic Health Records (EHR) or using Electroencephalographic (EEG) data with traditional machine learning models to predict MDD phenotypes. However, MDD phenotypic information is also documented in free text EHR data, such as clinical notes. While clinical notes may provide more accurate phenotyping information, natural language processing (NLP) algorithms must be developed to abstract such information. Recent advancements in NLP resulted in state of the art neural language models, such as Bidirectional Encoder Representations for Transformers (BERT) model, which is a transformer based model that can be pre trained from a corpus of unsupervised text data and then fine tuned on specific tasks. However, such neural language models have been underutilized in clinical NLP tasks due to the lack of large training datasets. In the literature, researchers have utilized the distant supervision paradigm to train machine learning models on clinical text classification tasks to mitigate the issue of lacking annotated training data. It is still unknown whether the paradigm is effective for neural language models. In this paper, we propose to leverage the neural language models in a distant supervision paradigm to identify MDD phenotypes from clinical notes. The experimental results indicate that our proposed approach is effective in identifying MDD phenotypes and that the BioClinical BERT, a specific BERT model for clinical data, achieved the best performance in comparison with conventional machine learning models.
author_names: Bhavani Singh Agnikula Kshatriya

Nicolas A Nunez

Manuel Gardea-Resendez

Euijung Ryu

Brandon J Coombes

Sunyang Fu

Mark A Frye

Joanna M Biernacka

Yanshan Wang
title: Neural Language Models with Distant Supervision to Identify Major Depressive Disorder from Clinical Notes
venue: ArXiv
year: 2021

abstract: Increasing concerns and regulations about data privacy, necessitate the study of privacypreserving methods for natural language processing (NLP) applications. Federated learning (FL) provides promising methods for a large number of clients (i.e. personal devices or organizations) to collaboratively learn a shared global model to benefit all clients, while allowing users to keep their data locally. To facilitate FL research in NLP, we present the FedNLP1, a research platform for federated learning in NLP. FedNLP supports various popular task formulations in NLP such as text classification, sequence tagging, question answering, seq2seq generation, and language modeling. We also implement an interface between Transformer language models (e.g. BERT) and FL methods (e.g. FedAvg, FedOpt, etc. for distributed training. The evaluation protocol of this interface supports a comprehensive collection of non IID partitioning strategies. Our preliminary experiments with FedNLP reveal that there exists a large performance gap between learning on decentralized and centralized datasets opening intriguing and exciting future research directions aimed at developing FL methods suited to NLP tasks.
author_names: Bill Yuchen Lin

ZiHang Zeng

Hulin Wang

Yufen Huang

Mahdi Soltanolkotabi

Xiang Ren

Salman Avestimehr
title: FedNLP: A Research Platform for Federated Learning in Natural Language Processing
venue: ArXiv
year: 2021

abstract: Transformer based pre trained language models have significantly improved the performance of various natural language processing (NLP) tasks in the recent years. While effective and prevalent, these models are usually prohibitively large for resource limited deployment scenarios. A thread of research has thus been working on applying network pruning techniques under the pretrain then finetune paradigm widely adopted in NLP. However, the existing pruning results on benchmark transformers, such as BERT, are not as remarkable as the pruning results in the literature of convolutional neural networks (CNNs) In particular, common wisdom in pruning CNN states that sparse pruning technique compresses a model more than that obtained by reducing number of channels and layers, while existing works on sparse pruning of BERT yields inferior results than its small dense counterparts such as TinyBERT. In this work, we aim to fill this gap by studying how knowledge are transferred and lost during the pre train, fine tune, and pruning process, and proposing a knowledge aware sparse pruning process that achieves significantly superior results than existing literature. We show for the first time that sparse pruning compresses a BERT model significantly more than reducing its number of channels and layers. Experiments on multiple data sets of GLUE benchmark show that our method outperforms the leading competitors with a 20 times weight/FLOPs compression and neglectable loss in prediction accuracy.
author_names: Dongkuan Xu

Ian En-Hsu Yen

Jinxi Zhao

Zhibin Xiao
title: Rethinking Network Pruning under the Pre train and Fine tune Paradigm
venue: NAACL
year: 2021

abstract: We present the Hierarchical Transformer Networks for modeling long term dependencies across clinical notes for the purpose of patientlevel prediction. The network is equipped with three levels of Transformer based encoders to learn progressively from words to sentences, sentences to notes, and finally notes to patients. The first level from word to sentence directly applies a pre trained BERT model, and the second and third levels both implement a stack of 2 layer encoders before the final patient representation is fed into the classification layer for clinical predictions. Compared to traditional BERT models, our model increases the maximum input length from 512 words to much longer sequences that are appropriate for long sequences of clinical notes. We empirically examine and experiment with different parameters to identify an optimal trade off given computational resource limits. Our experimental results on the MIMIC III dataset for different prediction tasks demonstrate that our proposed hierarchical model outperforms previous state of the art hierarchical neural networks1. Codes are available at https:/github.com/Yuqi92/hier_ transformer_networks.
author_names: Yuqi Si

Kirk Roberts
title: Hierarchical Transformer Networks for Longitudinal Clinical Document Classification
venue: ArXiv
year: 2021