Query: BERT and transformer

Candidate papers:

abstract: High performance gate drive power supply (GDPS) plays a crucial role in ensuring the reliability and safety of the gate driver for power semiconductor devices. This article focuses on the design of a high voltage insulated GDPS for the 10 kV silicon carbide MOSFET in medium voltage (MV) application. Design considerations, including insulation scheme, high voltage insulated transformer design, and load voltage regulation scheme, are proposed. In addition, the performance of the secondary side regulated (SSR) GDPS and that of the primary side regulated (PSR) GDPS are compared for several aspects, including interwinding capacitance, load voltage regulation rate, conversion efficiency, and hardware complexity. Finally, an SSR GDPS and a PSR GDPS, with an insulation voltage of 20 kV, are built in the lab. The test results demonstrate that the PSR GDPS is more preferable because of lower interwinding capacitance, lower load voltage regulation rate, higher conversion efficiency, and simpler control circuit.
author_names: Li Zhang

Shiqi Ji

Shida Gu

Xingxuan Huang

James Palmer

William Giewont

Leon M Tolbert
title: Design Considerations for High Voltage Insulated Gate Drive Power Supply for 10 kV SiC MOSFET Applied in Medium Voltage Converter
venue: IEEE Transactions on Industrial Electronics
year: 2021

abstract: BERT based models are currently used for solving nearly all Natural Language Processing (NLP) tasks and most often achieve state of the art results. Therefore, the NLP community conducts extensive research on understanding these models, but above all on designing effective and efficient training procedures. Several ablation studies investigating how to train BERT like models have been carried out, but the vast majority of them concerned only the English language. A training procedure designed for English does not have to be universal and applicable to other especially typologically different languages. Therefore, this paper presents the first ablation study focused on Polish, which, unlike the isolating English language, is a fusional language. We design and thoroughly evaluate a pretraining procedure of transferring knowledge from multilingual to monolingual BERT based models. In addition to multilingual model initialization, other factors that possibly influence pretraining are also explored, i.e. training objective, corpus size, BPE Dropout, and pretraining length. Based on the proposed procedure, a Polish BERT based language model HerBERT is trained. This model achieves state of the art results on multiple downstream tasks.
author_names: Robert Mroczkowski

Piotr Rybak

Alina Wroblewska

Ireneusz Gawlik
title: HerBERT: Efficiently Pretrained Transformer based Language Model for Polish
venue: BSNLP
year: 2021

abstract: Technology assisted review (TAR) refers to iterative active learning workflows for document review in high recall retrieval (HRR) tasks. TAR research and most commercial TAR software has applied linear models such as logistic regression or support vector machines to lexical features. Transformer based models with supervised tuning have been found to improve effectiveness on many text classification tasks, suggesting their use in TAR. We indeed find that the pre trained BERT model reduces review volume by 30% in TAR workflows simulated on the RCV1 v2 newswire collection. In contrast, we find that linear models outperform BERT for simulated legal discovery topics on the Jeb Bush e mail collection. This suggests the match between transformer pre training corpora and the task domain is more important than generally appreciated. Additionally, we show that just right language model fine tuning on the task collection before starting active learning is critical. Both too little or too much fine tuning results in performance worse than that of linear models, even for RCV1 v2.
author_names: Eugene Yang

Sean MacAvaney

David D Lewis

Ophir Frieder
title: Goldilocks: Just Right Tuning of BERT for Technology Assisted Review
venue: ArXiv
year: 2021

abstract: In this article, we introduce a fractional N all digital phase locked loop (ADPLL) architecture based on a single LC tank, featuring an ultra wide tuning range (TR) and optimized for ultra low area in 10 nm FinFET CMOS. Underpinned by excellent switches in the FinFET technology, a high turn on/off capacitance ratio of LC tank switched capacitors, in addition to an adjustable magnetic coupling technique, yields almost an octave TR from 10.8 to 19.3GHz. A new method to compensate for the tracking bank resolution can maintain its quantization noise level over this wide TR. A new scheme is adopted to overcome the metastability resolution problem in a fractional N ADPLL operation. A low complexity TDC gain estimator reduces the digital core area by progressive averaging and time division multiplexing. Among the published fractional N PLLs with an area smaller than 0.1mm2, this work achieves an rms jitter of 725fs in an internal fractional N mode of ADPLL's phase detector (2.7 4.825GHz) yielding the best overall jitter figure of merit (FOM) of 232dB. This topology features small area (0.034mm2) wide TR (56.5% and good supply noise rejection (1.8%/V) resulting in FOMs with normalized TR (FOMT) of 247dB, and normalized TR and area (FOMTA) of 262dB.
author_names: Min-Shueh Yuan

Chia-Chun Liao

Chih-Hsien Chang

Yu-Tso Lin

Tsung-Hsien Tsai

Tien-Chien Huang

Hsien-Yuan Liao

Chung-Ting Lu

Hung-Yi Kuo

Augusto Ronchini Ximenes

Robert Bogdan Staszewski
title: A Compact Transformer Based Fractional N ADPLL in 10 nm FinFET CMOS
venue: IEEE Transactions on Circuits and Systems I: Regular Papers
year: 2021

abstract: This paper demonstrates a wideband 2.4 GHz <inline formula> <tex math notation="LaTeX"$2\times 9$ /tex math>/inline formula> bit Cartesian radio frequency digital to analog converter (RFDAC) Active under coil integration is introduced in the physical implementation, where all key active circuitry is located underneath the matching network transformer, achieving a core area of merely 0.35 mm<sup>2</sup> An <inline formula> <tex math notation="LaTeX"$8\times /tex math>/inline formula> analog linear interpolation at the RF rate is proposed to suppress replicas close to the carrier while avoiding any high order and high speed digital filters in digital processing back end. The multi port transformer is adopted in the matching network to improve the back off efficiency. The measured peak output power and drain efficiency at the center frequency of 2.4 GHz are 17.47 dBm and 17.6% respectively, while the peak efficiency is 19.03% Moreover, the 6 dB back off efficiency is at 66% of that at the peak output power. The active under coil integration helps this RFDAC to achieve the smallest area among comparable prior arts.
author_names: Feifei Zhang

Jeffrey Sean Walling

Anding Zhu

Robert Bogdan Staszewski
title: An Active Under Coil RFDAC With Analog Linear Interpolation in 28 nm CMOS
venue: IEEE Transactions on Circuits and Systems I: Regular Papers
year: 2021

abstract: Vision and Language (VL) pre training has shown great potential on many related downstream tasks, such as Visual Question Answering (VQA) one of the most popular problems in the VL field. All of these pre trained models (such as VisualBERT, ViLBERT, LXMERT and UNITER) are built with Transformer, which extends the classical attention mechanism to multiple layers and heads. To investigate why and how these models work on VQA so well, in this paper we explore the roles of individual heads and layers in Transformer models when handling 12 different types of questions. Specifically, we manually remove (chop) heads (or layers) from a pre trained VisualBERT model at a time, and test it on different levels of questions to record its performance. As shown in the interesting echelon shape of the result matrices, experiments reveal different heads and layers are responsible for different question types, with higher level layers activated by higher level visual reasoning questions. Based on this observation, we design a dynamic chopping module that can automatically remove heads and layers of the VisualBERT at an instance level when dealing with different questions. Our dynamic chopping module can effectively reduce the parameters of the original model by 50% while only damaging the accuracy by less than 1% on the VQA task.
author_names: Chenyu Gao

Qi Wu
title: Chop Chop BERT: Visual Question Answering by Chopping VisualBERT's Heads
venue: IJCAI
year: 2021

abstract: Social web contains a large amount of information with user sentiment and opinions across different fields. For example, drugs.com provides users' textual review and numeric ratings of drugs. However, text reviews may not always be consistent with the numeric ratings. In this project, we built different classification models to classify user ratings of drugs with their textual review. Multiple supervised machine learning models including Random Forest and Naive Bayesian classifiers were built with drug reviews using TF IDF features as input. Also, transformer based neural network models including BERT, BioBERT, RoBERTa, XLNet, ELECTRA, and ALBERT were built for classification using the raw text as input. Overall, BioBERT model outperformed the other models with the overall accuracy of 87% This research demonstrated that transformer based classification models can be used to classify drug reviews and identify reviews that are inconsistent with the ratings.
author_names: A A Shiju

Z He
title: Classifying Drug Ratings Using User Reviews with Transformer Based Language Models
venue: medRxiv
year: 2021

abstract: Transformer based neural networks have heavily impacted the field of natural language processing, outperforming most previous state of the art models. However, well known models such as BERT, RoBERTa, and GPT 2 require a huge compute budget to create a high quality contextualised representations. In this paper, we study several efficient pre training objectives for Transformersbased models. By testing these objectives on different tasks, we determine which of the ELECTRA model's new features is the most relevant: (i) Transformers pre training can be improved when the input is not altered with artificial symbols, e.g. masked tokens; and (ii) loss functions computed using the whole output reduce training time. (iii) Additionally, we study efficient models composed of two blocks: a discriminator and a simple generator (inspired by the ELECTRA architecture) Our generator is based on a much simpler statistical approach, which minimally increases the computational cost. Our experiments show that it is possible to efficiently train BERT like models using a discriminative approach as in ELECTRA but without a complex generator. Finally, we show that ELECTRA largely benefits from a deep hyper parameter search.
author_names: Luca Di Liello

Matteo Gabburo

Alessandro Moschitti
title: Efficient pre training objectives for Transformers
venue: ArXiv
year: 2021

abstract: In this work, we show the process of building a large scale training set from digital and digitized collections at a national library. The resulting Bidirectional Encoder Representations from Transformers (BERT) based language model for Norwegian outperforms multilingual BERT (mBERT) models in several token and sequence classification tasks for both Norwegian Bokmal and Norwegian Nynorsk. Our model also improves the mBERT performance for other languages present in the corpus such as English, Swedish, and Danish. For languages not included in the corpus, the weights degrade moderately while keeping strong multilingual properties. Therefore, we show that building high quality models within a memory institution using somewhat noisy optical character recognition (OCR) content is feasible, and we hope to pave the way for other memory institutions to follow.
author_names: Per Egil Kummervold

Javier de la Rosa

Freddy Wetjen

Svein Arne Brygfjeld
title: Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model
venue: NODALIDA
year: 2021

abstract: We probe pre trained transformer language models for bridging inference. We first investigate individual attention heads in BERT and observe that attention heads at higher layers prominently focus on bridging relations in comparison with the lower and middle layers, also, few specific attention heads concentrate consistently on bridging. More importantly, we consider language models as a whole in our second approach where bridging anaphora resolution is formulated as a masked token prediction task (Of Cloze test) Our formulation produces optimistic results without any fine tuning, which indicates that pre trained language models substantially capture bridging inference. Our further investigation shows that the distance between anaphor antecedent and the context provided to language models play an important role in the inference.
author_names: Onkar Arun Pandit

Yufang Hou
title: Probing for Bridging Inference in Transformer Language Models
venue: NAACL
year: 2021